Total Pageviews

My YouTube Channel

Thursday, 11 April 2013

Determining and changing the rate of timer interrupts a guest operating system requests (1005802)


Many aspects of timekeeping are influenced by the rate of timer interrupts that a guest operating system requests. This article lists the default values for a number of kernels and how to measure the rate.
Many guest operating systems keep track of time by programming a periodic timer interrupt and incrementing the current time by the period of the interrupt every time an interrupt is received. Such periodic timer interrupts are often referred to as "ticks" and this method of timekeeping is known as tick counting.



Windows generally requests 64 or 100 interrupts per second depending on which HAL is use. There is a Windows multimedia timer API that allows applications to raise this to 1024 interrupts per second. There is no easy way to determine what the timer interrupt rate rate is from within the virtual machine. Timekeeping in a VMware Virtual Machinedescribes how to use TimeTrackerStats to determine the rate of timer interrupts the virtual machine is requesting.   


Linux kernels request a timer interrupt from the Programmable Interval Timer (PIT) and an interrupt from each Local APIC Timer, of which there is one per virtual CPU. These are all requested at the same rate, but that rate varies from kernel to kernel. The requested rate of timer interrupts is referred to in the Linux kernel as HZ. The total number of timer interrupts requested by a Linux kernel is HZ * (N + 1) where N is the number of virtual CPUs in the virtual machine.

Default mainline values

Different values can be selected at compile time. Vendor kernels based on a given mainline kernel may have a different value than the default.
  • Linux 2.4: HZ = 100Hz.
  • Linux 2.6.0 - 2.6.13: HZ = 1000Hz.
  • Linux 2.6.14 onward: HZ = 250Hz.

Tickless Linux Kernels

Starting with Linux 2.6.18, Linux kernels began transitioning away from tick counting. Some new kernels use aperiodic interrupts rather than programming a periodic timer interrupt. These kernels are called tickless kernels. For tickless Linux kernels, HZ still influences the rate at which aperiodic timer interrupts are requested, but since these interrupts are not counted for timekeeping, the value of HZ does not impact timekeeping.
Generally tickless Linux kernels do not use the PIT at all, and instead just use aperiodic Local APIC timer interrupts.

Affected Vendor kernels

  • Redhat Enterprise Linux 4: HZ = 1000Hz
  • Redhat Enterprise Linux 5: HZ = 1000Hz
  • SUSE Linux Enterprise Server 9: HZ = 1000Hz
  • SUSE Linux Enterprise Server 10: HZ = 250Hz
  • Ubuntu 6.10: HZ = 250Hz
  • Ubuntu 7.04 and 7.10 Desktop: HZ = 250Hz  

"divider" kernel command line option

Redhat Enterprise Linux 5 starting with version 5.1 and Redhat Enterprise Linux 4 starting with version 4.7 support a kernel command line option that allows a divider to be supplied that makes the kernel request HZ/divider timer interrupts per second. The syntax is divider=x where x is the divider. 
To change the requested rate to 100Hz, pass divider=10 on the kernel command line. RHEL 5.1 64bit kernels have a bug in their implementation of the divider= option. The bug is fixed in Redhat RHBA-2007-0959. Do not use thedivider= option with 64bit RHEL 5.1 unless this fix is applied.

Recompiling with a different HZ value

In standard 2.6 kernels, the timer interrupt rate is fixed at kernel compile time and cannot be changed by command line parameters. You can, however, recompile your kernel with a lower timer interrupt rate. 100Hz is adequate for most applications in a Linux guest. See the documentation for your Linux distribution for detailed instructions on how to build and run a custom kernel. Some version of Linux allow the value of HZ to be selected during the "make config" stage. Other kernels require the source to be modified. For those kernels, before recompiling the guest kernel, locate the following line in include/asm-i386/param.h orinclude/asm-x86_64/param.h: #define HZ 1000 . Change the value of HZ to 100: #define HZ 100 .

Measuring HZ

To measure the actual rate of timer interrupts in a Linux virtual machine:
  1. Compare /proc/interrupts at the beginning and end of a 10 second interval. 
  2. From a shell prompt run cat /proc/interrupts ; sleep 10; cat /proc/interrupts .

    There are several columns. 
    The first is the IRQ number. The PIT is the device that generates the interrupts that Linux counts for time keeping. This is IRQ 0. Also of interest is "LOC" which is the Local APIC timer. For each vCPU there is a column listing the number of interrupts that device has raised on that vCPU. The final two columns give the interrupt type and a name for the devices on that IRQ.
To compute the rate of timer interrupts:
  • before = Sum of all PIT interrupts received on all vCPUs before sleep.
  • after = Sum of all PIT interrupts received on all vCPUs after sleep.
  • timer rate = (after - before) / 10 seconds
For example:
         CPU0       CPU1        
0:     125251      79291    IO-APIC-edge  timer 
1:        591        585    IO-APIC-edge  i8042 
8:          0          0    IO-APIC-edge  rtc 
9:          0          0    IO-APIC-level  acpi
12:        67          8    IO-APIC-edge  i8042
14:       753        643    IO-APIC-edge  ide0
169:     2840        142    IO-APIC-level  ioc0
177:      748         19    IO-APIC-level  eth0
NMI:       43         35   
LOC:     204282     204830
ERR:        0
MIS:        0          
         CPU0       CPU1        
0:     134539      80039    IO-APIC-edge  timer 
1:        592        585    IO-APIC-edge  i8042 
8:          0          0    IO-APIC-edge  rtc 
9:          0          0    IO-APIC-level  acpi
12:        67          8    IO-APIC-edge  i8042
14:       771        715    IO-APIC-edge  ide0
169:     2840        147    IO-APIC-level  ioc0
177:      800         19    IO-APIC-level  eth0
NMI:       43         36   
LOC:     214314     214862
ERR:        0
MIS:        0
before = 125251 + 79291 = 204542
after = 134539 + 80039 = 214578
timer rate = (214578 - 204542) / 10 seconds = 1003/sec
This matches the 1000Hz rate for Redhat Enterprise Linux 4 that is expected.
If no PIT interrupts are seen in 10 seconds, the kernel is a tickless kernel and the HZ value does not matter for timekeeping.