Postmortem Index

Explore incident reports from various companies

Linux kernel leap second futex timer issue

Linux · Linux kernel

2012-07-01 cloud time

On July 1, 2012, a discussion on the Linux Kernel Mailing List (LKML) detailed a critical issue related to leap seconds. When a leap second occurred, CLOCK_REALTIME was correctly rewound by one second. However, the hrtimer subsystem’s internal base.offset value, which tracks wall time for CLOCK_REALTIME timers, was not updated. This caused the hrtimer’s perception of time to be one second ahead of the system’s actual CLOCK_REALTIME.

The root cause was identified as the omission of a call to clock_was_set() when a leap second was applied. Without this function call, the hrtimer base.offset remained out of sync with the CLOCK_REALTIME adjustment, leading to an inconsistency in timekeeping within the kernel. This state would persist until clock_was_set() was manually invoked, for example, via settimeofday().

The primary impact was that all TIMER_ABSTIME CLOCK_REALTIME timers would expire one second early. More critically, sub-second timers of this type, especially those used in loops (such as with futex_wait or other timeouts), would return immediately instead of sleeping. This behavior caused applications to spin-wait, leading to significant load spikes on affected systems and potentially causing widespread service disruptions.

This issue was reproducible across various Linux kernel versions, including v3.5-rc4, v2.6.37, and v2.6.32.59, with its origin potentially dating back to v2.6.22. The proposed remediation involved modifying the kernel to ensure that clock_was_set() is explicitly invoked when a leap second is applied, thereby synchronizing the hrtimer’s offset and resolving the timing inconsistency.

Keywords

linuxkernelleap secondclock_realtimehrtimerfutextimersystem loadlinux kerneltimekeeping