Our Flaky Test Was Actually a Daylight Saving Time Bug Hiding for Three Years
It failed twice a year, always on a Monday, and we kept clicking re-run.
Every team has a flaky test. Ours was called test_policy_renewal_window, and it failed roughly twice a year. When it failed, someone would click re-run, it would pass, and we would all move on. It had been doing this for at least three years before I finally sat down with it.
What made me look was boredom, honestly. It was a slow Monday in early November, the test failed on my branch, and instead of re-running it I opened the file.
The pattern nobody noticed
I pulled our CI history for the test going back as far as we kept logs. It had failed nine times. I wrote the dates down on a sticky note, and the pattern was embarrassing in hindsight. Every single failure was on the Monday or Tuesday after a daylight saving time change, in either March or November.
The test created a policy, set its renewal date to "30 days from now", and then checked that a function returned true for "is within renewal window". The function computed the window by subtracting dates. The test computed the expected value by subtracting timestamps in seconds and dividing by 86,400.
When a clock change fell inside that 30-day window, one of those days was 23 or 25 hours long. The timestamp math came out to 29.96 or 30.04 days. Depending on rounding and the exact time the test ran, it would occasionally flip.
The bug was not only in the test
This is the part that made my stomach drop. The test was wrong, but it was wrong in a way that mirrored a real bug in a reporting job. That job used the same seconds-divided-by-86,400 trick to count days until renewal for a weekly email to agents.
For a few days each spring and autumn, some policies were showing up one day early or one day late in that email. Nobody had ever reported it, because agents mostly ignored the exact number. But it was wrong, and it had been wrong for years.
What we changed
The fix itself was small.
- We replaced every place that counted days by dividing seconds with a proper calendar-date difference in the policy's local time zone.
- We added a test helper that freezes time at a set of nasty dates: the two clock changes, the last day of February in a leap year, and December 31.
- We added a lint rule that flags the literal 86400 anywhere outside one approved utility module.
That lint rule found four more places in other services.
Re-run is a decision
The thing I keep coming back to is how many times someone clicked re-run. Nine failures, maybe five different engineers, and every one of us decided the test was the problem rather than the code. That is a reasonable call most of the time. Flaky tests are usually flaky because of timing or shared state.
But now when a test fails, I write the date down before I re-run it. It takes five seconds. A list of dates is often all you need to see the shape of a bug that only shows up twice a year.
