Why We Moved Every Cron Job Into One Boring Queue
Fourteen crontabs on six servers, zero alerts, one very bad Monday. This is how we cleaned it up.
We discovered we had a cron problem the way most teams do: something stopped happening and nobody noticed for nine days.
The something was a nightly job that expired unused trial accounts. The server it ran on had been replaced during a routine upgrade. The new machine came up healthy, served traffic perfectly, and had an empty crontab. Nine days of trials quietly stayed open.
The audit
We went looking and found fourteen scheduled jobs spread across six servers. Some were in crontab files. Some were in systemd timers. One was a setInterval inside a web process, which meant it ran once per replica, which meant it ran four times every hour.
None of them reported success or failure anywhere. The only signal that a job had run was the absence of complaints.
What we wanted
We wrote down requirements before picking anything:
- One place to see every scheduled job and its last run.
- An alert if a job fails or simply does not run.
- Exactly one execution per schedule, no matter how many servers exist.
- Retries for transient failures, with a limit.
- Something the whole team already understood.
That last point ruled out the shiny options. We already ran a job queue backed by Postgres for emails and webhooks. It had a dashboard, retry support and a dead-letter table. It just did not know about time.
The design
We added a tiny scheduler: one process, elected by a database lock, whose only job is to enqueue work when a schedule fires. It does no work itself. Regular workers pick up the jobs like any other task.
Each job records three timestamps: when it was scheduled, when it started and when it finished. A separate check runs every five minutes and alerts if any job has not finished successfully within twice its expected interval. That catches crashes, silent hangs and, crucially, jobs that never started at all.
The migration
We moved jobs one at a time over three weeks. For each one we:
- Ported the logic into a queue handler.
- Ran both versions in parallel for a few days, with the old one in dry-run mode.
- Compared outputs, then deleted the old schedule.
The setInterval job was the most satisfying deletion. It turned out the four duplicate runs had been masking a bug that the single run immediately exposed.
Six months later
We now have 19 scheduled jobs, all visible on one page. We have had three real alerts, each caught within an hour. Adding a new job is a 15-line PR.
The best infrastructure is the kind that makes missing work loud.
Cron is not bad. Cron that nobody watches is. If you cannot answer the question did it run last night? in under a minute, you probably have the same problem we did.
Cover: Photo by Taylor Vick on Unsplash.
