Feature Flags Turned Our Codebase Into a Haunted House

Feature flags are a lovely idea. You ship code dark, turn it on for a few users, watch the graphs, and roll it out. We adopted them enthusiastically around 2018. By last year, our main service had 312 of them.

I know the exact number because I wrote a script to count them after a production incident in which someone flipped a flag that had been off for 22 months. The code behind it called an API endpoint that no longer existed.

How 312 happens

Nobody sets out to create hundreds of flags. Each one made sense on the day it was added. The problem is that adding a flag is a five-minute job and removing one is a chore with no deadline.

When I categorized them, the breakdown looked like this:

• 41 flags were on for every user and had been for more than six months.
• 38 were off for every user and had been for more than six months.
• 19 were referenced in configuration but not in any code.
• About 60 were real, active, and doing their job.
• The rest were somewhere in between, mostly partial rollouts that someone forgot to finish.

So roughly a quarter of our flags were ghosts. They were permanently set one way, but the code still contained both paths, and the tests still covered both.

The incident that started it

The flag that bit us was called something like use_new_rating_engine. It was off. An engineer on another team saw it in the admin panel during a debugging session and thought it might help. They turned it on for a single state.

The "new" rating engine was actually the old new rating engine, from a project that had been cancelled. It called a service that had been shut down. Quotes in that state failed for 14 minutes before someone noticed.

Nobody did anything wrong in the usual sense. The admin panel let them flip it, and nothing told them it was dead.

What we put in place

We did three things, and the first one did most of the work.

Every new flag gets an owner and an expiry date. The expiry is not enforced by turning the flag off. It is enforced by failing a CI check on the owner's next pull request after the date passes. That turned out to be exactly annoying enough.

We hide stale flags in the admin panel. Anything that has not changed in 90 days moves to a collapsed section with a warning label.

We held one flag cleanup week per quarter. The first one removed 104 flags. The second removed 47. Now we average about 15 per quarter, which is roughly the rate we add them.

A flag is a fork in the road

What changed for me was how I think about adding one. A flag is not a switch. It is a second version of the code that someone has to read, test, and understand. When I add one now, I write the removal ticket at the same time and put it on the board.

We are down to 88 flags. The house is a lot quieter.