The AI Feature We Shipped, Measured, and Then Quietly Turned Off
Our smart search summaries looked great in demos. Real users scrolled right past them.
Backend developer in Wroclaw. I keep a bug journal, argue about Rust, and test AI tools so my team does not have to.
Our smart search summaries looked great in demos. Real users scrolled right past them.
Two servers, one nightly job, and invoices that went out twice. The fix was eleven lines.
It wrote 412 tests. I kept 260. The ones I threw away taught me more than the ones I kept.
Over 300 questions in six weeks, it was wrong 1 time in 9. It sounded sure almost every time.
Eight tokens per second, no internet, no bill, and a surprising amount of usefulness for boring tasks.
I thought I was bad at concurrency. It turns out I am bad at reading error messages to the end.
The benchmark we bragged about improved 6x. The thing users noticed improved by 11 percent.