The AI Feature We Shipped, Measured, and Then Quietly Turned Off

Last autumn my team shipped an AI feature into our B2B procurement app. When a buyer searched for a product, a short generated summary appeared at the top of the results comparing the top three suppliers on price, delivery time, and minimum order quantity. It looked fantastic in the demo. The sales team loved it. Our CEO mentioned it in a customer newsletter.

Four months later we turned it off. Nobody complained.

How we built it

The summary was generated by a hosted language model. We passed it structured data for the top results and asked for a short comparison in plain language. We added caching so the same search did not regenerate constantly, and a feedback button with a thumbs up and thumbs down.

It took two engineers about five weeks, including evaluation, prompt tuning, and handling cases where the model hallucinated a delivery time that was not in the data. That happened in roughly 3 percent of early outputs. We got it down to near zero by giving the model less freedom and more structure.

What the data said

We tracked a few things. The summary was shown on about 41,000 searches in four months. The feedback button was clicked 212 times, roughly evenly split between up and down. That is a 0.5 percent interaction rate.

More telling was scroll behavior. On searches where the summary appeared, users scrolled past it to the results table in a median of 1.4 seconds. Most of them were clearly not reading it.

We also checked whether it changed outcomes. Conversion from search to purchase order was 0.3 percentage points higher with the summary, which was within the noise for our sample size.

Why it did not land

We interviewed eight buyers who used the product heavily. The answers were consistent and a little embarrassing.

Procurement buyers are experts. They already know their suppliers. They wanted the table, sorted by the column they cared about, and they wanted it fast. A paragraph of prose was slower to read than a row of numbers. One buyer said, kindly, that the summary felt like "someone reading the table aloud to me".

Two buyers said they did not trust it because they could not tell where the numbers came from, even though they came straight from the table below.

What we did instead

We turned off the summary and spent two weeks on something much less impressive: a saved sort preference per user and a column for the supplier's on-time delivery rate over the last 90 days. That change got more positive feedback in its first week than the summary got in four months.

The cost of the summary was also real. Model calls added about 600 euros a month for a feature people scrolled past.

I am not anti-AI features. I build them. But this one taught me to ask a question I skipped: would an expert user rather read this sentence, or the data it was made from? For our users, the answer was the data, almost every time.