I Counted How Often My AI Assistant Was Confidently Wrong

I use an AI chat assistant at work every day, mostly for programming questions: library APIs, error messages, configuration options, small snippets. At some point I realized I had no idea how often it was right. I had a vague feeling of "mostly", but I had never checked. So for six weeks, I logged every technical question I asked and whether the answer turned out to be correct.

I ended up with 318 questions. Here is what I found.

How I scored answers

I used three labels. Correct meant the answer worked as given or needed only trivial changes. Partly correct meant the general direction was right but something specific was wrong, like a parameter name. Wrong meant following the answer would have led me astray.

I only scored an answer after I had verified it, either by running the code, reading official documentation, or both.

The numbers

Out of 318 questions:

• 241 were correct, about 76 percent.
• 42 were partly correct, about 13 percent.
• 35 were wrong, about 11 percent.

So roughly one answer in nine was wrong. That is better than I feared and worse than it felt.

I also noted whether the assistant expressed any uncertainty, phrases like "I think", "you may want to check", or "this may differ by version". It hedged on 22 answers. Of those, 9 were wrong. Of the 296 answers with no hedging, 26 were wrong. So hedging was somewhat informative, but most wrong answers came with no warning at all.

Where it was wrong

The wrong answers clustered in a few places.

Version-specific details. A configuration option that existed in an older version of a library but had been renamed. This was the biggest group, 14 of the 35.

Niche libraries. Anything with fewer users got more invented function names.

Our internal conventions. Of course it did not know them, but it sometimes guessed confidently instead of saying so.

It was very reliable on fundamentals: language syntax, standard library behavior for popular languages, general algorithm questions, and explaining error messages.

What I changed

Three habits came out of this.

First, for anything version-specific, I now tell the assistant the exact version in my question. This alone seemed to cut errors in that category by about half in the two weeks after the experiment, though my sample there was small.

Second, I treat answers about niche libraries as a starting point for reading the docs, not a replacement.

Third, I verify before I paste. That sounds obvious, but before this I would often paste a snippet, see it compile, and move on. Compiling is not the same as correct.

The confidence problem

The most useful thing I learned is that the tone of an answer told me very little. Wrong answers were just as fluent and specific as right ones. I trust the tool more now that I know the actual rate, and I trust its tone less. Both feel like the right adjustment.