I Let an AI Coding Assistant Write My Tests for a Month

In March I ran a small experiment on myself. For one month, every time I needed to write unit tests, I asked an AI coding assistant to write them first. I would review, edit, or reject them, and I kept a tally in a spreadsheet. I work on backend services in Go and a bit of TypeScript, so most of the tests were for HTTP handlers, data transformations, and some database code.

By the end of the month, the assistant had produced 412 test cases across 38 files. I kept 260 of them unchanged or with minor edits. I rewrote 71. I deleted 81.

What it was good at

The assistant was excellent at the tedious parts. Table-driven tests with fifteen input cases, for example. It would happily generate empty strings, very long strings, unicode, negative numbers, and nil values. It thought of edge cases I would have skipped out of laziness, not ignorance.

It was also good at matching the style of existing tests in a file. If I had a helper function for building fake requests, it found and used it about 80 percent of the time.

On a typical handler, writing tests by hand would have taken me maybe 40 minutes. With the assistant, it took about 15 including review.

The 81 I deleted

This is where it got interesting. The deleted tests fell into three groups.

Tests that tested the mock. About 30 of them set up a mock to return a value and then asserted that the code returned the same value. They passed, they covered lines, and they proved nothing.

Tests that encoded bugs. About 20 tests asserted the current behavior of the code, including behavior that was wrong. In one case, a function silently dropped records with a missing timestamp. The assistant wrote a test confirming that records were dropped. If I had accepted it, the bug would have been locked in.

Redundant tests. The rest duplicated each other with trivial variations.

Why the bad ones matter

The tests that encoded bugs worried me most. The assistant was reading my code and describing it, not reading my intentions. Tests are supposed to say what the code should do. A test generated from the implementation can only say what it does.

I changed my workflow halfway through the month. Instead of giving the assistant my implementation, I gave it the function signature and a short paragraph describing what the function was supposed to do. The tests got noticeably better. Fewer mock-testing-mock cases, and two of them actually failed against my code, revealing real bugs.

Where I landed

I still use the assistant for tests, but in a specific way. I write a description of the expected behavior first, let it generate cases, then read each one and ask whether it would fail if the code were wrong. If the answer is no, it gets deleted.

My coverage numbers went up during the experiment, from 64 to 78 percent. But I trust the 260 tests I kept more than the number. Coverage tells you which lines ran. It does not tell you whether anything was checked.