Labelling 9,000 Tomato Photos Showed Me My Own Bias Before the Model's

Last summer our team trained a model to grade tomatoes by ripeness on a sorting line at one of our partner farms near Parma. The line photographs every tomato from two angles, and the model sorts them into four ripeness grades for different uses: fresh market, sauce, paste, and reject. I labelled a large part of the training set myself, about 9,000 images across five weeks.

When the first model came back, I was annoyed. It disagreed with my labels on around 14 percent of the validation set, mostly between neighbouring grades. I assumed the model was the problem.

Looking at the disagreements

I pulled 300 disagreements and looked at them one by one with a colleague, Giacomo, who has worked with growers for fifteen years. We did not tell each other our labels at first. We just labelled again.

On about a third of them, Giacomo agreed with the model, not with me.

That stung. So I did something I should have done earlier. I went back to my original labelling logs, which had timestamps, and looked at when I had labelled the images I now disagreed with.

The Monday effect

The pattern was clear and slightly embarrassing. My labels drifted over each session. In the first 20 minutes of a session, I was strict, and pushed borderline tomatoes toward the less ripe grade. After about an hour, I became more generous. And on Monday mornings, after a weekend away, my first batch of labels was noticeably stricter than the rest of the week.

When I compared my Monday morning labels against a re-label done by Giacomo, my agreement with him was 79 percent. For mid-week afternoon sessions, it was 91 percent.

My eyes were adjusting to the colour of the tomatoes I had just seen. After an hour of red, a slightly orange tomato looked less ripe than it would have at the start.

Changes to how we label

We made several changes for the next round of labelling.

• Shorter sessions. Maximum 40 minutes, followed by a break.
• Anchor images. Before each session, the labeller reviews 12 reference images, three per grade, agreed on by the whole team.
• Shuffled batches. Images are presented in random order across farms and days so a labeller never sees a long run of similar tomatoes.
• Overlap. Ten percent of images are labelled by two people, and we track agreement per labeller per session.

With those changes, agreement between labellers rose from 84 to 93 percent. The retrained model's disagreement with the new consensus labels fell to about 6 percent.

The model was a mirror

What I took from this is that the model was not just learning tomatoes. It was learning me, including my tiredness and my Monday moods. When it disagreed with me, it was sometimes averaging out my inconsistency better than I could.

Now when I see model errors, my first question is no longer "what did the model get wrong?" It is "what were we doing when we made these labels?"