7 Questions I Ask Before Training Any Model

I have killed more machine learning projects in the first meeting than I have shipped. That sounds negative, but it is the most useful thing I do. A project that dies in week one costs a meeting. A project that dies in month six costs a team.

Here are the questions I ask before anyone opens a notebook.

The questions

1. What decision will this change?

If nobody can name a specific decision that will be made differently because of the model, stop. A prediction that does not change an action is a dashboard, not a product. Good answer: we will route high-risk orders to manual review. Bad answer: it will give us insights.

2. What happens today without the model?

There is always a baseline. Sometimes it is a rule of thumb, sometimes a person, sometimes nothing. Write it down and measure it. If the baseline already gets you 90% of the value, a model has to justify the remaining 10% plus all the maintenance cost.

3. Do we have labels, and are they honest?

A fraud model needs examples of fraud. A quality model needs examples of good and bad. Many teams discover that their labels are missing, inconsistent, or quietly biased. If fraud was only labelled when someone happened to investigate, your labels describe what investigators looked at, not what fraud looks like.

4. Will the data exist at prediction time?

This one catches experienced people. A feature like total refunds for the order looks fantastic in training, until you realise it is only known weeks after the moment you need to predict. That is called leakage, and it produces models that look brilliant offline and useless in production.

5. What does a wrong answer cost?

False positives and false negatives almost never cost the same. Flagging a good customer as fraud might cost a support call and some goodwill. Missing real fraud might cost $500. Agree on these costs up front, because they decide your threshold, and your threshold decides whether the project is worth it.

6. Who maintains it after launch?

Models decay. Customer behaviour shifts, products change, a new marketing campaign brings a different crowd. If nobody owns monitoring and retraining, the model will slowly get worse and nobody will notice until something breaks.

7. How will we know it worked?

Pick the business metric before you build. Not AUC, not F1. Something like manual review hours, chargeback rate, or time to resolution. Then plan how you will measure it, ideally with a holdout group that does not get the model.

Why this matters

These questions take about an hour. In my experience roughly one in three proposed projects does not survive them, and that is a good outcome. The ones that survive are clearer, smaller and far more likely to make it to production.

Models are the fun part. Questions are the useful part. Do the useful part first.

Cover: Photo by Volodymyr Hryshchenko on Unsplash.