Why I Plot Everything Before I Trust a Single Summary Statistic

Early in my career, a manager asked me which of two olive oil suppliers was more reliable. I computed the average delivery delay for each over the previous year. Supplier A averaged 2.1 days late. Supplier B averaged 2.0 days late. I said they were basically the same.

I was wrong, and I only found out because a logistics coordinator named Marta laughed when I told her.

The averages were the same, the deliveries were not

When I finally plotted the delays as two histograms, the picture was obvious. Supplier A was consistently about two days late. Nearly every delivery arrived between one and three days after the promised date. Planners could simply add two days and be fine.

Supplier B was usually on time or early, but roughly one delivery in eight arrived more than a week late. Those late deliveries pulled the average up to the same value as Supplier A. For a production line that runs on tight schedules, Supplier B was far worse. A predictable delay is a planning problem. An unpredictable one is a crisis.

Marta had known this for years from experience. My average had erased it.

My routine now

Since then, I follow a simple routine before I report any summary statistic. It takes a few minutes and has saved me many times.

• Histogram first. Before an average, I look at the shape of the distribution. Is it one hump, two humps, a long tail?
• Look at the extremes. I sort and read the ten largest and ten smallest values. Data errors and interesting events both live there.
• Plot over time. A stable average can hide a trend that reversed halfway through the period.
• Split by the obvious groups. By region, by product, by weekday. Averages love to hide mixtures.

For the suppliers, the fix in reporting was to show the median delay alongside the 90th percentile delay. Supplier A was 2 and 3 days. Supplier B was 0 and 9 days. Those two numbers told the planners everything they needed.

A more recent example

Last spring, someone on our team reported that average customer complaint resolution time had improved from 4.2 days to 3.1 days. Great news. I asked for a plot by week.

The improvement came entirely from a single week in which a batch of 300 trivial complaints, all about a misprinted label, were closed in one afternoon. Excluding that week, resolution time had actually crept up slightly. The team was not doing worse, but they were not doing better either.

Pictures before numbers

I sometimes get teased for opening every analysis with a page of plots, many of them ugly and never shown to anyone. But the plots are not for the audience. They are for me. They are how I find out what the data actually looks like before I decide which single number is honest enough to stand in for it.