Our Attribution Dashboard Was Lying, and a Simple Survey Proved It
One question at signup showed that our most-credited channel was mostly taking credit for word of mouth.
For most of my career I have trusted attribution software more than I should have. It is comforting. You open a dashboard, you see colored bars, and every dollar seems to have a clear story. Last spring I learned that our story was mostly fiction.
The fix was embarrassingly simple: we added one open-text question to our signup form. "How did you first hear about us?" No dropdown, no checkboxes, just a blank field.
What the dashboard told us
According to our tracking, paid search was our best channel by a wide margin. It got credit for about 46 percent of new trials. Organic social barely registered at 4 percent. Podcasts, which we had sponsored a few times, showed close to zero, because people rarely click a link from an episode they heard while driving.
Based on that picture, we had been steadily moving budget into search. We were spending around $22,000 a month there and cutting almost everything else.
What people actually wrote
We collected about 1,100 answers over three months. I read every single one, which took a few evenings and a lot of coffee. Then I grouped them by hand.
The results looked nothing like the dashboard:
- Word of mouth from a colleague or friend: 38 percent
- A podcast: 14 percent
- A post or comment in a professional community: 12 percent
- Searched for a scheduling tool: 19 percent
- Everything else, including "no idea," made up the rest.
Many of the people our software credited to paid search had written things like "my office manager told me about you, then I googled the name." They searched for us by name, clicked our ad because it was at the top, and the ad got the credit.
The experiment that settled the argument
Our head of growth was skeptical, reasonably. Self-reported data is messy. People forget. So we ran a test. For four weeks we turned off our branded search ads, the ones that show up when someone types our company name, while keeping generic search running.
Total signups dropped by about 3 percent, which was within normal weekly noise. We had been paying roughly $6,000 a month to intercept people who were already looking for us.
What we do now
We still use attribution software. It is good at telling us which specific ad performs better than another. But we no longer treat it as the truth about where demand comes from. Every month I read the survey answers and share the most interesting twenty in a team channel.
We moved the saved search budget into two things the dashboard could never see: sponsoring more podcasts that clinic owners actually listen to, and paying for customer referral rewards. Both are hard to measure precisely. Both, according to the people signing up, are working.
I think about a line one customer wrote that I printed and taped above my desk. In the "how did you hear about us" field, she wrote: "My sister would not stop talking about you at Thanksgiving." No pixel will ever track that, and it is probably the best marketing we have.
