Our Landing Page Test Lifted Signups 31 Percent. Then It Didn't

Last year I ran an A/B test on the signup page of a B2B scheduling tool I was advising. The variant had a shorter headline, one less form field, and a customer logo strip. After nine days, it showed a 31 percent lift in signups. The founder posted the result in the company channel with three fire emojis. I was quietly thrilled.

Six weeks after we shipped the variant to everyone, the signup rate was almost exactly where it had been before the test.

What actually happened

When I went back through the data, the problems were textbook. I had simply ignored them because the result was exciting.

• The sample was small. Each arm got roughly 1,100 visitors. The control converted at 4.2 percent and the variant at 5.5 percent. That looks impressive, but it is a difference of about 14 signups.
• I peeked. I checked results every morning and stopped the test the first day it crossed my significance threshold. Checking repeatedly and stopping on a good day inflates the chance of a false positive considerably.
• Traffic was unusual. During those nine days, a partner newsletter sent a burst of well-qualified visitors. Their timing happened to favour the variant.

Regression to the mean

When a measurement is partly noise, extreme results tend to be followed by more ordinary ones. A test that stops at its luckiest moment almost guarantees disappointment afterwards. Our 31 percent was mostly a lucky streak.

What I do differently now

Decide the sample size before starting

I use a standard power calculation with a realistic minimum effect I care about. For this client, detecting a 15 percent relative lift on a 4 percent baseline required around 15,000 visitors per arm. That would have meant running the test for about four months, which told us the test was not worth running at all at that traffic level.

Do not stop early

The duration is fixed in advance and written down. I still look at dashboards, but only to check nothing is broken.

Test bigger changes when traffic is low

Low traffic startups cannot detect small effects. Instead of tweaking a headline, test a completely different offer, such as a free trial against a live demo. Big swings produce effects you can actually measure.

Track one downstream metric

Signups are cheap. I now also compare activation, meaning users who completed a meaningful first action. If signups rise but activation does not, the variant may just be attracting less committed people.

The honest summary

The variant was probably not worse. It may even have been slightly better. But it was not a 31 percent improvement, and announcing it as one cost us credibility when the numbers settled. A boring, correctly run test beats an exciting, wrong one every time.

Cover: Photo by Campaign Creators on Unsplash.