Plan and read your A/B test
Before the test: how many visitors you need, and how long it will take. After: the probability that version B really does better than version A.
Before the test: how many visitors?
Result
Visitors per version
22,000
That is 44,000 visitors in total.
Enough to see version B go from 2% to 2.4%.
- Test length
- 147 days
More than 60 days: at this traffic, this test will not reach a verdict. Test a bolder change, or a busier page.
After the test: who wins?
Result
Probability that B beats A
94%
No verdict yet
The gap may still be due to chance. At this rate, you need about 22,000 visitors per version to see a lift of 20%.
- Rate of A
- 2%
- Rate of B
- 2.75%
- Gap of B over A
- +37.5%
How to read the result
The number of visitors per version is the volume to gather before looking at the result. Stopping earlier, as soon as B seems to win, is the first cause of false winners.
The probability that B does better is only a verdict from 95% (or at 5% and below: A wins then). In between, nobody knows yet.
The smaller the lift to detect, the more visitors you need: seeing +5% takes about sixteen times more traffic than seeing +20%.
The formula
- Visitors per version: two-sided test on two proportions, 95% threshold, 80% power
- Probability that B wins: Beta posterior on each rate (uniform prior), compared through a normal approximation
- Verdict: not before 100 visitors per version and 10 conversions in total
This is the method of the A/B tests in SellNext: the tool and the app give the same result on the same numbers. The number of visitors is rounded up.
What the tool leaves out
The tool does not see how your test ran: neither its length (count at least 7 days), nor an uneven split between A and B, nor orders attributed to the wrong version. It does not judge a price test: a higher price converts less and can still earn more.
Frequently asked questions
- How long should an A/B test run?
- At least a full week, even with a lot of traffic: Sunday does not buy like Tuesday. Then until you have gathered the calculated number of visitors. After two months without a result, the test will not reach a verdict.
- Can I look at the results along the way?
- Yes, but without deciding. The more often you look, the more likely you are to land on a gap due to chance. Set the volume first, and decide once it is reached.
- What should I test first?
- What every visitor sees: the title and the image at the top of the product page, the text of the buy button. One change at a time, otherwise you will not know which one mattered.
Go from the math to your store
SellNext starts from your product and prepares your brand, your Shopify store and your ads. You get profitability, A/B tests and stock tracking there, calculated on your real numbers.
Build my brandNo credit card to start