The most common way an A/B test fails on Shopify is not a bad hypothesis or a broken script. It is a test that never had enough traffic to answer anything, stopped after a fortnight, and got read as though it had.
The number is knowable in advance, and it is worth knowing before you build the variant rather than after. The figures below are live and use the same functions as the Zinx Signal dashboard, so change them and the numbers are the ones the app would give you.
The arithmetic
Sample size depends on three things: your current conversion rate, the smallest improvement you want to be able to detect, and how confident you want to be. Nothing about the tool changes them.
We plan at 95% confidence and 80% power, the usual settings. That leaves two dials you control.
↳ the curve is steepest exactly where small tests live
At a 3% conversion rate, catching a 20% lift takes 13,911 visitors per group. Catching half of that, 10%, takes 53,208: 3.8 times as many. Try a 1% conversion rate and every number roughly triples again.
“Let’s also see if it moves the needle a little” is the most expensive sentence in testing.
What that means for a real store
Plenty of Shopify stores see under 10,000 visitors a month. Put your own numbers in. The defaults are a 3% conversion rate and a 10% lift, which is the app’s default minimum detectable effect.
Not conclusive at this rate.
The dashboard says exactly this past three months. Test something bigger, scope to a busier page, or drop a group.
↳ the verdict uses the dashboard's own 90-day cut-off
At 10,000 visitors a month, all of them entering the test:
- A 10% lift needs 106,416 visitors in total: nearly eleven months (10.6).
- A 20% lift needs 27,822: about three months (2.8).
That is not a limitation of any particular app. It is the arithmetic, and every tool in this category is subject to it, including Shopify’s own native testing.
Under roughly 25,000 visitors a month, testing is probably not your highest-leverage work yet. Traffic is.
We would rather say that on a public page than take a subscription from someone who will spend three months learning it.
Four ways to need less traffic
1. Test bigger changes
A new page layout moves conversion more than a button colour, and needs about a quarter of the traffic to prove it. Small tests are not cheaper. They are dramatically more expensive.
2. Test fewer groups
Every group splits the same traffic, and every extra variant is another chance to see a difference that isn’t there. So the bar a variant must clear to be called a winner goes up with the number of variants:
- One variant (A/B): 95%.
- Two variants (A/B/C): 97.47%.
- Three variants (A/B/C/D): 98.30%.
More groups, and each one needs more visitors to clear a higher bar. At a 3% conversion rate and a 10% lift, an A/B/C test needs about 1.8 times the total traffic of an A/B test. Switch Figure 02 to A/B/C to see it. Most three-arm tests exist because nobody could choose.
3. Scope to the page that matters
Testing one landing page means only its visitors enter the test: a smaller denominator, but a cleaner one. A sitewide test that changes something only 5% of visitors ever see spends 95% of its sample on people the change could not have affected.
4. Measure revenue, but expect it to be slower
Revenue per visitor is noisier than conversion rate, so it is not a free win, but it is the number you actually act on. The app reports no revenue statistics at all until a group has 30 orders, because below that the spread of order values can’t be estimated honestly.
Why you can’t just stop when it looks good
Knowing the number is half of it. The other half is not reading the result before you get there. Early days swing, and they swing through the winner bar.
It looks like a winner. It isn’t.
18 days before the planned sample. Both groups are the same page; this is noise clearing the bar.
↳ both groups are the same page
Both groups in that figure convert at a true 3%. There is nothing to find. On day 10 the variant still reads 3.52% against 2.82% and is 98% likely to be better, and it clears the 95% bar on seven of the first fourteen days. By day 28, when the planned sample of 13,911 a group is reached, it is 22%. By day 35 it is 49%: a coin toss, which is the truth.
A test stopped the first time it looks good will usually look good.
What we do about it
The dashboard does four things to keep a result from being read before it means anything:
- A date, not a shrug. It computes the sample needed from your control’s own conversion rate and the test’s minimum detectable effect, then shows a “time to a conclusive read” paced off the smallest group. Past three months it says “not conclusive at this rate” instead.
- No comparison under 25 orders. Fewer is noise, and a number on screen invites someone to act on it.
- The frequentist engine waits for the sample. The comparison is withheld until every group reaches the planned size, because a method whose error rate depends on not peeking should not invite you to peek.
- The Bayesian engine shows its working. It reports the probability as it stands, and only names a winner past the bar above. On day 10 of Figure 03 it would show that 98%, beside an estimate saying the test needs about three weeks more.
None of that makes a small store’s test faster. It just stops the result being read before it means anything.