A headline that lifts add-to-cart rate by 12% and leaves revenue flat has not helped anyone. It has moved shoppers one step further down a funnel they were going to leave anyway, and it will show up in your testing tool as a win.
This happens constantly, and it has a boring cause: upper-funnel metrics are easier to move. Urgency copy, a bigger button, a discount badge: all of them reliably increase intention and unreliably increase purchases.
The figures below are live and run on the same statistics as the results page. Change them.
Revenue per visitor is the honest headline
Revenue per visitor is the revenue attributed to a group divided by the visitors bucketed into it. It is conversion rate times order value in one number, so it can’t be gamed by moving people around inside the funnel. A variant that converts better at a smaller basket can still be a loss.
Revenue per visitor is conversion × order value. 20,000 visitors a group.
Variant wins on conversion rate.
98.8% likely to be better, past the 95% bar.
↳ revenue per visitor is computed, never typed in
The urgency banner lifts conversion from 3.0% to 3.4%, and on conversion rate alone it wins: 98.9% likely to be better. It also cuts the average order from £80 to £68. Revenue per visitor falls from £2.40 to £2.31, and the chance it is better is 32%. No winner.
Order value gets no verdict at all. The app shows the difference, but there is no statistical model behind it, so it makes no call. Only conversion rate and revenue per visitor are modelled, and the verdict follows whichever one the test is judged on.
The same test is a win, a loss and a shrug, depending on which number you ask.
What “revenue” means here
- Net: product plus shipping, less discounts. Tax and duties are left out, because they vary with where the shopper lives, not with your variant.
- Refunds come off, booked to the date of the original order, so a refund in March reduces February’s numbers.
- Some orders never count: point-of-sale orders, and draft orders completed in the admin. Nobody saw a storefront variant.
Profit per visitor, and its limit
A variant that sells more of your worst-margin product is a revenue win and a business loss. Profit per visitor catches that, and it is built: available on every paid plan and the trial.
- It needs a cost per item set in Shopify.
- An order counts toward profit only when every item in it has a cost. An uncosted product would otherwise read as pure margin.
What it costs: variance
Revenue is noisier than clicks. One shopper buying a £1,500 order moves the mean; nobody clicks fifteen hundred times. So a revenue test needs more traffic than a click test to reach the same confidence.
History: average £64, spread £30. 4,000 visitors. Flag line £154 (3σ), checked once a group has 30 orders.
Flagged, and still counted.
Past £154, so the app marks it. It stays in the numbers until you choose to exclude flagged orders.
↳ flagged is not the same as removed
A group of 120 orders averaging £64 takes one £1,500 order. The average jumps to £75.87 and revenue per visitor rises 19.5%, from one shopper. It sits past three standard deviations, so the app flags it. It stays in the numbers unless you choose to exclude flagged orders. Switch to 20 orders and nothing is flagged at all: under 30 there is no history to be unusual against.
The answer is not to retreat to click-through rate. It is to report the uncertainty honestly:
- A winner is named only when the chance of beating the control clears95%. With two variants the bar is 97.47%, with three 98.30%.
- No comparison is shown at all until the test has 25 orders.
- Revenue statistics need 30 orders in a group. Below that the page shows a blank, not a guess.
- Revenue per visitor shows a range on the lift, not just a point.
What it costs: attribution
Counting clicks needs nothing but the page. Counting revenue means following a shopper into checkout, which on Shopify is a place apps can’t reach directly. Script tags were retired there, and the modern route is a sandboxed web pixel with no DOM access.
So the group has to travel with the shopper, and it travels two ways. When a visitor is bucketed, the group and the visitor id go into a first-party cookie and onto the cart as cart attributes: _zinx_grp and _zinx_vid.
- 1 · holdsCookie_zinx_grp and _zinx_vid, set on the storefront when the visitor is bucketed.
- 2 · holdsCart attributesThe same two values, written onto the cart so they travel into checkout.
- 3 · holdsWeb pixelReads the checkout's attributes first.
- 4 · not neededReconcile sweepEvery 6 hours, reads the same attributes off the order through Shopify's Admin API.
Attributed by the pixel, at checkout.
Read from the checkout's cart attributes.
↳ two paths, one source of truth: the cart
The web pixel reads the checkout’s cart attributes first and the cookie only as a fallback. That is what lets a headless or cross-domain checkout be attributed at all: the cookie only exists on the domain that set it. Every six hours a reconcile sweep reads the same attributes off orders through Shopify’s Admin API, and recovers the orders the pixel missed.
Every link is still a place attribution can be lost. Being specific about which ones is more useful than claiming they never break.
One visitor, many orders
A repeat customer is counted once in the denominator. What happens on top depends on the metric:
- Conversion rate counts people: three orders from one shopper is one converted visitor.
- Revenue per visitor counts every order. If somebody buys three times, their loyalty should push revenue up for their group, not be flattened into three mediocre visitors.
Visitors who decline tracking consent are not in a test at all. Shopify blocks the pixel for them, so counting them would add visitors whose orders could never be seen.
The tradeoff, stated
Measuring revenue means slower tests and a harder attribution problem than measuring clicks. We think that is the right trade, because the alternative is a fast, confident answer to a question that does not decide anything.
But it is a trade, and a tool that presents it as a free upgrade is selling you something.