A test is only worth running if you can trust the number at the end.

The first time someone lands on your store with a test running, our loader gives them a random identifier and stores it in a first-party cookie. That identifier decides their group.

The decision is a hash, not a coin flip. The identifier and the experiment id are folded together into a number between 0 and 1, and the traffic split decides which band that number falls in. Two consequences follow, and both matter:

• It is stable. The same visitor re-derives the same group every time, so a shopper who returns three weeks later sees the version they saw before.
• It survives a lost cookie. Where the identifier itself is still there but the assignment record is gone, the group is recomputed rather than re-rolled.

Splits hold to a fraction of a percent at any realistic traffic level. You will see 49.8/50.2 rather than exactly half, and that is the hash being honest rather than a bug.

Consent decides whether any of this happens

Before writing anything, the loader asks Shopify’s own Customer Privacy API whether analytics processing is allowed for this visitor. Shopify’s answer already encodes the regional default, so there is nothing for us to infer about geography. Inferring it is how a tool gets it wrong.

Allowed

The visitor enters the test and is counted.

Refused

Nothing is written, nothing is measured, and anything stored earlier is deleted from their device.

No answer

A store with no consent banner has no consent to respect, so the default is to run. Switching the experiment to strict makes silence a refusal instead.

Shopify serves storefront HTML from its own CDN cache, so the page arrives already rendered and the variant is applied in the browser after it lands. That is true of every A/B testing app on the platform, including the native one.

Content edits

Text and attributes on elements you pick with a selector. Applied in place, with no navigation.

Styles & JavaScript

Scoped CSS and JS per group, size-capped and wrapped so a throw in merchant code can never stop the page revealing.

Template changes

An alternate template for a page type your theme already supports. Applied by reloading the same URL with the template named.

Theme changes

A whole variant theme against your live one. Applied by reloading with the theme preview parameter, and the preview bar hidden.

Redirects

Two URLs head to head. The query string is carried across the hop, so ad attribution survives the redirect.

Price tests

A different price per group for the same product. The page shows the group's price and Shopify charges it at checkout.

Offer tests

A percentage off, an amount off, free shipping or no offer, per group. Applied by Shopify at the cart. Nothing on the page changes.

Because all seven share one pipeline, results are directly comparable across them. A content test and a price test produce the same columns, computed the same way.

Price and offer tests change the charge, not only the page

A price shown in the browser is only half a price test. The other half is what Shopify charges, and no script on the page can reach that. So these two run through Shopify Functions, which apply each group’s price or offer at the cart. One rule governs both halves: never show a price we cannot charge.

• The page waits for the cart. A test price is shown only once the visitor’s group is on their cart, where Shopify reads it. If that cannot be confirmed, they see and pay your normal price and are left out of the test.
• Prices go up as well as down, on any plan. On Shopify Plus the price is set directly at checkout. On other plans, a rise lifts the product’s price in Shopify to the highest test price while the test runs, and every other group is discounted from it. The original price is put back when the test is paused or ended.
• Every market is in the test. A shopper abroad gets the same percentage change on the price their market already shows, with that market’s rounding kept.
• Subscriptions are included. You choose whether a subscriber keeps the test price on every renewal or on the first order only.
• “Buy it now” is hidden on tested products, for every group, because it skips the cart that carries the group.
• Crawlers never see a test price. Google Merchant Center compares your product pages to your feed, and a mismatch can disapprove the product.

Prices on the page are found by their value rather than by theme selectors, so there is no theme to edit and no tagging to do. Where a theme draws a price the scan cannot find, you point at it with the element picker.

Check setup, before and while it runs

Every test has a Check setup button on its Preview tab. It loads the real storefront pages the test runs on and runs our shopper code against them. No AI and no guessing: the same inputs give the same findings.

• Whether the app embed renders on those pages.
• Whether each content selector still matches something.
• Whether a template exists and a redirect lands somewhere real.
• Whether the price on each tested product page changes.
• Whether a running test is actually being served.

Each finding is a pass, a warning or a problem, with what to change and a link to where.

One distinction decides almost every combination: a visitor can be in any number of tests that change the page in front of them, but they can only be on one page.

Change in place

Content edits and styles & JS. They apply to the page that already loaded, so any number of them can act on the same view.

Move the visitor

Theme, template and redirect tests. Each loads a different URL, so only one of them can ever act. The second would send the visitor to its own page while the first still counted them as having seen its variant.

Change the charge

Price and offer tests. They move nobody, so they run alongside any of the above. Two price tests can run at once, but not on the same product: their discounts would stack on one line.

What can run together

Starting a test that would clash is refused with the name of the one already running, rather than accepted and quietly corrupted. Two theme tests are also refused when they share a theme. The same rendering measured against two different baselines tells you nothing about either.

Four things worth knowing

• One navigating test per set of pages. Two of them can run at once when they are scoped to different URLs; they are refused only where those scopes overlap. A rolled-out theme test still holds its pages.
• One price test per product. A second price test on a product already under test is refused, with the name of the one running.
• Keep content tests off the same element. Two tests rewriting the same headline are allowed, and both results are then measuring each other.
• Turn the app embed on in every theme a theme test uses. The variant theme is where the visitor lands, and it is that page that records them.

And it costs nothing extra

This is the part worth reading twice, because it is unusual. A visitor who enters four tests in a month is one billed visitor, not four.

Usage is counted per person, per month, across your whole workspace, not per test and not per entry. Someone bucketed into a content test, a styles test and a theme test on the same afternoon is metered once and never counted again until the month rolls over.

Charging per entry would put a price on running more tests. That is the exact behaviour the product exists to encourage, so it is the one thing we refuse to bill for.

Two further consequences of counting people rather than pageviews: a shopper who visits forty times is still one, and a shopper who arrived on a page no test was scoped to is zero. They never entered a test, so they never reach your allowance.

Applying a variant after the page loads means the shopper could glimpse the original first. The fix is to hide the page until the decision is made, which is itself a cost, paid in the one place a store cannot afford it.

Four rules keep that cost small:

• Never hide without a pending change. If nothing on this page view will differ, there is nothing to hide for.
• Always fail open. If our script is blocked, by an ad blocker or a network blip, the page reveals the moment we can tell rather than after a wait, and the shopper sees the control. A test never takes a store down with it.
• Wait long enough to be worth waiting for. The decision needs one round trip, and a hidden page that gives up early shows the control and then moves the visitor. That is not just a flicker: on a theme, template or redirect test only the variant navigates, so giving up early charges one side of the comparison a reload the other never pays. The budget is set from measured response times, not a guess.
• Never reveal into a navigation. Theme, template and redirect tests move the visitor. Revealing first would paint the outgoing page for the whole hop.
• Never hide a preview. A preview link already names its group, so hiding would cost a blank second and buy nothing.

The loader is capped at 10 KB gzipped, loaded async, and measured against the file actually served rather than the file in our repository. Your configuration is mirrored into the browser’s storage, so a returning visitor applies their variant with no network wait at all.

Shopify retired script tags at checkout. The supported replacement is a web pixel, a sandboxed script with no access to the page, so the chain from assignment to order looks like this:

1. The group is decided and written to a first-party cookie.
2. An exposure is recorded, stamped with device, new-or-returning, coarse traffic source, referring site, country and landing path.
3. At checkout the pixel reads the cookie and reports the order against that group.
4. The order id is recorded so a retried pixel event cannot count it twice. Shopify delivers at least once, which means duplicates are normal rather than exceptional.

Everything the dashboard breaks results down by comes from the exposure, not the order. The pixel runs with no referrer, no user agent and no URL, so it could not know any of it. That is why a device breakdown covers the whole funnel rather than just the part still visible at checkout.

And a refund takes it back

Counting revenue at checkout and never revisiting it is the most expensive way a testing tool can be wrong: a variant that sells more and is returned more reads as a clean win, gets rolled out, and the loss appears months later on a different report.

So refunds and cancellations come back through Shopify’s webhooks and are subtracted from the group that earned the sale, proportionally for a partial return, and booked to the date of the order rather than the date of the refund, so the trend chart still shows what each day actually earned.

The order itself stays counted. Someone decided to buy, and that is what conversion rate measures; the return is a second outcome of the same sale, reported separately as a refund rate.

A repeat customer counts once

Someone who buys three times over a month is one visitor and three orders. Their loyalty pushes revenue per visitor up for their group, which is the honest reading. Counting them as three visitors would flatten one excellent customer into three mediocre ones.

Visitors

People bucketed into the group. Counted once, ever: not once per session, and not once per page.

Conversion rate

Share of those visitors who completed at least one order.

Revenue per visitor

All attributed revenue divided by visitors. The headline number, because it moves only when the money moves.

Profit per visitor

The same, less cost of goods read from your Shopify catalogue. Blank rather than guessed where a cost is missing. The headline for price and offer tests, where a lower price can lift revenue and still lose money; without cost data they fall back to revenue per visitor.

Average order value

Attributed revenue divided by attributed orders. Diagnostic: it explains a revenue move that conversion rate alone does not.

Funnel steps

Product view, collection view, add to cart, checkout started, contact, address. Each is a share of visitors, so a person counts once per step.

Any of these can be sliced by device, visitor type, traffic source, referring site, country or landing page, and every slice sums back to the same totals, which is checked rather than assumed.

Bayesian (the default)

Every comparison reports four things:

Probability to beat

How likely this group is genuinely better than the control, rather than ahead by luck.

Credible interval

The range the true value could plausibly be. While two bands overlap, nothing has separated from chance.

Range on the lift

Not just “+3.5%” but the range that lift could be. One spanning zero says “still no result” more plainly than any probability.

Expected loss

What choosing this group costs if it turns out to be worse. A 92% chance of winning means little if the other 8% costs more than the win gains.

All four are computed for conversion rate and revenue per visitor. Revenue is the harder of the two and the one most tools leave as a bare number. It needs the spread of your order values, not just their total, so we record that as orders arrive.

With three or more groups you also get probability to be best. Two variants can each be 90% likely to beat the control while nothing tells you which of the two to ship, and that is the decision actually in front of you.

It can be read at any point without invalidating itself, which is what makes it the sane default for a merchant checking in on a Tuesday. “91% to beat control, at most 0.4% downside if wrong” is a sentence you can make a decision from.

The numbers never move on their own

Every figure here is computed by arithmetic rather than by simulation, so the same data always gives the same answer to the last digit. Tools that sample their way to a probability show one that drifts slightly on every refresh. That is harmless to the decision but corrosive to the trust.

The one exception is probability-to-be-best, which genuinely cannot be solved in closed form. It is simulated from a seed derived from your own counters, so it too is identical on every read.

Frequentist (for a pre-registered call)

Fixes the sample size up front from your own conversion rate, the lift you want to detect, and your confidence level, then withholds the comparison entirely until that sample is reached.

That is deliberate and slightly annoying by design. A p-value’s error rate depends on not peeking, so a tool that shows you one early is quietly breaking the method it is claiming to use.

Once the sample is reached you get the p-value itself, and the verdict is written in those terms rather than translated back into a probability. Choosing this engine should change what the page says, not merely when it says it.

The threshold also tightens with the number of variants. Four variants is four chances to see a difference that isn’t there, so significance is judged at 0.0127 rather than 0.05, and the page says which figure it used.

Either way, the arithmetic is the arithmetic

At a 3% conversion rate, detecting a 20% lift needs roughly 28,000 visitors across two groups. Detecting a 10% lift needs about 106,000. Halving the effect you want to catch roughly quadruples the traffic required.

If your store is under roughly 25,000 monthly visitors, A/B testing is probably not your highest-leverage activity yet. Traffic is.

We would rather say that here than take a subscription from someone who will spend three months finding it out. The full arithmetic is worked out here.

Milliseconds on the page

Client-side application means a brief hide when a change is pending. Small and bounded, but not zero, and we would rather quantify it than claim otherwise.

Cookie-based attribution

A visitor who blocks cookies, or shops from a second device, is a new visitor. Cross-device journeys are not stitched together. No tool on this platform does that without asking shoppers to log in.

Refunds arrive late

Refunds and cancellations are subtracted from the group that earned the sale, booked to the date of the order rather than the date of the return. But a return can take weeks, so a test read the day it ends will always show less refunding than it eventually had.

Consent-blocked visitors are invisible

Where a shopper refuses analytics, they enter no test and appear in no total. In a market with high refusal rates your sample is a subset of your traffic, and it may not be a representative one.

Revenue is noisier than clicks

One large order moves the mean in a way no click can. Revenue tests need more traffic than click tests to reach the same confidence. That is the price of measuring something that decides anything.

Revenue statistics need 30 orders

Below that, a group gets a revenue figure but no interval and no probability. The spread of order values cannot be estimated from a handful of them, and a range drawn from a guess would be read as a measurement.

Outliers are kept unless you say otherwise

Most tools silently drop orders beyond three standard deviations, which is how a large sale leaves a merchant's results unannounced. We count them, show you what they were worth, and let you exclude them, in that order.

Without Plus, a price test shows a discount at checkout

Only Shopify Plus lets an app set a line price directly. On other plans each group's price is charged as a discount from the list price, so the cart shows the list price and a discount line. The product page shows the test price cleanly, and the discount line can still affect checkout completion.

A price rise changes your catalog while it runs

Without Plus, a rise lifts the product's price in Shopify to the highest test price. Other sales channels, POS and product feeds show that price until the test is paused or ended, when it is put back. Uninstalling mid-test leaves it raised.

Theme and template tests reload

Both apply by loading a different URL, so those groups pay one extra navigation. Content and style tests do not.

Only orders through Shopify checkout

Attribution rides Shopify's pixel. Draft orders, POS sales and anything placed outside the storefront checkout are not attributed to a group.