A test is only worth running if you can trust the number at the end.

The first time someone lands on your store with a test running, our loader gives them a random identifier and stores it in a first-party cookie. That identifier decides their group.

The decision is a hash, not a coin flip — the identifier and the experiment id are folded together into a number between 0 and 1, and the traffic split decides which band that number falls in. Two consequences follow, and both matter:

• It is stable. The same visitor re-derives the same group every time, so a shopper who returns three weeks later sees the version they saw before.
• It survives a lost cookie. Where the identifier itself is still there but the assignment record is gone, the group is recomputed rather than re-rolled.

Splits hold to a fraction of a percent at any realistic traffic level. You will see 49.8/50.2 rather than exactly half, and that is the hash being honest rather than a bug.

Consent decides whether any of this happens

Before writing anything, the loader asks Shopify’s own Customer Privacy API whether analytics processing is allowed for this visitor. Shopify’s answer already encodes the regional default, so there is nothing for us to infer about geography — and inferring it is how a tool gets it wrong.

Allowed

The visitor enters the test and is counted.

Refused

Nothing is written, nothing is measured, and anything stored earlier is deleted from their device.

No answer

A store with no consent banner has no consent to respect, so the default is to run. Switching the experiment to strict makes silence a refusal instead.

Shopify serves storefront HTML from its own CDN cache, so the page arrives already rendered and the variant is applied in the browser after it lands. That is true of every A/B testing app on the platform, including the native one.

Content edits

Text and attributes on elements you pick with a selector. Applied in place, with no navigation.

Styles & JavaScript

Scoped CSS and JS per group, size-capped and wrapped so a throw in merchant code can never stop the page revealing.

Template changes

An alternate template for a page type your theme already supports. Applied by reloading the same URL with the template named.

Theme changes

A whole variant theme against your live one. Applied by reloading with the theme preview parameter, and the preview bar hidden.

Redirects

Two URLs head to head. The query string is carried across the hop, so ad attribution survives the redirect.

Because all five share one pipeline, results are directly comparable across them — a content test and a theme test produce the same columns, computed the same way.

Applying a variant after the page loads means the shopper could glimpse the original first. The fix is to hide the page until the decision is made — which is itself a cost, paid in the one place a store cannot afford it.

Four rules keep that cost small:

• Never hide without a pending change. If nothing on this page view will differ, there is nothing to hide for.
• Always fail open. If our config never arrives — a network blip, an ad blocker, a slow backend — the page reveals and the shopper sees the control. A test never takes a store down with it.
• Never reveal into a navigation. Theme, template and redirect tests move the visitor. Revealing first would paint the outgoing page for the whole hop.
• Never hide a preview. A preview link already names its group, so hiding would cost a blank second and buy nothing.

The loader is capped at 10 KB gzipped, loaded async, and asserted on every build against the file actually served rather than the file in our repository. Your configuration is mirrored into the browser’s storage, so a returning visitor applies their variant with no network wait at all.

Shopify retired script tags at checkout. The supported replacement is a web pixel — a sandboxed script with no access to the page — so the chain from assignment to order looks like this:

1. The group is decided and written to a first-party cookie.
2. An exposure is recorded, stamped with device, new-or-returning, coarse traffic source, referring site, country and landing path.
3. At checkout the pixel reads the cookie and reports the order against that group.
4. The order id is recorded so a retried pixel event cannot count it twice. Shopify delivers at least once, which means duplicates are normal rather than exceptional.

Everything the dashboard breaks results down by comes from the exposure, not the order — the pixel runs with no referrer, no user agent and no URL, so it could not know any of it. That is why a device breakdown covers the whole funnel rather than just the part still visible at checkout.

A repeat customer counts once

Someone who buys three times over a month is one visitor and three orders. Their loyalty pushes revenue per visitor up for their group, which is the honest reading — counting them as three visitors would flatten one excellent customer into three mediocre ones.

Visitors

People bucketed into the group. Counted once, ever — not once per session and not once per page.

Conversion rate

Share of those visitors who completed at least one order.

Revenue per visitor

All attributed revenue divided by visitors. The headline number, because it moves only when the money moves.

Profit per visitor

The same, less cost of goods read from your Shopify catalogue. Blank rather than guessed where a cost is missing.

Average order value

Attributed revenue divided by attributed orders. Diagnostic: it explains a revenue move that conversion rate alone does not.

Funnel steps

Product view, collection view, add to cart, checkout started, contact, address. Each is a share of visitors, so a person counts once per step.

Any of these can be sliced by device, visitor type, traffic source, referring site, country or landing page — and every slice sums back to the same totals, which is checked rather than assumed.

Bayesian — the default

Reports the probability that each group beats the control, a credible interval around each rate, and the expected loss from choosing a given group and being wrong.

It can be read at any point without invalidating itself, which is what makes it the sane default for a merchant checking in on a Tuesday. “91% to beat control, at most 0.4% downside if wrong” is a sentence you can make a decision from.

Frequentist — for a pre-registered call

Fixes the sample size up front from your own conversion rate, the lift you want to detect, and your confidence level — then withholds the comparison entirely until that sample is reached.

That is deliberate and slightly annoying by design. A p-value’s error rate depends on not peeking, so a tool that shows you one early is quietly breaking the method it is claiming to use.

Either way, the arithmetic is the arithmetic

At a 3% conversion rate, detecting a 20% lift needs roughly 28,000 visitors across two groups. Detecting a 10% lift needs about 106,000. Halving the effect you want to catch roughly quadruples the traffic required.

If your store is under roughly 25,000 monthly visitors, A/B testing is probably not your highest-leverage activity yet. Traffic is.

We would rather say that here than take a subscription from someone who will spend three months finding it out. The full arithmetic is worked out here.

Milliseconds on the page

Client-side application means a brief hide when a change is pending. Small and bounded, but not zero, and we would rather quantify it than claim otherwise.

Cookie-based attribution

A visitor who blocks cookies, or shops from a second device, is a new visitor. Cross-device journeys are not stitched together — no tool on this platform does that without asking shoppers to log in.

Refunds are not subtracted

Revenue is counted at checkout. A later refund or cancellation does not currently reduce it, so a group with a high return rate will read better than it is. Order reconciliation is on the roadmap and this line stays here until it ships.

Consent-blocked visitors are invisible

Where a shopper refuses analytics, they enter no test and appear in no total. In a market with high refusal rates your sample is a subset of your traffic, and it may not be a representative one.

Revenue is noisier than clicks

One large order moves the mean in a way no click can. Revenue tests need more traffic than click tests to reach the same confidence — the price of measuring something that decides anything.

Theme and template tests reload

Both apply by loading a different URL, so those groups pay one extra navigation. Content and style tests do not.

Only orders through Shopify checkout

Attribution rides Shopify's pixel. Draft orders, POS sales and anything placed outside the storefront checkout are not attributed to a group.