An experiment is one job spread across six screens, ordered the way the work goes: name the groups, say what they do, decide who sees them, choose what to measure, check it, read it. The figures below are live — they run the same bucketing and targeting code your storefront does.
One draft, six tabs, one save. Each tab is its own URL, so you can share one or reload in place. Each owns a form; none owns a save. Save lives in the bar above the tabs, with the name and Start, because those belong to the experiment rather than to any one screen. It cannot scroll away from a long panel, and switching tabs never asks you to discard anything.
Saving is where the tabs differ. On a draft, change anything. Once a test is running, some edits keep the results and some start a new run, because the numbers before and after would describe two different tests.
- Two to six groups
- Name, colour, weight
- Which one is the control
Saves. The results carry on.
Labels. Nothing measured changes.
↳ Labels never restart. What shoppers see does.
In its default state the figure shows a running test, on Test groups, with a rename: it saves and the results carry on. Switch the edit to Remove a group and it restarts. Switch the test to Rolled out and the same edit is refused. The rest of this post walks the tabs in order.
Test groups
A group is a name, a colour, a weight, and whether it is the control. Two to six groups. Nothing about what a group does is decided here.
Weight is not a percentage
Weight is a relative number; share is what it works out to, weight divided by the total. They need not add up to 100, and that is the point: three groups at 1 each is an exact third apiece, where 34/33/33 would quietly hand the control an advantage. A group at weight 0 gets no visitors, which is a legitimate way to park an arm without deleting it.
A weight decides where a visitor lands. The visitor id is hashed with the experiment id to a point between 0 and 1, and the line is cut into lengths by weight.
- Control52.0% of 50.0%
- New headline48.0% of 50.0%
Control, on every page, for as long as the test runs.
The group is arithmetic on the visitor id and the experiment id, not a coin toss. Lose the assignment cookie and the same visitor re-derives the same group.
↳ Same visitor, same group, every time
The default visitor, v_8k2m4q1z, hashes to 0.373 and lands in the control on a 50/50 split. Press “Same visitor, next page” as often as you like: the answer does not move. Over 2,000 made-up visitors the split comes out at 52.0% and 48.0% — close to half, never exactly, which is what random looks like.
Now switch to 1/1/1. The same point, 0.373, falls in Variant A. That is why a weight change on a running test does not reshuffle anyone: the group is written to a cookie the first time, and the hash is only asked again if that cookie is lost. The results page records when the split changed, so its split check does not flag your own edit as a problem.
Every visitor gets one group, and keeps it.
The control is chosen here, configured later
Exactly one group is the control, chosen with a select below the table. It is one decision about the whole experiment, not a property of a row.
The control is not “the page left alone”. It is configured on Modifications like every other group. A theme test’s control is usually an explicitly chosen theme. Leaving a group empty is what makes it change nothing — a normal thing for a control to be, and a normal thing for it not to be.
Colour
Per group, and no two groups may share one. It is the same colour in the split bar, on the modification card and in every results chart, so telling the groups apart is its entire job. Greys are not offered: grey reads as disabled.
Modifications
The only tab whose contents depend on the kind of test. Every group appears here, control included. One rule: no two groups may be configured identically. Two identical groups split the traffic and measure noise. That is why an empty control is fine and two empty groups are not.
- Content edits
- A list of changes per group, chosen with the visual picker.
- Redirect
- A destination per group. The control’s is the origin.
- Styles & JavaScript
- A CSS and a JS blob per group.
- Theme
- A theme per group, your published one included.
- Template
- One row per comparison, across all groups at once.
- Offer
- A discount, free shipping, or nothing, per group.
- Price
- The products, chosen once above the groups, then a price per group for each — above or below the current price.
Offer and price tests are the Pricing category: on the Growth plan and above, and in the trial. Both are in beta, so check the first few orders after starting one. The other five are Content tests, on every plan.
Theme and template tests need a Shopify theme, so they are unavailable on a headless storefront. Content, redirect, styles, offer and price tests work there through the loader script or the SDK.
Editing a running test restarts it
On a running or paused test, saving a change to what a group shows or charges clears its results and starts a new run, exactly as Clear results does. Visitors before and after the edit saw different things, and one total over both would describe neither.
- Restarts: any change to what a group does (content, CSS or JS, theme, template, redirect URL, prices, offer), redirect frequency, a price test’s products, and removing a group.
- Does not: renaming, recolouring, changing weights, adding a group, and anything on Targeting.
- Refused: an edit to a running test that would leave it unable to start — groups that no longer differ, say.
- Never restarted: drafts, ended tests and roll-outs. Their results are the record, so their group set and framework are fixed.
The app says so before you save. Visitors keep their groups either way.
Theme and template tests re-read your store when you open this tab if the cached list is more than five minutes old, because merchants add themes and duplicate templates while a test is being built.
Targeting
Who is eligible. Two cards: Where it runs and Audience. Anything left unrestricted includes everyone, and every rule only narrows.
Where it runs
URL patterns, one per line, matched against the path, with * wildcards. Empty means every page — right for a styles or content test, and catastrophic for a redirect. A redirect test takes its pattern from the control’s origin URL on Modifications instead, and one with no origin is refused rather than run.
Audience
- Device type
- All, desktop or mobile.
- Visitor type
- All, new or returning.
- Traffic source
- Direct, Referral, Google, Facebook, Instagram, TikTok, Twitter and YouTube, detected from the referring site with no setup. Plus any
utm_sourcevalue you add yourself, matched exactly: link with?utm_source=newsletter, then addnewsletter. - Storefront country
- The country Shopify localised your store to for this visit — from the shopper’s country selector, a market domain or subfolder, or Shopify’s own geolocation. Not an IP lookup.
- Markets
- Shopify markets your visitors have been seen in, or any market handle you type, such as
north-america. - Storefront language
- The language Shopify showed the store in, not the browser’s setting. A bare language matches its regional forms:
ptcoverspt-BR;pt-BRcovers only itself.
All of it is evaluated in the loader, in the shopper’s browser, from what Shopify already put on the page. An IP lookup would add a third-party round trip to every shopper’s first page. That makes country spoofable, and that is fine: targeting decides who is in a test, not who may buy.
Audience (the rules)
One visit (what the page knows)
| Rule | Wants | Visit has | Result |
|---|---|---|---|
| Device type | Mobile | Mobile | Matches |
| Visitor type | Anyone | New | No rule |
| Traffic source | Anyone | No rule | |
| Storefront country | United Kingdom, Germany | Can't tell | Unknown, included |
| Markets | Anyone | Europe | No rule |
| Storefront language | Portuguese | Portuguese (Brazil) | Matches |
In the test. This visitor is bucketed.
Included even though we can't read their storefront country. Dropping them would make the live split stop matching the one you set.
↳ A visitor we can't place is included
The default audience is mobile shoppers, in the United Kingdom or Germany, seeing the store in Portuguese. The visit is a new mobile shopper from Instagram whose country the page cannot read, browsing in pt-BR. They are in: the country rule cannot be judged, so it includes them, and pt covers pt-BR. Pick Germany as their country and they stay in. Pick Brazil and they are out.
That inclusion is deliberate. A visitor dropped for an unreadable value would make the live split stop matching the configured one, with nothing anywhere saying why. On a headless storefront, where there is no theme to publish the market, market rules include everyone.
Measurement
Named for what it decides. “Analytics” promises reports and this tab produces none — it decides how the result will be judged.
The framework, and why changing it restarts
Bayesian by default, frequentist if you prefer. They disagree about when you are allowed to look, so data gathered under one cannot honestly be read under the other. Switching on a running or paused test clears what was measured and starts a new run, and the app says so before you save. On an ended or rolled-out test the switch is refused, because those results are the record.
Minimum detectable effect
The smallest improvement worth detecting, as a relative lift. 10% by default. The frequentist framework will not call a result before reaching the sample size it implies. Both use it to estimate how long the test needs. A smaller number finds subtler wins and needs dramatically more traffic.
Metrics
Up to nine on the front page, one of them primary — the one a winner is called on. On a price test charged through a discount, the two discount metrics would count the test’s own discount, so they are disabled with that reason rather than hidden.
Revenue per visitor is the default primary for content tests, because conversion rate alone can rise while revenue falls. Pricing tests default to profit per visitor: a lower price reliably lifts revenue and can destroy margin at the same time.
Metrics stay editable after launch, unlike the framework. They change how the same collected events are read, not what is collected.
Preview
Check each group before launch without being bucketed into it.
- Signed. The dashboard signs each link; it works for 72 hours. A group pin without a valid signature is ignored.
- Bypasses targeting. A mobile-only test still shows on your laptop.
- Honours URL patterns. Those say where the change belongs, so a preview of a product-page test does nothing on the home page.
- Never writes the assignment cookie. That is what keeps your own walkthrough out of your results.
Preview reads what is saved, not what is on screen. An unsaved modification previews as nothing, so each row says what its link will actually do — “Theme: Pitch”, “No change”.
“Is this reaching the store?” asks the config endpoint the same question the loader asks. It checks our half only.
Check setup checks the other half. It loads the real storefront pages the test runs on and runs our shopper code against them — no AI, so the same store gives the same answer. Among what it reports:
- whether the loader is on those pages, and whether it is installed twice;
- page-builder layouts that the loader does not reach;
- whether each content selector matches something;
- CSS or JavaScript that will not parse;
- a missing template, a redirect that loops, 404s or leaves your store;
- whether the price on each tested product page changes;
- “Buy it now” hidden on tested products, and whether the pricing extension is installed;
- whether a running test is actually being served.
Each finding is a pass, a warning or a problem, with what to change and a link to where.
Results
Visitors, orders and revenue are always shown, because rates are unreadable without the denominators they are built from. Around them: the metrics you chose, comparison bars with intervals, the cumulative series, a breakdown by audience, and per-product numbers.
The page refuses to call a winner early:
- No comparison is shown under 25 orders across the test.
- The bar a variant must clear rises with the number of variants: 95%with one, 97.47% with two, 98.30% with three. The most extreme of several readings beats a do-nothing control by luck far more often than one reading does.
- On the frequentist framework, nothing is called before every group reaches the pre-computed sample size.
Roll-out keeps the test running at 100%. The loader is what applies the winning change, so ending the test would switch off the thing that just won. It stays live until you have baked the change into your store and ended it.
To see a finished results page before you have one, load a sample test from Settings. There is one for each of the seven types, read-only, with invented numbers.
Starting
Start is refused while anything would make the result wrong or empty, and the reason is always a sentence rather than a disabled button. The two that catch nearly everybody: the web pixel is not registered, so no order would ever be attributed; or the app embed is off in a theme the test needs, so the change reaches nobody. Everything softer is a warning.
Your plan decides whether a test can start, and how long one runs:
- Trial ends
- New tests cannot start. Running tests pause at the trial’s end.
- Paid plan cancelled
- It runs to the end of the period you paid for. Running tests pause one day after that.
- Payment fails
- New tests cannot start, at once. Running tests pause one day later.
- Over the month’s tested visitors
- Nothing new starts. Running tests carry on.
None of these blocks building a test, reading results, pausing or ending, and none deletes anything measured.