An experiment is one job spread across six screens. Each is its own URL, so you can share one or reload in place, and they are ordered the way the work actually goes: name the groups, say what they do, decide who sees them, choose what to measure, check it, read it.
One draft, six tabs, one save. Nothing is committed as you move between them. Each tab owns a form; none of them owns a save. The Save button lives in the bar above the tabs, along with the name and Start, because those belong to the experiment rather than to any one screen — so Save cannot scroll away from a long panel, and switching tabs never prompts you to discard anything.
Test groups
A group is a name, a colour, a weight, and a flag saying which one is the baseline. Two to six groups. Nothing about what a group does is decided here.
Weight is not a percentage
Weight is a relative number; share is what it works out to, weight divided by the total. They need not add up to 100, and that is the point: three groups at 50 each is an exact third apiece, where 34/33/33 would quietly hand the control an advantage. A group at weight 0 gets no visitors, which is a legitimate way to park an arm without deleting it.
The control is chosen here, configured later
Exactly one group is the control. It is a select below the table rather than a button on each row — it is one decision about the whole experiment, not a property of a row, and as a per-row control it made every row a different width.
The important part: the control is not “the page left alone”. It is configured on Modifications like every other group. A theme test’s control is usually an explicitly chosen theme, and your live theme is not necessarily any group’s. Leaving a group empty is what makes it change nothing — a normal thing for a control to be, and a normal thing for it not to be.
Colour
Per group, and no two groups may share one. It is the same colour in the split bar, on the modification card and in every results chart, so telling the groups apart is its entire job. Greys are not offered: grey reads as disabled and vanishes against the page.
Modifications
The only tab whose contents depend on the kind of test. Every group appears here, control included.
There is one rule, and it is not “fill everything in”: no two groups may be configured identically. Two identical groups are the same experiment twice — the traffic is split and the comparison measures noise. That is why an empty control is fine and two empty groups are not.
Each kind of test has its own guide; the short version of what this tab asks for:
- Content edits
- A list of changes per group, chosen with the visual picker.
- Redirect
- A destination per group. The control’s is the origin.
- Styles & JavaScript
- A CSS and a JS blob per group.
- Theme
- A theme per group, your published one included.
- Template
- One row per comparison, across all groups at once — because a template comparison is a row rather than a per-group setting.
- Offer
- A discount, free shipping, or nothing, per group.
- Price
- The products, chosen once above the groups, then a price per group for each — above or below the current price.
Offer and price tests form a second category, Pricing, and are on the Growth plan and above. The other five are Content and are on every plan.
Theme and template tests re-read your store when you open this tab if the cached list is more than five minutes old, because merchants add themes and duplicate templates while a test is being built and none of it reaches us until something asks.
Targeting
Who is eligible. Empty means no restriction — the absence of a rule never excludes anyone.
Common
- Device
- All, desktop or mobile.
- Visitor
- All, new or returning.
- Source
- Direct, email, referral, affiliate, organic and paid social, organic and paid search, paid shopping, SMS, and the named platforms.
- Countries
- Searchable over the full ISO list. This asks where a visitor is, not where you ship, so it cannot be trimmed to your markets.
Country is read from Shopify’s own localisation rather than from an IP lookup, which would put a third-party round trip on every shopper’s critical path. That makes it spoofable, and that is fine: targeting decides who is in a test, not who may buy. A visitor we cannot place is included, never dropped — silently excluding them would make the live split stop matching the configured one with nothing anywhere saying why.
Where it runs
URL patterns, matched against the path, with * wildcards. Empty means every page, which is right for a styles or content test and catastrophic for a redirect — so a redirect test with no pattern is refused rather than run.
Custom and Advanced
UTM parameters, cookie and storage predicates, and a JavaScript predicate evaluated in the loader. Reach for these last; a test nobody can explain the audience of is a test nobody can act on.
Measurement
Named for what it decides rather than what it resembles. “Analytics” promises reports and this tab produces none — it decides how the result will be judged.
The framework, and why it locks
Bayesian by default, frequentist if you prefer. They disagree about when you are allowed to look, which is why the choice has to be made before data collection begins: switching mid-flight invalidates the answer. Changing it after the test has started clears what was measured and restarts the run, and the app says so before you save rather than after.
Minimum detectable effect
The smallest improvement worth detecting, as a relative lift. Both frameworks use it — only the frequentist one gates a verdict on reaching the sample size, but the “how long until this can answer” estimate is arithmetic either way. A smaller number finds subtler wins and needs dramatically more traffic.
Metrics
Up to seven on the front page, one of them primary — the one a winner is called on. The catalogue lists every metric a store might want, including ones we cannot compute yet; those carry a line saying exactly what is missing and are disabled. Hiding them would make the catalogue look complete when it is not.
Revenue per visitor is the default primary for content tests, because conversion rate alone can rise while revenue falls. Pricing tests default to profit per visitor instead: a lower price reliably lifts revenue and can destroy margin at the same time, and price is the one kind of test where that happens as a matter of course.
Metrics stay editable after launch, unlike the framework. They change how the same collected events are read, not what is collected.
Preview
Check each group before launch without being bucketed into it. A preview link pins your group for the session, bypasses targeting — checking a mobile-only test on a laptop should still show it — and never writes to the assignment cookie. That last part is what keeps your own walkthrough out of your results.
Preview reads what is saved, not what is on screen. An unsaved modification previews as nothing, so each row says what its link will actually do — “Theme: Pitch”, “No change”. Two links that look different and resolve to the same page is the most confusing failure this screen has, and reading the rows is the only way to see it without opening both.
“Is this reaching the store?” asks the config endpoint the same question the loader asks, and reports whether the experiment is served and how many groups carry a payload. It checks our half only: if it passes and the storefront still does nothing, the cause is the app embed or a cached loader.
Check setup checks the other half. It loads the real storefront pages the test runs on and runs our shopper code against them — no AI, so the same store gives the same answer. It reports:
- whether the app embed renders on those pages;
- whether each content selector matches something;
- whether a template exists and a redirect lands somewhere real;
- whether the price on each tested product page changes;
- whether a running test is actually being served.
Each finding is a pass, a warning or a problem, with what to change and a link to where. The launch banner links to it too.
Results
Visitors, orders and revenue are always shown, because rates are unreadable without the denominators they are built from. Around them: the metrics you chose, comparison bars with intervals, the cumulative series, a breakdown by audience, and per-product numbers.
The page refuses to call a winner early. On the frequentist framework it will not read a verdict before the pre-computed sample size; on either, it tells you how far off you are rather than showing a number that will change.
Roll-out keeps the test running at 100%. That is deliberate: the loader is what applies the winning change, so ending the test would switch off the thing that just won. It stays live until you have baked the change into your store and ended it — at which point we stop being load-bearing in your storefront, which is where you want to get to.
Starting
Start is refused while anything would make the result wrong or empty, and the reason is always a sentence rather than a disabled button. The two that catch nearly everybody: the web pixel is not registered, so visitors would be counted and no order ever attributed; or the app embed is off in a theme the test needs, so the change reaches nobody.
Everything softer is a warning rather than a block, because the cost of a false block is someone stuck behind a button nobody can explain.
Your plan decides whether a test can start at all:
- When the trial ends without a plan, running tests pause until you pick one.
- When a paid plan is cancelled or a payment fails, running tests pause a day after it ends.
- Over the month’s tested visitors, new tests cannot start. Running tests carry on.
None of these blocks building a test, reading results, pausing or ending, and none deletes anything measured.