Shopify serves storefront HTML from its own CDN cache. That single fact decides the architecture of every A/B testing app on the platform: the page arrives already rendered, so the variant has to be applied in the browser, after it lands.
Which leaves a choice. Either the shopper sees the original for a moment before the variant applies, which is the flicker, or the page is hidden until the decision is made. On a theme there is no third option. The only way out is to render the variant on the server, which a headless store can now do (more on that at the end).
The figures below are live. Play with them; they show what a shopper actually sees.
Why the flicker is worse than it looks
A visible flash is not only ugly. It biases the result. The visitors who see it longest are the ones on slow connections, and they are not a random sample.
Free delivery over £50
The shopper saw the control first.
For 700 ms, then the headline changed under them. The slower the connection, the longer the flash — so one kind of visitor is shown both versions.
↳ slow connections pay for both choices longest
With the overlay off, a shopper on an average mobile connection reads the control’s headline for 700 ms before it changes under them. On a slow connection, 1,600 ms. Those visitors skew mobile, skew older devices, and convert differently. A test that degrades for one segment is measuring that segment’s patience alongside your headline.
Why hiding the page is also worse than it looks
Switch the overlay on and the flash is gone, replaced by the same 700 ms of nothing. Every millisecond of that is time a shopper spends looking at a blank page that makes someone’s living. Done carelessly it is a conversion cost you are paying to measure conversion.
So the overlay has to earn every millisecond, on every page view, or not run.
What each page load actually does
The overlay is not one behaviour. It depends on whether there is anything to change, whether we already know the answer, and whether anything went wrong.
Page hidden from 0 to 620 ms, Variant showing from 620 to 3000 ms; Overlay added in <head>, before anything paints at 0 ms; Loader downloaded and running at 250 ms; Config arrives over the network at 600 ms; Variant applied, page revealed at 620 ms; Fixed 2,500 ms ceiling at 2500 ms
- 0 msOverlay added in <head>, before anything paints
- 250 msLoader downloaded and running
- 600 msConfig arrives over the network
- 620 msVariant applied, page revealed
What the shopper sees
A short blank, then the variant. Never the control.
↳ illustrative timings; the order of events is real
In this illustration a first visit is hidden for 620 ms and a repeat visit for 140, because the second one never waits on the network. A store with nothing running hides nothing at all. The only scenario that reaches the full 2,500 ms is a backend that never answers, and there the page is revealed untested.
The rules
1. Never hide without a pending change
If nothing is going to change on this page view, there is nothing to hide for.
- No test running. The loader marks the store idle beside its cached config, and the next page view skips the overlay before the script has even downloaded.
- A preview. The group is already decided by the signed link, so hiding costs a blank second and buys nothing.
- The Shopify theme editor. Hiding there would blank the pane the merchant is working in, every time it reloads.
2. Fail open, always
The page reveals the moment a decision is made, not when a timer runs out. The timer is a ceiling for loads that never decide:
- A fixed 2,500 ms. Not a setting. If the config never arrives, the page reveals and the shopper sees the store untested.
- A blocked script reveals at once. When an ad blocker or a network failure kills the loader, an error listener catches it and reveals immediately rather than waiting the timeout out.
- Declined consent reveals at once. Where a test would bucket the visitor, the loader asks Shopify’s privacy API, waiting at most about 500 ms for it to load. A visitor who has refused is revealed straight away and never enters the test.
3. Hide <html>, not just <body>
The script runs in <head>, before <body> exists. A rule scoped to body covers nothing until the parser reaches it, so a theme that paints its background on <html> gets a frame or two on screen regardless.
4. Never reveal into a navigation
Theme, template and redirect tests move the visitor to a different URL. A redirect keeps the page hidden until the new page loads. Revealing before then paints the outgoing page for the whole navigation, which is exactly the flash the overlay exists to prevent, arriving at the end instead of the start.
The budget
Our loader is capped at 10 KB gzipped, measured against the file actually served rather than the file in the repository. It had been broken twice before anything measured it, and it is now about 9.7 KB. That is close enough to the limit that anything new goes in a separate file.
The whole footprint is the loader.
Async, cached, and the only thing an ordinary page view pays for.
↳ the loader is the only file every page pays for
An ordinary page view downloads the loader and nothing else. Price rewriting (prices.js), the preview toolbar (preview.js) and the script that tells a store’s analytics apps about test groups (integrations.js, only for stores that switch it on) are fetched only where they apply. The element picker (picker.js) only ever loads for a merchant building a test, never for a shopper.
Configuration is mirrored to the browser’s own storage, so a returning visitor applies their variant with no network wait at all. The round trip only happens on a first visit, and it refreshes in the background for the next one.
The exception: headless stores
A headless store, built on Shopify’s Storefront API instead of a theme, renders its own pages. There our SDK can choose the variant on the server, so the page arrives already rendered as the variant: no overlay and no flicker. Headless stores that prefer no code can still use the loader script, with the same rules as above.
The honest summary
Client-side testing on Shopify has a real cost, and it is measured in milliseconds on the critical path of somebody else’s storefront. The right response is to make that cost small and to be specific about it, not to claim it is zero.