A theme test sends each group to a different theme. It is the biggest change you can test and the one that needs the most traffic to call, because you have changed everything at once: if the new theme wins you know it won, and not which part of it did.
That is a reasonable trade when the question genuinely is “should we launch this redesign”. It is a poor one when you want to know why — for that, break it into template or content tests and get answers you can reuse.
Set the themes up first
Duplicate your live theme in Shopify and change what you want to test. Starting from a duplicate rather than a fresh theme keeps everything else constant, which is the whole point of a comparison.
The app embed must be on in every theme
This is the mistake that catches people, and it fails in the confusing direction. In Shopify, for each theme in the test: Customise, App embeds, turn Zinx Signal on.
Including the control. A control theme without the embed is exactly as broken as a variant without it — visitors sent there are never counted, so that group’s numbers stay flat forever while the other group’s climb. The group that looks like it is “doing nothing” is the one silently collecting nothing, and the result reads as a landslide.
The app refuses to start a theme test where any group’s theme does not definitely have the embed on, including themes it has not managed to read yet. A shopper sent to a theme without it also arrives wearing Shopify’s draft preview bar, which is a failure customers can see rather than one merely missing from a report.
Template names must match across themes
A deep link into a page that exists in one theme and not the other will 404 for half your traffic. Compare the templates in both themes for the pages this test covers before you start.
The control is a theme you chose
Every group points at a theme here, control included. That is usually your live one — but it does not have to be, and your published theme is not automatically any group’s. The picker labels the live theme and flags any theme where the loader is off, so you can see both facts without leaving the page.
What a shopper experiences
The loader lands on the current page, decides the group, and moves non-control visitors to that theme. Shopify then keeps them there for the session with its own preview cookie.
Two things we do that you should know about, because they are the difference between this feeling like your store and feeling like a demo:
- The page stays hidden across the hop. Revealing first would paint the old theme for the length of the navigation, which is the exact flash the overlay exists to prevent.
- Shopify’s draft bar is removed. “Draft — Exit preview” is addressed to a merchant looking at their own work. A shopper in a live theme test is not previewing anything, and we are the reason it appeared.
What to watch while it runs
- Run Check setup first, and again if anything changes. It is on the Preview tab, and reports whether every theme in the test is still on your store, whether the app embed is on your live theme, and whether the test is being served.
- Do not edit a theme that is in the test. Two “live” themes means every change has to be mirrored, and an edit to one arm mid-test is a change to the thing being measured.
- Do not delete or rename one. If a theme in the test disappears, the test stops being served rather than sending shoppers somewhere broken — but the numbers collected until then are still good, so pick a replacement or end it cleanly.
- Do not run two navigating tests on the same pages. Theme, template and redirect tests all move the visitor. Two theme tests may not share a theme either — both would end up measuring the same rendering against different baselines.
- Content tests under a theme test are a warning. Selectors written against one theme may match nothing in the other. Half the traffic then reads as “no difference” rather than as a failure, which is the most expensive mistake in this product.
When it wins
Publish the winning theme in Shopify: Online Store, Themes, Actions, Publish. Do it within a month or so — deleting a theme in Shopify is permanent, and the winning theme is what the test earned you.
Then end the experiment. While it is running we are routing every shopper on every page view, for a decision you have already made.