Running a theme test: the heaviest change you can make

A theme test sends each group to a different theme. It is the biggest change you can test and the one that needs the most traffic to call, because you have changed everything at once: if the new theme wins you know it won, and not which part of it did.

That is a reasonable trade when the question genuinely is “should we launch this redesign”. It is a poor one when you want to know why — for that, break it into template or content tests and get answers you can reuse.

The figures below are live, and two of them run the app’s own checks. Set them up wrong on purpose and read what comes back.

Set the themes up first

Duplicate your live theme in Shopify and change what you want to test. Starting from a duplicate rather than a fresh theme keeps everything else constant, which is the whole point of a comparison.

Theme tests need the Online Store. A headless storefront has no Shopify theme to send anyone to, so the type is refused at launch there and marked “Not available on headless” in the picker.

The loader must be on every theme

This is the mistake that catches people, and it fails in the confusing direction. Every theme in the test needs the loader, by one of two routes:

The app embed, switched on

In Shopify, for that theme: Customise, App embeds, turn Zinx Signal on. This is the usual route for an Online Store 2.0 theme.

The manual script, seen on that theme’s own page

Pasted into the theme’s layouts from Store connection. A vintage theme has no app embeds, so this is its only route — and the right one for a page builder with its own layout. It counts once we have seen the loader on a page that theme served.

Including the control. A control theme without the loader is exactly as broken as a variant without it — visitors sent there are never counted, so that group’s numbers stay flat forever while the other group’s climb. The group that looks like it is “doing nothing” is the one silently collecting nothing, and the result reads as a landslide.

The figure calls the same function the Start button does, so the refusal is the app’s own sentence. It opens blocked, by a theme nobody has checked yet. That is deliberate. A shopper sent to a theme without the loader arrives wearing Shopify’s draft preview bar, a failure customers can see rather than one merely missing from a report, so an unread theme is not worth the gamble. Run the checks on Store connection and it gets read.

Notice the vintage theme has no “App embed on” to choose. Its loader can only be the script, and seeing it is what counts.

The control is a group like any other. It needs the loader too.

One install per theme, not two

Both routes on the same theme is a mistake of its own. It works — the second loader stands down — but it is two installs to keep in step, and the stale one is the copy nobody remembers to remove. The app spots it on your live theme and says so.

The warning in the figure is the one the app shows, from the same function. Turn the embed on with a pasted script and it tells you to keep one.

Template names must match across themes

A deep link into a page that exists in one theme and not the other will 404 for half your traffic. Compare the templates in both themes for the pages this test covers before you start.

The control is a theme you chose

Every group points at a theme here, control included. That is usually your live one — but it does not have to be, and your published theme is not automatically any group’s. The picker labels the live theme and flags any theme where the loader isn’t on, so you can see both facts without leaving the page.

What a shopper experiences

The loader lands on the current page, decides the group, and moves non-control visitors to that theme. Shopify then keeps them there for the session with its own preview cookie.

Three things in that figure are worth knowing:

• The page stays hidden across the hop. Revealing first would paint the old theme for the length of the navigation, which is the exact flash the overlay exists to prevent.
• Shopify’s draft bar is switched off. “Exit preview” is addressed to a merchant looking at their own work. A shopper in a live theme test is not previewing anything, and we are the reason it would have appeared.
• Your preview link loads once. It carries preview_theme_id itself, so it goes straight to the theme instead of landing and hopping.

What to watch while it runs

• Run Check setup first, and again if anything changes. It is on the Preview tab, and reports whether every theme in the test is still on your store, whether the loader is on, and whether the test is being served.
• Do not edit a theme that is in the test. Two “live” themes means every change has to be mirrored, and an edit to one arm mid-test is a change to the thing being measured.
• Do not delete one. A theme test whose theme is gone stops being served at once, so nobody is sent somewhere broken, and the test is paused with the reason on it. Restore the theme or point the group at another, then resume. The numbers collected until then are still good.
• One navigating test at a time. Theme, template and redirect tests all move the visitor. Two on overlapping URLs are refused at Start, and only one is ever served to shoppers at once. Two theme tests may not share a theme either — both would end up measuring the same rendering against different baselines.
• Content tests under a theme test are a warning. Selectors written against one theme may match nothing in the other. Half the traffic then reads as “no difference” rather than as a failure, which is the most expensive mistake in this product.

When it wins

Publish the winning theme in Shopify: Online Store, Themes, Actions, Publish. Then end the experiment. While it is running we are routing every shopper on every page view, for a decision you have already made.