Running a content test: headlines, buttons and the selector problem

A content test rewrites specific things on the page: a headline, a button label, an image, a block you want hidden. No code, no second theme, nothing to set up in Shopify first. It is the fastest test to build and the one most likely to fail without telling you.

The figures below are live. The first shows what each change does; the second lets you break a selector the way theme updates break them.

Setting one up

On Modifications, each group gets a list of changes, applied in order. Use the picker: Pick from your store opens your live storefront, you click the thing you want to change, and it comes back with a selector and a note of what that element looked like when you chose it. Picked elements appear in the editor while you are still in the other tab.

Each change is a selector, an operation and a value. There are four operations — exactly the four the loader implements, because an operation it ignored would be a test that silently changes nothing.

By default the figure shows Replace text on the product title: the variant swaps “Linen shirt” for a longer line and nothing else on the block moves. Turn the variant off to see the control.

Replace text

The safe one. Swaps the words — and anything inside the element, an icon or a span, goes with them.

Replace HTML

For markup you control, when text is genuinely not enough. Scripts in it never run, and you are responsible for it being valid.

Set an attribute

An image source, a link target, an aria label. Switch off “Attribute named” in the figure: Save refuses an attribute change with no attribute, because it would set nothing.

Hide it

Removes a block from view. “Does this badge help or distract” is a good question.

Leave one group with no changes and that group is your store as it is. That is normally the control, though it does not have to be: two rewrites against each other, with nobody seeing the original, is a perfectly good test.

The selector problem, stated plainly

A selector that stops matching does not produce an error. The change simply does not happen. Those visitors still count as having been in the test, so the group reports as “no different from the control” — indistinguishable from a real finding of no effect.

A confident decision made on evidence that was never collected.

That is the most expensive thing this kind of test can do. Everything below is about avoiding it.

Pick, do not type

The picker never uses class names. Themes rewrite them on every update, so a class-based selector is the one thing guaranteed to rot. It tries, in order:

1. an id a developer wrote on purpose;
2. a stable data attribute — data-testid, data-section-type, itemprop and a few others;
3. a :nth-of-type path, walked up until it points at one element and nothing else.

It rejects Shopify’s generated section ids, the long ones like shopify-section-template--17234567890123__main. Shopify regenerates them, so a selector anchored to one works until it doesn’t — worse than not using it, because it looks like it worked.

The figure opens on the common case: a class name typed from devtools, after a theme update. The update renamed product__title to product-title, the selector matches nothing, and nothing says so. Switch to the id or the data attribute and the same update changes nothing.

The path is the honest last resort. It survives a regenerated section id, but when an app slips a banner in above the product it points at the banner instead — a wrong element rather than a missing one. That is why it comes third.

Check it on a phone

Plenty of themes render a different DOM at mobile widths — a separate header, a drawer instead of a nav, a duplicated price block. A selector picked on a laptop can match nothing on the device most of your traffic uses. Open the preview link on your phone before starting.

Watch for something overwriting you

Changes are applied before the page is shown, then re-applied for three seconds afterwards, because themes render sections, drawers and quick views late. What that cannot survive is another app that owns the same element and rewrites it continuously. Load the page and watch: if your wording flashes in and back out, something else owns it, and the fix is to change it at the source.

Two tests, one element

Two content tests that edit the same element on overlapping pages get a warning. Whichever applies second wins, so neither measures what it claims to. Two content tests on different elements are a perfectly good pair, which is why this is a warning and not a refusal.

What to test, and what not to

Content tests are good at questions with a short causal chain. Does this headline sell the product better than that one. Does naming the delivery time near the button reduce abandonment. Does the badge earn its space.

They are poor at anything requiring layout to change. If the honest answer involves moving things around, that is a template or theme test — reaching it with a dozen DOM edits produces a test that is fragile in a dozen places.

Before you start

• Open a preview link per group and confirm each one looks different. Preview reads what is saved, so save first.
• Check the mobile rendering, not just the desktop one.
• Run Check setup on the Preview tab. It loads your live pages and names any selector that matches nothing, before a shopper finds out for you.
• Check the Targeting tab’s URL patterns. A test scoped to one page collects a fraction of the traffic, which is fine as long as you expected it.

When it wins

Make the wording real: edit it in your theme or your product content, then end the test. Leaving it running means every shopper waits on our script for a change that could be in your store, and content tests are the easiest of all to make permanent.