A/B testing compares the original quiz, arm A, with a challenger, arm B. It is free on every plan and uses no AI credit. Each participating browser is assigned once for the test. All placements of the original quiz use that assignment: the theme quiz, popup, launcher and quiz page. Nothing is published by creating a copy.
Start a useful comparison
- Open your quiz and select A/B test. Publish the original first.
- Choose a published Challenger quiz (B), or select Create a challenger copy. The existing copy flow makes a draft with the same questions, logic and design, without copying shopper history. Edit the copy, publish it, then return to the original quiz's A/B page and select it.
- Choose Shopper split and Winning metric per shopper who started. Completion, email capture, adding to cart and an attributed order are binary outcomes: a shopper either did it or did not. Each is measured per shopper who started.
- Choose Difference worth detecting, the smallest absolute conversion-rate difference you want the traffic estimate to plan for. For example, 30% versus 40% is 10 percentage points. This is a planning assumption, not a predicted result.
- Leave Automatically use a supported winner off to decide yourself, or enable it to let Quinn use a statistically supported winner. Select Start test. Both quizzes must belong to your store, be published, and be available for a test.
For a useful comparison, change one idea at a time, such as a shorter question sequence. The existing Quinn editor can help rewrite the challenger using metered AI credit; the test itself has no AI charge. Changing quiz content or unpublishing during a test stops and invalidates that test so results do not mix different versions. Start a new test after your changes.
Read the results honestly
We use confidence sequences that stay valid when you check the results every day. This avoids the extra false winners caused by repeatedly looking at an ordinary fixed-sample significance test. The supported range describes the possible difference in conversion rates, in percentage points, with B minus A as the direction. A range entirely above zero supports B; entirely below zero supports A. A range spanning zero supports neither.
Every shopper gets the same seven-day conversion window, starting at their first start event. Only shoppers whose full window has ended enter the statistical verdict. This prevents fast completers being counted before slower shoppers have had the same opportunity to convert. The live totals still show all shoppers and are labelled separately. Refreshing or repeatedly starting the quiz does not create additional experimental shoppers in the same browser.
Before there is a winner, Quinn estimates how many more shoppers with completed windows would be needed under your chosen difference and split. This is a planning estimate, not a deadline or a guarantee of significance. A smaller real difference can need more traffic than this estimate, or never produce a winner. A small store may never establish a winner, even when one displayed percentage is larger. You can keep testing, stop without a winner, or choose a version for business reasons.
Attributed orders means the orders already linked to a quiz by its cart receipt. For the winning metric, a shopper converts once if at least one non-test attributed order is recorded within the seven-day window. Late arrivals are excluded from that shopper's statistical outcome, and later refunds or cancellations do not rewrite an already observed purchase conversion. Live order and revenue totals continue to reflect the existing attribution rules and current order data. Revenue is descriptive, shown separately for each currency, and is not a winning metric. Attribution does not prove that the quiz caused an order.
Stop or use a winner
- Stop and return to original returns new loads of every original placement to the original quiz. The page asks you to confirm the storefront change.
- Use arm A or Use arm B lets you choose which quiz serves new loads. A manual choice is your decision; it is not proof of a statistical winner. The original placement and URL remain usable.
- If automatic promotion is enabled, Quinn promotes only a supported statistical winner. Evaluation happens when results are checked or shoppers next load the original quiz; it is not a timer or an immediate background promise. No supported verdict means no automatic change.
- Remembered and shared results keep the actual quiz that produced them, including after stopping or promotion, using a private receipt issued after completion. The receipt expires after 30 days. Older unsigned links, expired receipts and deleted results ask the shopper to start again rather than choosing an arm from a link.
Counting, identity and privacy
Theme design previews, the merchant's own Open look, recognised bots and Shopify replay requests are excluded. The server owns the assignment and bounds arm metadata to A or B. Repeated events and completion retries do not count as additional conversions. Existing analytics and exports can be filtered by test and arm; integrations receive bounded arm context when present. Ordinary CSV downloads keep their existing columns. Downloads explicitly filtered by test or arm add experiment_id and experiment_arm. A quiz outside a test keeps its existing behavior.
Restoring an old version of either arm keeps its current routing and membership. If authored content changes, the comparison stops and its statistical verdict is invalidated because the treatment changed. Starting, stopping or promoting a test creates no authored version. Customer exports and erasure use the identified customer’s own records; sharing a browser never authorizes access to another person’s contact details or receipts.
Assignment uses anonymous browser storage rather than fingerprinting. Clearing storage or switching devices can create a new apparent shopper and can put the same person in both arms. We cannot reliably identify everyone across devices or exclude every bot. Unstarted visits expire, pending enrollment and repeat sessions are capped, and a shop request budget limits automated probing. These limits cannot establish a person's identity. Counts describe recorded browser shoppers, not a verified count of distinct people. The statistical guarantee assumes independent shopper outcomes and stable conversion rates within each arm; changing traffic sources, stock or offers can limit how well the result represents future shoppers. Do not use a small numerical lead as proof.
Shopper data requests and erasure also cover test context. Erasing shopper data or changing a quiz stops and invalidates affected tests, because selectively removing observations would make the old statistical guarantee unreliable. Remaining live totals can still be viewed; start a new test for a reliable verdict.