Stopping at the first significant result changes the testing procedure
Suppose you change a purchase button on your website and compare versions A and B. Checking every day and stopping when B has a higher purchase rate and a p-value below 0.05 might seem like a quick way to choose a winner.
Adding that rule to an ordinary test designed for one analysis at a predetermined sample size can increase the probability of declaring a difference when none exists. If your results determine when you stop, your statistical method must account for that stopping rule. Johari and colleagues examine this problem and develop inference methods that accommodate continuous monitoring. (Paper on continuous monitoring of A/B tests)
Here, an A/B test randomly assigns users to A or B and compares their outcomes over the same period. Comparing last month's A results with this month's B results is a different design.
Five percent is not the probability that this winner is wrong
A p-value is the probability, assuming equal purchase probabilities for both versions, of obtaining a test statistic at least as extreme as the observed statistic. It does not directly state the probability that B is better or that this particular decision is wrong.
A test designed at a 5% significance level aims to limit erroneous declarations of a difference to at most 5% of repeated experiments in which no difference exists, provided its assumptions hold and the same procedure is followed. NIST distinguishes the significance level from the observed p-value and advises choosing the threshold in advance. (Significance levels and p-values)
Stopping when a result is significant while continuing otherwise creates repeated opportunities to declare a difference. The error rate for that stopping rule need not equal the 5% associated with one fixed-sample analysis. Because cumulative data include earlier users, daily analyses are also dependent; multiplying a single-test error rate by the number of days does not calculate the actual error rate.
An interim purchase rate of 13% can become 10.4% at the planned endpoint
The following figures are hypothetical calculations, not observations from a real website or research study.
Assume each user continues to see the same version and is counted once as either purchasing or not purchasing. Observe purchases for 24 hours after the user's first exposure, and include only users whose observation window is complete. Before starting, suppose you decide to analyze once each version has 5,000 fully observed users. That sample size is an assumption for this example, not a recommendation.
| Analysis point | A purchasers and users | A purchase rate | B purchasers and users | B purchase rate | Ordinary two-sided p-value |
|---|
| Interim | 100 / 1,000 | 10% | 130 / 1,000 | 13% | About 0.035 |
| Predetermined endpoint | 500 / 5,000 | 10% | 520 / 5,000 | 10.4% | About 0.509 |
The calculations use a two-sample proportion z-test with a pooled purchase rate under the equal-probability null hypothesis and no continuity correction. The pooled rate is total purchasers divided by total users across both versions. A two-sided test can detect a difference in either direction. Outcomes are assumed independent across users.
At the interim analysis, B is ahead by 3 percentage points and the p-value is below 0.05. Among the next 4,000 users per version, however, A has 400 purchasers and B has 390, reducing the cumulative difference to 0.4 percentage points.
The interim result meets the significance threshold, while the planned final analysis does not detect a significant difference. A nonsignificant final result also does not prove equal purchase probabilities. This single hypothetical dataset cannot establish the false-positive rate of early stopping. The table does not specify the true purchase probabilities either.
Fix the endpoint or use a method that permits early stopping
For a fixed-sample analysis, calculate the required sample size before starting, using the purchase-rate difference you want to detect, the usual purchase rate, the tolerated false-positive rate, and the probability of detecting that difference. In this example, follow the planned analysis after 5,000 users per version complete their 24-hour windows; do not adopt B because an interim p-value crossed 0.05.
Continuing after a nonsignificant planned endpoint and retesting until significance appears also changes the stopping rule. Specify the outcome metric, analysis point, and treatment of interrupted experiments before examining results.
Sequential tests account for repeated analyses as observations accumulate. Johari and colleagues construct p-values and intervals that control false positives under data-dependent stopping. These methods have a different statistical design from recalculating an ordinary p-value every day. (Inference allowing data-dependent stopping)
Read your analysis tool's documentation to establish whether its reported values support a fixed-sample analysis or sequential testing. Sequential methods still require adherence to their assumptions about metrics, randomization units, and observation windows.
Distinguish monitoring failures from declaring a winner
Continue monitoring payment errors and broken pages during an experiment. Microsoft's experimentation guidance describes early detection and shutdown of harmful bugs while also requiring statistical adjustments for repeated interim analyses. (Microsoft's during-experiment monitoring guidance)
If B's purchase button fails, you can stop the experiment to repair it. That interruption should not be reinterpreted as a completed planned test establishing that A increases the purchase rate.
Ending an A/B test solely because an ordinary p-value first falls below 0.05 can invalidate the intended false-positive rate. Decide before starting whether to follow a fixed-sample endpoint or use a statistical method that accommodates early stopping.
Editing a button and evaluating the effect of that edit are separate tasks. When updating your website, specify what you will count and when you will judge the results, alongside the wording you will change. This avoids choosing a version solely because an interim result looked favorable.
SpreadSite is a no-code website builder that lets you create, publish, and update a website from Google Sheets using ready-made templates.
To build a website whose service descriptions and signup button labels you can update yourself, give SpreadSite a try!
Build a website with SpreadSite →
