A/B testing compares outcomes from two randomly assigned versions over the same period
An A/B test randomly assigns visitors to versions A and B and measures their behavior during the same period. Assess a website change using predefined metrics and an experiment design, together with data volume and tracking quality. (Microsoft experiment design)
For an inquiry-button change, record both clicks and completed inquiries. A comparison of different publication periods also reflects changes in advertising, weekdays, or pricing, which complicate attributing a difference to the wording.
Define the hypothesis and outcome before the experiment
State what will change and which behavior it should affect in one hypothesis. Choose the decision metrics before examining results.
Match the change to the actual operation
An inquiry button could compare Submit with Send inquiry. A booking confirmation and a request to discuss a booking need labels that describe their respective operations. See Microcopy for buttons and forms for examples.
Keep the first comparison focused. Changing the heading, photograph, and button together evaluates that combination; attributing the result to any single element requires further evidence.
Measure completion and problems as well as clicks
If completed inquiries are the goal, use their completion rate as the main outcome. Record clicks as an intermediate action, along with input errors and page speed. Microsoft's experiment practice combines outcome metrics with indicators of regressions and data-quality problems. (Microsoft experiment monitoring)
Align visitor assignment and tracking
Serve both versions during the same period. Define the identifier used for assignment and the unit used for counting outcomes. Verify returning-visitor behavior, assignment logs, and completion tracking in the experimentation system.
Keep the assigned version consistent for an identifier
For a visitor-level comparison, configure the same identifier to receive the same version. Cookie-based identifiers can change when cookies are cleared or another device is used. State whether your counts represent people, browsers, or sessions.
Compare the planned allocation with recorded counts
A configured 50:50 allocation will usually have some variation in a finite sample. A discrepancy beyond expected random variation calls for investigation of assignment and missing records. This issue is called sample ratio mismatch, or SRM; Microsoft checks it before analyzing treatment effects. (Microsoft SRM diagnosis)
Missing completion events for version B can make actual inquiries appear to decline. Connect assigned versions and outcomes through the same identifier, correct tracking problems, and rerun an invalid comparison.
Assess the observed difference and its uncertainty
Calculate the rates, then assess the precision of the estimated difference. A small sample can produce large rate changes from only a few outcomes.
State the denominator and the unit of change
Calculation example with assumed figures: each version receives 500 assigned identifiers. Twenty identifiers in A and 25 in B complete at least one inquiry. Count repeated submissions from the same identifier once.
| Version | Assigned identifiers | Identifiers completing an inquiry | Completion rate |
|---|
| A | 500 | 20 | 4% |
| B | 500 | 25 | 5% |
The observed difference is one percentage point, or a 25% relative increase over A. Choosing B also requires an assessment of uncertainty and other metrics.
Plan data requirements and the stopping rule
The baseline completion rate, smallest change worth detecting, and statistical method affect the required sample size. Plan the sample and duration in the experimentation system. An encouraging interim result alone is insufficient grounds for stopping; methods that support interim decisions must account for repeated testing.
A confidence interval expresses the range of estimates supported by a method accounting for sampling variation. NIST describes methods for the difference between two proportions. An interval spanning both improvement and deterioration leaves the direction uncertain. (NIST proportion-difference intervals)
An inconclusive result can reflect a small effect or insufficient sensitivity. On a low-traffic site, observing people using the page can help identify concrete problems before a large experiment is feasible.
Record the adopted change and its evidence
Keep the versions, dates, metrics, assignment unit, tracking issues, observed difference, uncertainty, and decision together. After rollout, check inquiry delivery and page behavior.
When putting an adopted service description on your website, choose a publishing method that also supports subsequent content updates.
SpreadSite is a no-code website builder that lets you create, publish, and update a website from Google Sheets using ready-made templates.
To build a website that reflects the service descriptions and guidance you have evaluated, give SpreadSite a try!
Build a website with SpreadSite →
