Understand what an A/B test is and avoid drawing conclusions without adequate sample size, statistical significance, or a causal basis.
A/B Tests: What They Are and Where They Go Wrong
A/B Tests: What They Are and Where They Go Wrong
An A/B test (also called a controlled experiment or split test) is a method for comparing two variants of a product feature — a control (A) and a treatment (B) — by randomly assigning users to each group and measuring a pre-defined metric. Random assignment is what separates an experiment from a correlation study: it gives you a causal basis for conclusions, not just an observed association.
A result is decision-ready only when three conditions are met: (1) the sample is large enough — small samples produce noisy results where random chance looks like a real effect; (2) the difference is statistically significant — the result must be unlikely to have occurred by chance alone (a pre-set threshold, commonly p < 0.05); and (3) the metric was chosen before the test started, not after reviewing results. Choosing the metric after seeing the data (called 'post-hoc picking' or 'cherry-picking') inflates the chance of a false positive.
Two especially common errors are 'peeking' — stopping the test as soon as results look good instead of waiting for the pre-set sample size — and confusing correlation with causation. Even a properly run A/B test only proves causation within that specific context; extrapolating to a different audience or time period requires additional evidence. Unit-3 experiment thinking applies directly here: state a hypothesis first, run cheaply to learn, and respect the limits of what one experiment can prove.
Worked example of an invalid conclusion: Variant B shows a 15% higher click-through rate after only 80 users per group. The team declares B the winner. But with 80 users per group, the result is almost certainly within normal random variation — no statistical test at standard thresholds would confirm it. The right action is to continue the test until the pre-set sample size is reached, then evaluate significance.
Lesson notes
A/B Tests: What They Are and Where They Go Wrong
An A/B test (also called a controlled experiment or split test) is a method for comparing two variants of a product feature — a control (A) and a treatment (B) — by randomly assigning users to each group and measuring a pre-defined metric. Random assignment is what separates an experiment from a correlation study: it gives you a causal basis for conclusions, not just an observed association.
A result is decision-ready only when three conditions are met: (1) the sample is large enough — small samples produce noisy results where random chance looks like a real effect; (2) the difference is statistically significant — the result must be unlikely to have occurred by chance alone (a pre-set threshold, commonly p < 0.05); and (3) the metric was chosen before the test started, not after reviewing results. Choosing the metric after seeing the data (called 'post-hoc picking' or 'cherry-picking') inflates the chance of a false positive.
Two especially common errors are 'peeking' — stopping the test as soon as results look good instead of waiting for the pre-set sample size — and confusing correlation with causation. Even a properly run A/B test only proves causation within that specific context; extrapolating to a different audience or time period requires additional evidence. Unit-3 experiment thinking applies directly here: state a hypothesis first, run cheaply to learn, and respect the limits of what one experiment can prove.
Worked example of an invalid conclusion: Variant B shows a 15% higher click-through rate after only 80 users per group. The team declares B the winner. But with 80 users per group, the result is almost certainly within normal random variation — no statistical test at standard thresholds would confirm it. The right action is to continue the test until the pre-set sample size is reached, then evaluate significance.