A/B testing interview questions, answered out loud
An A/B testing question checks whether you can turn a product idea into a test that gives a trustworthy answer. Say the decision first, then one primary metric with its guardrails, the unit you randomize, how many users and days you need, and what could bias the result. Most candidates lose points on the last part: peeking, a broken split, novelty and users who affect each other.
Key points
- Start from the decision the test serves. The metric follows from it.
- Name one primary metric, two or three guardrails and the exact definition of each.
- Pick the randomization unit before the sample size: it changes the variance and the interference risk.
- Show the sample-size arithmetic with round numbers, then convert it into days, rounded up to full weeks.
- Check the split before you read the result. A sample ratio mismatch makes every other number suspect.
The order that works out loud
Interviewers rarely want a formula first. They want to hear that you know what question the test answers and what would make its answer wrong. A spoken answer that follows the same seven steps every time is easy to follow and hard to derail.
- The decision: what ships, or does not, depending on the result.
- The hypothesis in one sentence, with the direction you expect.
- The primary metric and its exact definition, plus guardrails that must not get worse.
- The randomization unit (user, session, device, cluster) and who is eligible.
- Sample size and duration, from the baseline, the smallest effect worth shipping, the significance level and the power.
- Threats to validity: peeking, sample ratio mismatch, novelty, interference between users, seasonality.
- How you will read the result, including what you do if the primary metric moves and a guardrail fails.
A worked example: a shorter checkout form
Say the case is a food delivery app that wants to cut its checkout form from five fields to three. About 20,000 eligible users reach checkout each day, and 8.0% of them complete an order today.
Decision: ship the shorter form if it raises completed orders without raising refunds. Primary metric: the share of users who reach checkout and complete an order within the same session, per user. Guardrails: refund rate within seven days, average order value, and payment errors.
Unit: the user, not the session, because one person may open checkout several times and should see the same form each time.
Smallest effect worth shipping: 5% relative, so 8.0% to 8.4%, an absolute change of 0.4 percentage points. With a two-sided 5% significance level and 80% power, the usual approximation is n per group ≈ (1.96 + 0.84)² × [p₁(1 − p₁) + p₂(1 − p₂)] / δ². That is 7.84 × (0.0736 + 0.0769) / 0.004², or about 74,000 users per group.
Duration: 148,000 users in total at 20,000 a day is about 7.4 days. Round up to 14 days so both groups see two full weekly cycles, including the weekend peak.
| Input | Value | Why it matters |
|---|---|---|
| Baseline conversion | 8.0% | Sets the variance, p(1 − p) |
| Minimum detectable effect | +0.4 points (5% relative) | Halving it roughly quadruples the sample |
| Significance, power | 5% two-sided, 80% | Gives the 1.96 and 0.84 in the formula |
| Users per group | About 74,000 | From the formula above |
| Duration | 14 days | 7.4 days of traffic, rounded up to full weeks |
How it sounds out loud
The arithmetic is short when you round on purpose and say that you are rounding. Interviewers care more that the inputs are the right ones than about the third digit.
“Before I size it, the unit is the user, because one person can open checkout many times and I want each person to see one version.”
“Baseline is 8%, and I would only ship for at least a 5% relative lift, so 8.0 to 8.4. With 5% significance and 80% power, that is roughly 74,000 users per group.”
“We get 20,000 a day, so about a week of traffic. I would run two full weeks so both arms cover two weekends.”
“If the refund guardrail rises by more than the noise, I would not ship even with a significant lift, and I would look at which orders got refunded.”
The threats interviewers ask about
- Peeking: checking the p-value every day and stopping at the first significant reading inflates false positives well above 5%. Fix the duration in advance, or use a sequential method designed for repeated looks.
- Sample ratio mismatch (SRM): with a 50/50 split, 100,700 users in one arm and 99,300 in the other gives a chi-square of 9.8 and a p-value near 0.002. Something in assignment or logging is broken; find it before reading any metric.
- Novelty and primacy: users click a new design because it is new, or avoid it because it is unfamiliar. Plot the effect by day since first exposure and look for a trend.
- Interference: in marketplaces and social products one user’s treatment changes another’s experience. Randomize by cluster (city, time window, friend group) and accept the loss of power.
- Multiple metrics: ten independent secondary metrics at 5% will show at least one false win by chance about 40% of the time. Name the primary metric before the test and treat the rest as diagnostics.
Reading the result
Suppose the test ends with 8.36% in treatment and 8.00% in control, about 74,000 users each. The difference is 0.36 points with a standard error of about 0.14 points, so a 95% interval of roughly 0.08 to 0.64 points and a p-value near 0.01.
Say three things in order: the split looked healthy, the effect is positive and its interval excludes zero, and the guardrails held. Then the business reading: the lower end of the interval is below the 0.4-point effect you said was worth shipping, so the honest summary is that the change probably helps, by an amount that could be smaller than you hoped. That is a product call, and saying so is part of a strong answer.
Common mistakes
- Naming a metric before saying what decision the test serves.
- Choosing “engagement” or “conversion” without a definition: per user or per session, within what window, counted from which event.
- Sizing the test without stating the minimum effect worth shipping.
- Randomizing by session when the treatment persists across sessions.
- Forgetting guardrails, or listing guardrails with no rule for what happens when one fails.
- Treating a significant result as the end of the answer instead of checking the split, the novelty trend and the size of the effect.
Questions
Do I need to memorize the sample-size formula?
Knowing the shape helps: the sample grows with the variance and with the inverse square of the effect. Interviewers mostly check that you pick the right inputs and can turn the result into days.
What if the company cannot randomize?
Say so and switch to an observational design: difference in differences across regions, a staggered rollout, or a synthetic control. Then name the assumption each one needs and how you would check it.
What should I do when the primary metric wins and a guardrail loses?
Do not ship by default. Size the guardrail loss against the gain, find which users drive it, and propose a follow-up test that targets the cause.
Practise this out loud with Vera
Mockich’s data science case is the data science system design round: 45 to 50 min on one case, such as an experiment, a metric framework or a causal question, on a shared whiteboard Vera can read. Vera changes one thing at a time, for example revenue up while cancellations double, and gives at most two small nudges. Today the round uses the general system design stages and scorecard (8 skills, including the API and data model), not a case-specific one; the report quotes your reasoning for each. To practise the power and sample-size arithmetic itself, use the data science statistics and experimentation round.
Related guides
Sources
We read these sources on 2026-11-03 for the concepts (power, sample ratio checks, interference, observational designs); the checkout case, its numbers and the spoken lines are our own.
- [1] Alexey Grigorev, Data Science Interviews (theory questions) (CC BY 4.0)
- [2] Matheus Facure, Causal Inference for the Brave and True (MIT)
- [3] OpenIntro, OpenIntro Statistics (CC BY-SA 3.0)
- [4] bookdown.org, A/B testing (Product Data Science Interview)
- [5] Dataford, Pitfalls that can invalidate an A/B test