How data science case interviews work
The data science case round, also called the product case or metrics round, usually runs about 45 minutes around one open product question: define success for a feature, design an A/B test, or find out why a metric moved. The guides we read agree on the order that works: the decision the analysis serves, one precise primary metric with guardrails, the experiment or analysis design, then a recommendation. The mistake they report most is reaching for a test statistic before the metric and the design are settled.
Key points
- Ask what decision the answer will change before you name a metric.
- Define the metric exactly: which users, which action, which time window, which denominator. Then add two or three guardrails.
- Design before statistics: the unit of randomization, the sample size and the duration come before any test.
- For a metric drop, check that the drop is real before you guess why: tracking, pipeline and definition first, then segments.
- End with a recommendation and the number that would change it. “It depends” is not an ending.
The usual shape of the round
The guides describe five steps for a 45-minute case. Not every case has all of them: a metric-drop question spends most of its time on diagnosis, an experiment question on the design. The times are our consolidation of the guides, not a fixed schedule.
| Step | Typical time | What happens |
|---|---|---|
| Clarify the goal and the decision | 2 to 5 min | Who acts on the answer and what they would do differently; the product, the users, the scope and your assumptions. |
| Metrics | 8 to 12 min | One primary metric with a precise definition, the drivers that explain it, two or three guardrails, and what the metric misses. |
| Experiment or analysis design | 12 to 18 min | Hypothesis, unit, sample size and duration, assignment and pitfalls; or an observational method when you cannot randomize. |
| Diagnosis, if a metric moved | 10 to 20 min | Check the data, scope and segment the change, separate internal from external causes, and rank hypotheses by likelihood and cost to check. |
| Decide | 3 to 5 min | A recommendation, the number that supports it, the number that would reverse it, and the launch criteria. |
What interviewers look for
No company publishes its case rubric. The guides we read describe the same signals:
- Problem structuring: an ambiguous question broken into parts, starting from the decision.
- Metric judgment: a defensible primary metric with a precise definition, drivers kept apart from guardrails, and an honest view of what it misses.
- Experiment design: unit, power and minimum detectable effect, duration, guardrails and a launch rule. Naming the pitfalls before you are asked is a strong signal.
- Causal reasoning without an experiment: methods such as difference-in-differences or a holdout, with the assumption each one needs.
- Diagnostic rigor: the data checked first, then segments, then hypotheses in order.
- Statistical correctness: what a p-value and a confidence interval mean, and the gap between statistical and practical significance.
- A decision: a recommendation turned into a business number, not a list of options.
Experiment questions and their pitfalls
Experiment questions are judged on the design before the statistics. The order the guides give: the hypothesis, the unit of randomization (user, session, region), the primary metric and guardrails, sample size and power (how each input moves it matters more than the formula), the duration, then the pitfalls you would watch for:
- Sample ratio mismatch: the split came out uneven, so something in assignment or logging is broken.
- Peeking: stopping the test the first time it looks significant.
- Novelty and learning effects: an early lift that fades, or a slow start that grows.
- Interference and network effects: treated users change what control users see, common in social products and marketplaces.
- Multiple testing: many metrics or variants, so some look significant by chance.
- Simpson’s paradox and mix shift: the overall effect hides opposite effects in segments.
Metric-drop questions
When the prompt is a metric that moved, the guides judge structure under uncertainty rather than finding “the” cause. Confirm the change is real (tracking, a pipeline delay, a changed definition), scope it by segment such as platform, region, app version or new against returning users, separate internal causes like a launch from external ones like seasonality, and rank the hypotheses by how likely and how cheap to check they are.
In this kind of case the interviewer usually holds the data and reveals a cut only when you ask for it. A candidate who asks for no data gets none.
The mistakes guides report most
- Jumping to a statistical test before the metric and the design.
- A vague metric, such as a count of active users with no window, no denominator and no link to user value.
- No guardrails until the interviewer asks.
- Guessing why a metric dropped before checking that the data is right, or never segmenting.
- Reading a p-value as the chance the null hypothesis is true, or a confidence interval as a 95 percent chance the true value is inside.
- Ignoring interference, novelty or an uneven split until prompted.
- Ending on “it depends” with no recommendation.
How the interviewer behaves
Expect twists on the experiment: the split came out uneven, the result is flat and a launch date is close, the primary metric went up while a guardrail got worse, or the effect after launch is larger than in the test. On metrics, expect “how exactly would you define retention?” and “what would make this metric go up while users are worse off?”.
Structure is judged early and the follow-ups push on its weakest part, so a framework recited without reasons does not survive the second question.
How the bar changes with level
Sources on levels are thin for this round; treat the table as a tendency, not a rule.
| Mid-level | Senior and above | |
|---|---|---|
| Metrics | A correct primary metric and guardrails for a given feature | What a team should measure, metric hierarchies, and checking that a short-term metric predicts the long-term one |
| Experiments | A sound A/B design, power and the basic pitfalls | Interference, long-term effects and holdouts, variance reduction, and cases that cannot be randomized |
| Diagnosis | Systematic validation and segmentation | Faster prioritization, the impact in numbers, and the monitoring that would catch it next time |
| Influence | Explains the result clearly | Pressure-tests the product hypothesis with partners and drives the decision |
Go deeper on this round
Practise the round out loud
Mockich’s data science case is the data science system design round: 45 to 50 min on one case, such as a metric framework, an experiment, a causal question or an analytics pipeline, on a shared whiteboard Vera can read. The case comes with its own clarifications, deep dives and twists. Today it uses the general system design stages and scorecard (8 skills, including the API and data model), not a case-specific one; the report quotes your reasoning for each.
Sources
We read sources 1 to 4 in full on 2026-10-30 (source 4 in an earlier reading) and sources 5 to 8 through search excerpts only, and summarised where they agree, in our own words. They report typical practice; no company publishes how it scores case rounds, the step times are our consolidation, and formats differ between companies.
- [1] PracHub, The Data Science Case Study Interview: a 4-step framework
- [2] PracHub, Product metrics, guardrails and retention
- [3] bookdown.org, A/B Testing (Product Data Science Interview)
- [4] bookdown.org, Metrics and problem solving (Product Data Science Interview)
- [5] Interview Query, How to answer a metrics drop question
- [6] DataInterview, A/B testing interview questions
- [7] Dataford, A/B test pitfalls: novelty effects, network interference and sample ratio mismatch
- [8] Interview Kickstart, Senior data scientist interview process