Stats Questions to Ask
Questions for interrogating a statistical claim, whether it arrives as a published study, a consultant's slide, or an internal dashboard. Written for reviewers, editors, and managers who have to judge an analysis without redoing it.
The questions
Open any question for the note
What question was this analysis actually built to answer?
Why ask it
A clean answer names one outcome and one comparison. If it wanders across several outcomes, you are probably being shown whichever result came out best rather than the one the work set out to test.
Who is in this sample, and who got left out?
Why ask it
The exclusions matter more than the total: volunteers, people who finished the survey, patients who came back for follow-up. Anyone who dropped out is invisible in the chart but still changes what the number means.
How many observations sit behind the smallest number you are reporting?
Why ask it
Headline totals look reassuring. It is the subgroup breakdowns where n falls to a couple of dozen and a two-point swing turns into a finding.
Was the sample size decided before the data came in?
Why ask it
If collection stopped once the result crossed significance, the p-value cannot be read the usual way. A target set in advance, even a rough one, is a different situation from running it until it worked.
How big is the effect in units I care about?
Why ask it
Ask for percentage points, dollars, or days rather than a standardized effect size. Answers that only ever come back as significant are a sign nobody has checked whether the effect is large enough to act on.
What is the confidence interval, and does your conclusion survive at both ends of it?
Why ask it
The width tells you more than the midpoint. If the story holds at the pessimistic end it is robust, and if the interval runs from no effect to enormous, the headline figure is just the middle of a wide guess.
How many comparisons did you run before this one?
Why ask it
Analysts who kept track can give you a count. Twenty tests at the usual threshold produce about one false positive by chance, so an unremembered total is itself informative.
Was this hypothesis written down before the analysis, or found in the data?
Why ask it
Both are legitimate work, but they carry different weight. The problem is a pattern discovered while exploring and then reported as though it had been predicted.
What did you do about missing data?
Why ask it
Dropping incomplete rows is the default in most software and it quietly assumes the missing rows resemble the ones that stayed. Ask whether people with missing values differ from the rest, because that is usually the whole issue.
How were outliers handled, and what happens if you leave them in?
Why ask it
The answer you want is that they ran it both ways. Removing points with a rule invented after seeing them is a different act from applying a threshold set in advance.
What else could explain this, besides the explanation you are offering?
Why ask it
Watch whether a confounder gets named without prompting. Someone who has tried to break their own result already has a list, while someone who has not will treat the question as an attack.
Is the comparison group really comparable?
Why ask it
This is where most observational claims come apart. Groups that select themselves into the treatment differ in ways nobody recorded, and statistical adjustment cannot repair an unmeasured difference.
How were the key variables actually measured?
Why ask it
Engagement, wellbeing, and productivity all end up as some concrete proxy. Noisy measurement usually shrinks real relationships, so a null result can mean the instrument was poor rather than that nothing happened.
Did you check the assumptions the model needs, and what did you find?
Why ask it
The useful follow-up is what they did when an assumption failed. A flat statement that everything was fine, with no detail, often means nobody looked past the default output.
Did you rescale or log any variables, and how should I read the coefficients now?
Why ask it
After a log step a coefficient describes a relative change rather than an absolute one. Plenty of write-ups report rescaled results in raw language and nobody catches it.
How much of the variation does the model explain?
Why ask it
A model can be statistically significant and still explain very little, which matters the moment someone wants to use it to predict individual cases instead of describing an average.
Has this held up on data the model did not see?
Why ask it
Validation on the same rows that fitted the model will look good almost regardless. Ask for a holdout set, a later time period, or an independent replication.
What decision would change if the true effect were half this size?
Why ask it
Moves the discussion from statistics to consequences. If nothing changes, the argument about precision was never really the point.
What part of this analysis are you least confident about?
Why ask it
People who know their own work answer quickly and specifically. A long pause, or nothing at all, usually means the limitations were written last and never revisited.
With another month and a bigger budget, what would you do differently?
Why ask it
A gentler route to the same limitations. It also surfaces the data they wanted but could not get, which tells you exactly what the current numbers cannot settle.
How to use these questions
Practical guidance for the conversation itself
Do this before you ask anything
Find the sample description first
Read the methods or the data notes before the results. Most weak claims are already visible in who was counted, over what period, and how many were lost along the way.
Write down what would change your mind
Decide in advance what result would be big enough to act on. Otherwise every number looks like support for whatever you already suspected.
Ask for the analysis file, not the summary
A request for the code, query, or spreadsheet is normal and answers several questions at once. Reluctance to share it is worth more attention than any single figure.
What the answers tend to tell you
- Specific numbers offered without checking usually mean the person has handled the data themselves.
- Repeated appeals to significance, with no effect size, suggest the practical size has never been discussed.
- An immediate, concrete limitation is a good sign. A defensive one means the limitation is the real story.
- Round thresholds and tidy cutoffs often mark decisions made after the results were seen.
- If every answer routes to a vendor, a model, or a tool, nobody in the room owns the analysis.
Common pitfalls
Treating a p-value as a measure of size
It is a statement about how surprising the data would be if there were no effect, and it grows more impressive with sample size alone. A huge dataset makes trivial differences significant.
Accepting an average for a decision about individuals
Group averages say little about any one case. If the plan is to score people, screen customers, or target patients, ask about the spread and the error rate for individuals.
Letting a chart do the arguing
Truncated axes, dual scales, and smoothed lines can make noise look like a trend. Ask for the underlying counts whenever a visual seems more persuasive than the numbers behind it.
Interrogating the analyst rather than the analysis
Rapid-fire technical challenges tend to produce defensiveness rather than information. Ask one question at a time and let the person show their work.