Statistical Questions to Ask
Questions to ask when a number is put in front of you, whether it comes from a paper, a dashboard, a press release or a colleague's slide. They run from the plain first question about where the figure came from to the ones that test how much weight it can carry.
The questions
Open any question for the note
Where did this number come from?
Why ask it
Ask before anything technical. A surprising share of figures in circulation trace back to a summary of a summary, and the chain often breaks at a press release. If nobody in the room can name the original source, that is the finding.
Who ran it, and who paid for it?
Why ask it
Funding does not make a result wrong, but it tells you which way to check first. The more useful follow-up is whether the funder had any say over publication, since the power to withhold a result shapes a literature more than any single study does.
How many people or cases is this based on, and how were they chosen?
Why ask it
The selection method matters more than the size. A self-selected online poll of fifty thousand can be less informative than a well-drawn sample of a thousand, because a big biased sample just gives you a precise estimate of the wrong quantity.
Compared to what?
Why ask it
Any single figure is uninterpretable without a baseline: last year, another group, a control arm, a plausible alternative. When someone cannot supply the comparison, they usually chose the number because it sounded large on its own.
Is that a change in the rate or a change in the number of cases?
Why ask it
This is where most misleading headlines live. A fifty per cent increase on a base of two in a million is one extra case per million; ask for both the relative and the absolute figure, because a source offering only the relative one has picked the more dramatic version.
What is the denominator?
Why ask it
Percentages travel without their base, and the base is often the interesting part: per patient, per visit, per thousand people, per attempt. Two teams reporting the same percentage while dividing by different things is a common and expensive mistake.
What exactly was measured?
Why ask it
Numbers frequently measure a stand-in for the thing you care about: reported intention rather than behaviour, clicks rather than purchases, a blood marker rather than survival. Ask what would follow if the stand-in and the real outcome came apart.
How was the question put to the people answering it?
Why ask it
Wording, order and the available options shape survey results substantially, and a leading question can produce whatever the writer expected. Ask to see the questionnaire itself rather than the summary of it.
Who is not in this sample?
Why ask it
Ask specifically about people without phones, without addresses, in institutions, who declined, who dropped out. The exclusions are usually systematic rather than random, and they bias the result in a direction you can often predict.
What is the interval around this estimate, and how wide is it?
Why ask it
A point estimate with no interval is an assertion. If the interval spans values that would lead to opposite decisions, the honest summary is that the study did not settle the question, whatever the headline number says.
How big is the effect in units I care about?
Why ask it
Ask for days, kilograms, pounds, cases per thousand rather than a standardised measure. Effects can be statistically clear and practically trivial, and this question is the fastest way to tell which you are looking at.
Was this the outcome they set out to measure?
Why ask it
Results found after the fact carry much less weight than the ones the study was designed to test. Ask whether the plan was registered in advance and whether the outcome reported now matches the one named then.
How many other things were tested before this one came out significant?
Why ask it
Test enough combinations of groups and outcomes and something will clear any threshold by chance. If twenty comparisons were run and one is being shown to you, the single result means much less than it appears to.
What happened to the people who dropped out or did not respond?
Why ask it
Loss is rarely random: the sickest, the busiest and the least satisfied leave first. Ask what proportion was lost and whether the analysis counts them, because a study of the people who stayed answers a different question.
How was missing data handled?
Why ask it
The choices, dropping rows, carrying values forward, imputing, can shift a conclusion on their own. You are checking that a decision was made deliberately and stated, not that any particular method was used.
What else changed at the same time?
Why ask it
The classic confounder question, and the one most likely to explain a result. Ask what else was going on in that period, in those groups, and whether anything could have caused both the supposed cause and the effect.
Would a handful of extreme cases move this much?
Why ask it
Averages are easily dragged by a few large values, which is why medians and distributions are worth asking for. If the answer is that a few cases would change the conclusion, ask to see the spread rather than the summary.
Has anyone else found the same thing?
Why ask it
One study is a claim; agreement across independent groups and methods is evidence. Ask specifically about attempts that failed to find it, since those are less likely to have been published or mentioned.
What would have to be true for this conclusion to be wrong?
Why ask it
This asks the analyst to think against their own result, which good ones enjoy. An answer naming a specific alternative explanation is reassuring; being told the conclusion could not be wrong is not.
If you had to bet on this holding up in five years, how would you bet?
Why ask it
Ask it lightly and last. People will express doubt in the language of a wager that they will not express in the language of methods, and the hedging in the answer usually tells you how much to rely on the number.
Interrogating a number without being the difficult one in the room
Practical guidance for the conversation itself
How to work through it
Read the methods before the abstract
Abstracts state conclusions; methods state what was actually done. Reading in that order stops you spending your attention checking whether the evidence supports a claim you have already absorbed.
Ask for the table, not the summary
Request the underlying counts, the sample sizes per group and the raw figures. Most disagreements about what a number means dissolve once everyone is looking at the same table, and some numbers do not survive the request.
Convert everything into the same units
Percentages, rates, odds and counts are easy to mix up in conversation. Restating each figure as a count out of a fixed base, say cases per thousand, makes comparisons possible and exposes the ones that were never comparable.
Write down the decision the number is meant to inform
Precision only matters relative to a decision. If the choice is the same across the whole plausible range of the estimate, you can stop asking questions; if it flips halfway across, you know exactly which question matters most.
Sequences for common situations
A figure quoted in the news
- 1Where did this number come from, and can I get to the original?
- 2Is it a change in the rate or in the number of cases?
- 3What is the denominator?
- 4Has anyone else found the same thing?
A result presented in a work meeting
- 1What exactly was measured?
- 2Compared to what?
- 3How many other things were tested?
- 4What decision does this change, and how wide is the interval around it?
A study you are asked to act on
- 1Who ran it, who funded it, and was the plan registered in advance?
- 2Who is not in the sample, and how many were lost along the way?
- 3How big is the effect in units that matter to the people affected?
- 4What would have to be true for the conclusion to be wrong?
Warning signs on the face of a chart or claim
- A percentage with no sample size attached anywhere in the document.
- A bar chart whose axis does not start at zero, or a line chart with no axis labels at all.
- A comparison between two groups that were not chosen the same way.
- The word significant used with no effect size and no interval.
- A single number carried through several documents with no citation to an original source.
- A rate presented for a period chosen after the data was seen, such as a start date that happens to be the lowest point.
- An average reported where the distribution is obviously skewed, such as income, waiting times or revenue per customer.
- A result described as proving something, which is not how any single study works.
Wording that keeps the room on your side
Asking in a meeting without sounding like an attack
- 1Start with the shared aim: "I want to make sure we can defend this if it gets challenged."
- 2Ask the factual question: "What is the denominator here?"
- 3Offer the alternative reading rather than an accusation: "Could the drop in March explain the whole difference?"
- 4Close with what would settle it: "If we had the per-group counts, would that resolve it?"
Following one number to its source
- 1Ask who produced the figure and when.
- 2Ask for the document rather than the slide.
- 3Check whether the document itself cites someone else, and repeat until you reach data.
- 4If the chain ends in a press release or an unsourced report, say so plainly and stop using the number.