Questions to Ask a Researcher
Questions for interviewing a researcher or reading a paper closely, whether you are a journalist, a practitioner, or a student. They move from what was found to who was studied, how much is uncertain, and where the result should not be applied.
The questions
Open any question for the note
In one sentence, what did you find?
Why ask it
Researchers rehearse this and it is still revealing. If the one-sentence version needs three caveats to survive being said out loud, the result is more fragile than the abstract suggests.
What question were you actually trying to answer, and who wanted to know?
Why ask it
Papers often answer something narrower than the title implies. Naming the audience, a regulator, a clinician, a funder, an open argument in the field, tells you what the study was built to be good at.
Why this design rather than the obvious alternative?
Why ask it
Every design is a trade. Hearing what an observational study gave up against a trial, or a lab task against field data, tells you where the finding is strong and where it is borrowed.
Who is in the sample, and who is not?
Why ask it
Ask for the recruitment route: a university subject pool, one hospital system, an online panel. The exclusions are usually the more informative half and rarely make the abstract.
How big is the effect in units I would recognize?
Why ask it
Push for minutes, dollars, percentage points, or cases avoided rather than standardized units. A statistically large effect can be a difference nobody in the field would act on.
How much of this could be chance, and where would I see that in the paper?
Why ask it
You want to be pointed at intervals and sample sizes rather than a p-value. A researcher who cannot show you the uncertainty in their own figures is describing a conclusion, not a result.
Was the analysis planned before you saw the data?
Why ask it
Preregistration, or its absence, changes how much weight a surprising subgroup finding deserves. Honest answers here often contain the phrase that one was exploratory.
Which result surprised you, and did you go back and check it?
Why ask it
Surprises are where both errors and discoveries live. What they did next, recoded, reran, called a collaborator, says more about their standards than the finding itself does.
What did you do about missing data and outliers?
Why ask it
Dropped cases and trimmed tails can carry a result on their own. The follow-up worth asking is whether the conclusion survives the other reasonable choice, and whether that check appears in the paper.
What is the most plausible alternative explanation?
Why ask it
Careful researchers keep a named rival explanation and a reason it is weaker. An answer that there is none means either they are far ahead of the field or they have not looked.
Where would you expect this not to hold?
Why ask it
Boundary conditions are the difference between a finding and a slogan. Specifics such as older patients, lower doses, or other countries tell you whether your own case sits inside the evidence.
What would change your mind?
Why ask it
This asks for a falsifiable condition, which separates a position from a commitment. Vagueness here tends to predict vagueness about limitations everywhere else.
Has anyone tried to replicate it, and what happened?
Why ask it
Failed and partial replications rarely get the attention the original did, and the author usually hears about them first. Ask directly, because silence in the literature is not agreement.
Are the data and code available, and what would I need to rerun it?
Why ask it
Answers range from a public repository to consent terms that genuinely prevent sharing. Either is workable; evasion about which one applies is not.
Who funded this, and did they see it before publication?
Why ask it
Funding alone is not a verdict, but review rights and approval clauses are. Ask about the arrangement rather than only the name on the grant.
How has this been reported, and what did the coverage get wrong?
Why ask it
Researchers are usually irritated by one specific distortion, and naming it is the fastest route to where the study gets over-read. It also shows how closely they police their own claims.
If a practitioner reads one figure, which one, and what should they do differently?
Why ask it
Forces a translation from evidence into action. If nothing follows for practice yet, a straight answer saying so is more useful than an implication that something does.
What did not make it into the paper?
Why ask it
A pilot study, a failed condition, an underpowered arm. What sits in the drawer shapes how to read what is on the page, and most researchers will say if asked plainly.
What would you do differently if you ran it again?
Why ask it
The answer is usually about measurement or sample size, and it points straight at the weakest link. Someone who would change nothing has stopped examining the work.
What is the open question you cannot answer yet?
Why ask it
Where a field is stuck is often more informative than what has been settled, and it tells you which claims elsewhere in the literature to treat as unresolved.
Reading a Study Without the Hype
Practical guidance for the conversation itself
Before you ask
Read the methods and the figures first
The discussion section is where authors interpret generously. Arriving with the sample size, the design, and one confusing figure gives you specific questions instead of general ones.
Write down what would change your own decision
If you are weighing a policy, a purchase, or a treatment, name the number that would move you. It keeps the conversation on effect sizes rather than on significance.
Settle the sample before the conclusion
Nearly every overreach traces back to who was studied. Establish that first and the remaining answers are much easier to weigh.
Ask them to define one term
Adherence, improvement, engagement, and severity are all operationalized differently between studies. The definition often narrows the claim considerably.
Phrases that should prompt a follow-up
- A trend toward significance usually means the result did not reach the threshold the authors themselves set.
- Associated with is not a causal claim, whatever the press release does with it.
- Well tolerated describes what was recorded, so ask how harms were measured and for how long.
- In a large study invites you to check whether large counts participants, observations, or clicks.
- Robust to specification should be followed by which specifications were tried and which were not.
Common pitfalls
Judging by the journal
Prestige tells you little about any single paper. The design, the sample, and whether the analysis was planned in advance tell you far more than the masthead.
Treating one study as the answer
A single result is one estimate. Ask where it sits relative to existing work and whether it is an outlier before you build anything on it.
Confusing relative and absolute change
A doubling of a rare risk can be a very small absolute change. Ask for the baseline rate before reacting to any percentage.
Reading only the abstract
Abstracts are written to be quotable. Exclusions, limitations, and the difference between primary and secondary outcomes usually live further in.
A four-step interview flow
- 1The claim: what was found, in one sentence and in real units
- 2The setup: who was studied, by what method, and what was decided in advance
- 3The uncertainty: how much could be chance, and what else could explain it
- 4The use: where it applies, where it does not, and what should change as a result