Questions to Ask About Data
Questions to ask when someone puts a number, a chart, or a dashboard in front of you: how the thing was defined and collected, what got dropped along the way, how much of the result rests on a handful of records, and what decision it is supposed to support.
The questions
Open any question for the note
Where did these numbers come from, and who pulled them?
Why ask it
Where a number came from is the first thing to establish and the thing most often skipped. A figure taken from a dashboard, from a hand-built spreadsheet, or from the source system carries three different degrees of reliability. Ask who ran the query, because that person can answer everything below.
What exactly counts as a user, a sale, or an account in this figure?
Why ask it
Definitions cause more trouble than arithmetic. Active can mean logged in once or used the main feature this week; a sale can be booked, invoiced, or collected. Ask for the definition in one sentence, then check that two people in the room give the same one.
What date range is this, and why that range?
Why ask it
Ranges are chosen, sometimes for sound reasons and sometimes because they flatter. Ask what the preceding window showed. A series that starts immediately after a bad quarter is a presentation decision rather than an analytical one, and it is worth naming as such.
How many rows did you start with, and how many did you drop?
Why ask it
Cleaning is where results are quietly made. Ask for both counts and the rule that removed the difference. If a large fraction disappeared, that exclusion rule deserves at least as much scrutiny as the conclusion it made possible.
Who is missing from this data altogether?
Why ask it
Absence never appears in a chart: customers who already left, people who abandoned the form, anyone without a smartphone, staff who ignored the survey. Ask whose records never entered the system, because that group is often the one a decision will land on hardest.
Is this a sample or everything, and if it's a sample, how was it drawn?
Why ask it
The two need different reading. If it is a sample, ask whether it was random, convenient, or self-selected. Anything built from people who volunteered overrepresents the enthusiastic and the aggrieved, and the quiet middle will simply be absent.
What's the uncertainty on that number?
Why ask it
Every estimate has a range, and a figure quoted to two decimal places implies a precision that rarely exists. If nobody can offer even a rough interval, say out loud that the number is directional, before it hardens into a target in someone's plan.
What does the distribution look like, not just the average?
Why ask it
A mean of forty can come from a crowd around forty or from two clusters at ten and eighty, and those call for opposite actions. Ask for the median and a histogram. When only the mean is offered, that is often because the shape is awkward.
How much of this comes from the largest few records?
Why ask it
Concentration is the norm: one enterprise account, one region, one promotional week. Ask what remains after removing the top one percent. If the finding vanishes, you have learned something about a few rows rather than about the business.
Is this change bigger than the normal week-to-week variation?
Why ask it
This separates signal from noise and deflates more confident charts than any other question. Ask for the historical variation across comparable periods. If the movement sits inside the usual bounce, there may be nothing here that needs explaining.
What else changed at the same time?
Why ask it
Almost nothing moves alone: a price change, a holiday, a release, a competitor's outage, a fix to the tracking code. Ask for a timeline of everything that shifted in that window before accepting a single cause for the movement in the numbers.
Did we decide what we were looking for before or after we looked?
Why ask it
Choosing the hypothesis afterwards makes almost any pattern findable, particularly across dozens of metrics. Ask whether the question was written down first and how many other cuts were tried. Honest analysts will tell you how many things they looked at.
If the opposite conclusion were true, what would this chart look like?
Why ask it
This makes the analysis falsifiable while everyone is still in the room. If no plausible version of the data would have produced a different conclusion, the chart is illustrating a decision already made rather than testing anything.
Why does the y-axis start there?
Why ask it
A truncated axis turns a two percent change into a cliff, and a second axis can imply a relationship the data does not contain. Ask about the scale and any smoothing, then look at the same series plotted from zero before you react to it.
Could someone else reproduce this from the raw source?
Why ask it
Reproducibility is the practical test of everything above. Ask whether the query and every step that reshaped the data are saved where another person could run them. Analysis that exists only inside one spreadsheet cannot be checked and will not survive that person changing jobs.
When was this last refreshed, and does the recent data still change?
Why ask it
Ask the refresh schedule, the lag, and whether late-arriving records rewrite earlier periods. Many dashboards show an incomplete current week that always looks like a decline, and that artifact has started more than one unnecessary crisis meeting.
Does this match the number finance has, and if not, why?
Why ask it
Reconciling against an independently maintained figure is the fastest quality check available. Disagreements almost always come down to a definition or a timing difference, and locating it improves both systems. A chronic unexplained gap means nobody has tried.
What did you expect to see that you didn't?
Why ask it
Analysts notice things they cannot explain and often drop them to keep the summary clean. Asking directly brings the anomaly back, and the anomaly is frequently the most useful thing produced by the whole exercise.
What would have to be true for us to be wrong about this?
Why ask it
Ask for the conditions under which the conclusion fails rather than for a confidence level. Good answers are concrete: if the tracking change on the fourth was wrong, if this cohort is unusual, if the response rate is skewed by region.
What decision changes if this number is off by twenty percent?
Why ask it
This sizes the entire discussion. If the action is the same either way, stop analyzing and act. If a modest error flips the decision, the analysis is not yet finished, and that is worth saying before anyone commits budget to it.
Are we allowed to hold this data, and would the people in it expect us to?
Why ask it
Legality and expectation are separate tests, and the second is what causes reputational damage. Ask what people were told when it was collected, whether records can be re-linked to individuals, and who has access now. Anonymized often means less than it sounds.
How to interrogate an analysis without stalling it
Practical guidance for the conversation itself
How to ask in the room
Settle definitions before methods
Most disputes that look statistical are two people using one word differently. Agree what counts as a user, a sale, or a churn before anyone argues about the direction of the trend.
Ask for the numerator and denominator separately
Rates conceal movement. A rising conversion rate on falling traffic is a different situation from a rising rate on steady traffic, and only the two raw counts tell them apart.
Make questioning cheap
If asking about methodology reads as an accusation, people stop volunteering caveats. Ask the same questions of results you like, and say plainly that you are checking the work rather than the person who did it.
Write down the decision first
Agree what action each possible result implies before looking at the analysis. That is what stops a number from being reinterpreted until it supports the plan somebody already had.
Fast checks before you accept a chart
- Read the axis labels, the units, and the range before the shape of the line.
- Find the denominator.
- Look for the point where tracking changed, a release shipped, or a system went down.
- Compare the mean against the median where both exist.
- Ask what the same chart looked like a quarter ago.
- Check whether the most recent period is complete.
Common pitfalls
Mistaking precision for accuracy
Three decimal places say nothing about whether the measurement was well designed. A confidently wrong figure travels much further than an honest range does.
Accepting a story built on one cut
If a finding appears in only one segment, one week, or one chart type, it is a hypothesis. Ask what happens under two other reasonable cuts before it becomes a fact in a deck.
Reading significance as importance
A statistically significant effect can be far too small to act on, and a non-significant one can still be the best guide available. Ask about effect size and cost alongside the test.
Letting the dashboard be the analysis
A dashboard shows what somebody once decided to measure. The question you actually have is usually not on it, and answering it means going back to the source data.
A five-minute review of any analysis
- Ask what decision this is meant to inform.
- Get the definitions and the date range in plain words.
- Ask what was excluded, and who never appeared in the data at all.
- Ask how much of the effect comes from the largest handful of records.
- Ask what would have to be true for the conclusion to be wrong, and whether anyone else could reproduce it.