Questions to Ask About Data
Critical questions to evaluate data quality, methodology, and insights before making decisions based on analytics and metrics.
1Where did this data come from, and how was it collected?
Click to see why this works
Where did this data come from, and how was it collected?
Click to see why this works
Why this works
Data source and collection method affect reliability and potential biases.
2What is the sample size, and is it representative?
Click to see why this works
What is the sample size, and is it representative?
Click to see why this works
Why this works
Small or biased samples lead to incorrect conclusions about populations.
3What time period does this data cover?
Click to see why this works
What time period does this data cover?
Click to see why this works
Why this works
Temporal context matters—recent data may differ from historical trends.
4What definitions and assumptions underlie this data?
Click to see why this works
What definitions and assumptions underlie this data?
Click to see why this works
Why this works
How metrics are defined dramatically affects what the numbers actually mean.
5What data is missing or excluded from this analysis?
Click to see why this works
What data is missing or excluded from this analysis?
Click to see why this works
Why this works
What's not shown often matters as much as what is—reveals blind spots.
6How was this data cleaned and processed?
Click to see why this works
How was this data cleaned and processed?
Click to see why this works
Why this works
Data cleaning decisions affect results; transparency prevents manipulation.
7What is the margin of error or confidence interval?
Click to see why this works
What is the margin of error or confidence interval?
Click to see why this works
Why this works
All measurements have uncertainty; knowing bounds prevents overconfidence.
8Are there any known biases in this data?
Click to see why this works
Are there any known biases in this data?
Click to see why this works
Why this works
Selection bias, survivorship bias, and others can skew findings.
9How do outliers affect these results?
Click to see why this works
How do outliers affect these results?
Click to see why this works
Why this works
Extreme values can dramatically impact averages and conclusions.
10What story is this data trying to tell versus what it actually shows?
Click to see why this works
What story is this data trying to tell versus what it actually shows?
Click to see why this works
Why this works
Separates interpretation from facts; reveals potential agenda.
11Can these correlations be explained by confounding variables?
Click to see why this works
Can these correlations be explained by confounding variables?
Click to see why this works
Why this works
Correlation doesn't equal causation; third factors may drive both.
12How sensitive are these conclusions to methodology changes?
Click to see why this works
How sensitive are these conclusions to methodology changes?
Click to see why this works
Why this works
Robust findings hold up under different analytical approaches.
13What other datasets or sources corroborate these findings?
Click to see why this works
What other datasets or sources corroborate these findings?
Click to see why this works
Why this works
Triangulation with multiple sources increases confidence in conclusions.
14Who benefits from this data being interpreted this way?
Click to see why this works
Who benefits from this data being interpreted this way?
Click to see why this works
Why this works
Understanding incentives reveals potential bias in presentation.
15What would it take to prove this conclusion wrong?
Click to see why this works
What would it take to prove this conclusion wrong?
Click to see why this works
Why this works
Falsifiability is key to scientific thinking; what's the counter-evidence?
16How current is this data, and how quickly does it change?
Click to see why this works
How current is this data, and how quickly does it change?
Click to see why this works
Why this works
Stale data in fast-moving environments leads to bad decisions.
17What privacy or ethical concerns exist with this data?
Click to see why this works
What privacy or ethical concerns exist with this data?
Click to see why this works
Why this works
Data collection and use have ethical implications beyond legality.
18How is this data being visualized, and does it mislead?
Click to see why this works
How is this data being visualized, and does it mislead?
Click to see why this works
Why this works
Chart choices, axis scaling, and colors can manipulate perception.
19What benchmarks or context help interpret these numbers?
Click to see why this works
What benchmarks or context help interpret these numbers?
Click to see why this works
Why this works
Numbers without comparison lack meaning—what's good/bad/average?
20What action should we take based on this data?
Click to see why this works
What action should we take based on this data?
Click to see why this works
Why this works
Connects analysis to decisions; data without action is just numbers.
How to Use These Questions
Expert tips and techniques for getting the most out of these questions.
Guide content not available