Questions to Ask About Data
Questions to Ask About Data
Critical questions to evaluate data quality, methodology, and insights before making decisions based on analytics and metrics.
1Where did this data come from, and how was it collected?
Where did this data come from, and how was it collected?
Data source and collection method affect reliability and potential biases.
2What is the sample size, and is it representative?
What is the sample size, and is it representative?
Small or biased samples lead to incorrect conclusions about populations.
3What time period does this data cover?
What time period does this data cover?
Temporal context matters—recent data may differ from historical trends.
4What definitions and assumptions underlie this data?
What definitions and assumptions underlie this data?
How metrics are defined dramatically affects what the numbers actually mean.
5What data is missing or excluded from this analysis?
What data is missing or excluded from this analysis?
What's not shown often matters as much as what is—reveals blind spots.
6How was this data cleaned and processed?
How was this data cleaned and processed?
Data cleaning decisions affect results; transparency prevents manipulation.
7What is the margin of error or confidence interval?
What is the margin of error or confidence interval?
All measurements have uncertainty; knowing bounds prevents overconfidence.
8Are there any known biases in this data?
Are there any known biases in this data?
Selection bias, survivorship bias, and others can skew findings.
9How do outliers affect these results?
How do outliers affect these results?
Extreme values can dramatically impact averages and conclusions.
10What story is this data trying to tell versus what it actually shows?
What story is this data trying to tell versus what it actually shows?
Separates interpretation from facts; reveals potential agenda.
11Can these correlations be explained by confounding variables?
Can these correlations be explained by confounding variables?
Correlation doesn't equal causation; third factors may drive both.
12How sensitive are these conclusions to methodology changes?
How sensitive are these conclusions to methodology changes?
Robust findings hold up under different analytical approaches.
13What other datasets or sources corroborate these findings?
What other datasets or sources corroborate these findings?
Triangulation with multiple sources increases confidence in conclusions.
14Who benefits from this data being interpreted this way?
Who benefits from this data being interpreted this way?
Understanding incentives reveals potential bias in presentation.
15What would it take to prove this conclusion wrong?
What would it take to prove this conclusion wrong?
Falsifiability is key to scientific thinking; what's the counter-evidence?
16How current is this data, and how quickly does it change?
How current is this data, and how quickly does it change?
Stale data in fast-moving environments leads to bad decisions.
17What privacy or ethical concerns exist with this data?
What privacy or ethical concerns exist with this data?
Data collection and use have ethical implications beyond legality.
18How is this data being visualized, and does it mislead?
How is this data being visualized, and does it mislead?
Chart choices, axis scaling, and colors can manipulate perception.
19What benchmarks or context help interpret these numbers?
What benchmarks or context help interpret these numbers?
Numbers without comparison lack meaning—what's good/bad/average?
20What action should we take based on this data?
What action should we take based on this data?
Connects analysis to decisions; data without action is just numbers.
Want to learn more?
How to Use These Questions
Want to learn more?
How to Use These Questions
Guide content not available