Questions to Ask When Analyzing Data
Questions for analysts and the people who commission analysis, covering how to frame the decision, audit the data, test assumptions, express uncertainty, and hand off a recommendation someone can act on.
The questions
Open any question for the note
What decision will this analysis inform, and who makes it?
Why ask it
If nobody can name the decision or its owner, the work is exploration and should be scoped as such. Answers like "leadership wants visibility" usually mean the output gets read once and shelved.
What is the minimal useful output: a number, a threshold, or a ranking?
Why ask it
Naming the shape of the answer up front prevents weeks of dashboard building when one figure was wanted. If the requester cannot say, ask what they would do differently at three different values.
How was this data generated, and who or what is missing from it?
Why ask it
Records tend to capture the people who were already engaged, already surveyed, already surviving. The answer tells you which population the numbers actually describe, which is rarely the one in the question.
What is the unit of analysis, and what time window are we using?
Why ask it
Mismatched units and windows produce figures that are individually correct and jointly meaningless: sessions divided by users, a 28-day numerator over a 30-day denominator.
Where are the missing values, outliers, and duplicates, and why are they there?
Why ask it
The reason matters more than the count. Nulls that start at a schema change, or duplicates in only one region, point to a pipeline fault rather than something to impute away.
What baseline or prior should we compare this against?
Why ask it
A result with no comparison point can be spun either direction. Agreeing the baseline before you see the answer is what stops the baseline being chosen to fit the conclusion.
What assumptions does this method require, and do they hold here?
Why ask it
Independence, stationarity, and a stable population are the ones that quietly fail. Ask what specifically breaks under each, so you know whether the risk is a wider interval or a reversed sign.
What else could explain this pattern?
Why ask it
Forcing two alternatives into the open catches seasonality, a marketing push, and a logging change before any of them get presented as a finding.
What sensitivity checks will we run before we believe it?
Why ask it
A result that survives a different time slice, a dropped segment, and a different cutoff is worth presenting. One that holds only on the slice you tried first is a coincidence.
What is the simplest approach that would be good enough?
Why ask it
Complex models cost more in review, monitoring, and explanation than in fitting. If a segment average answers the question, the sophisticated version has to beat it by enough to earn its upkeep.
How will we validate this: holdout, cross-validation, or backtest?
Why ask it
Watch for validation that leaks the future into the past: random splits on time series, or features computed after the outcome. Both look excellent in testing and fail in the first live week.
How much uncertainty is in this number, and how should we express it?
Why ask it
Ranges change decisions that point estimates do not. If the interval straddles the threshold for action, the honest recommendation is to wait or gather more, not to quote the midpoint.
If this is wrong, which way will it be wrong, and who is affected?
Why ask it
Overestimating and underestimating rarely cost the same. Naming the expensive direction tells you which way to err and whether a human check belongs in the loop.
How would we explain this to someone who will not read the appendix?
Why ask it
If the method cannot be stated in two plain sentences, it will be either ignored or misquoted. The real test is whether the audience can repeat the caveat, not just the number.
What value triggers action, and who takes the first step?
Why ask it
A finding with no threshold and no owner produces agreement in the room and nothing afterward. Settle both while the people are still present.
What data would we want if we did this again?
Why ask it
The answers become an instrumentation backlog. The gaps that hurt most are usually one missing timestamp or one unlogged field, cheap to fix before the next cycle and impossible to fix retroactively.
What privacy, consent, or security constraints apply to this data?
Why ask it
Ask which fields are restricted, where extracts may live, and how long they can be kept. Small aggregates can still identify people when a segment has three members in it.
What is acting on this worth, in money, hours, or risk avoided?
Why ask it
Even a rough figure ranks this against everything else competing for the same engineering time. If nobody will estimate it loosely, that is itself an answer about priority.
How will we know when this stops being true?
Why ask it
Models and dashboards decay quietly. Agree now on the metric to watch, the level that raises an alarm, and the person who looks at it, or nobody notices for a quarter.
What is the one-slide version a decision-maker needs?
Why ask it
Compressing to a single slide exposes whether you have a recommendation or only findings. If the slide needs three caveats to stay honest, the analysis is not finished.
Turning an analysis into a decision
Practical guidance for the conversation itself
Working practices that hold up under review
Write the recommendation first
Draft the sentence you expect to conclude with before you query anything, then treat the analysis as an attempt to falsify it. It exposes early whether the available data can settle the question at all.
Fix the comparison before you see the result
Agree the baseline, the segment definitions, and the success threshold in writing with whoever asked. Choosing them afterward is how honest people produce motivated conclusions.
Show the denominator
Most disputes about a chart are actually disputes about who was counted. Put the population definition and sample size on the same slide as the number.
Keep a decision log
Record each judgment call: rows excluded, outliers capped, windows chosen. Six weeks later this is the difference between defending a result and rebuilding it.
What to check before modeling anything
- Row counts by day, looking for gaps, spikes, and the date a pipeline changed
- Distribution of every key field, not just the mean, so bimodality and default values show up
- Join keys: how many rows fail to match, and whether the failures cluster in one segment
- Timestamps: what timezone, and whether event time or ingestion time is recorded
- Definitions of any metric someone else built, read from the query rather than the label
Failures that survive peer review
Leakage that flatters the model
A feature computed after the outcome, or a random split on time-ordered data, produces excellent validation and immediate live failure. Check the timestamp of every feature against the prediction moment.
Survivor-only populations
Cohorts filtered on people who stayed long enough to be measured will show whatever loyalty looks like. Define the cohort at entry, not at outcome.
Simpson reversals
An effect that holds in aggregate and reverses in every segment usually means the segments have different sizes and different base rates. Report both, and lead with the one that matches the decision.
Precision theater
Reporting a percentage to two decimals on a sample of forty invites false confidence. Round to the precision the data supports and say why.
Handing off so the work gets used
Structure for a decision memo
- 1The decision on the table and its owner
- 2The recommendation, in one sentence
- 3The two or three numbers that support it, with ranges
- 4What would change the recommendation, and the threshold at which it changes
- 5What we did not check, and why that is acceptable here
- 6The first action, its owner, and its date
Before you present
- 1Ask a colleague to argue the opposite conclusion from your own charts
- 2Rerun the headline number from raw data by a different route
- 3State the uncertainty out loud and check it does not change the recommendation
- 4Prepare the answer to "what would you do with twice the time"