Questions to Ask When Analyzing Data
For analysts, students and the managers who commission the work, these are the data analysis questions to ask from the first brief to the final report, whether the job is one spreadsheet or a full data analytics project. The six groups follow the order of the work: the decision behind the request, the data itself, the method, the checks meant to break the result, the reporting, and what happens after someone acts on it. Most are worded to be said aloud in a kickoff or a review meeting, and the group a deadline tends to squeeze out, Checks, is the one that catches a wrong answer before it is presented.
Want questions from the whole vault instead? Try the random question generator.
The questions
Each question, and why to ask it
The decision
What decision will this analysis inform, and who makes it?
Why ask it
If nobody can name the decision or its owner, the work is exploration and should be scoped as such. Answers like "leadership wants visibility" usually mean the output gets read once and shelved.
Can we put the question in one sentence that a number or a yes or no would answer?
Why ask it
"Look into churn" is a topic. "Did the March price change raise cancellations among annual subscribers?" is a question, and it already tells you which table, which dates and which comparison you need.
What form should the answer take: one number, a threshold, or a ranking?
Why ask it
Naming the shape of the answer up front prevents weeks of dashboard building when one figure was wanted. If the requester cannot say, ask what they would do differently at three different values.
What do we expect to find, and what would surprise us?
Why ask it
Write the guess down with a date before any query runs. A result that matches it deserves the same scrutiny as one that does not, and without the written guess everyone remembers having predicted whatever turned up.
What are we comparing this against?
Why ask it
Up eight percent means nothing until you know up from what: last month, the same month a year ago, the plan, or a group that was left alone. Agree which before the result is in, so the comparison cannot be picked to suit it.
Has someone already answered this, here or elsewhere?
Why ask it
Search the shared drive, the old tickets and the retired dashboards before starting. An earlier attempt that holds up spares you the work, and one that was abandoned is useful too: whatever stopped it is likely to be the first obstacle you meet.
When is the answer needed, and how accurate does it have to be by then?
Why ask it
A rough figure on Thursday and an exact one next month are different projects. If the decision gets made on Friday either way, plan the analysis that can be finished and checked by Thursday, and say what it leaves out.
What is out of scope, and who has agreed to that?
Why ask it
List the neighboring questions you will not answer: other regions, other products, the why behind the what. Put the list in the brief, because a question added halfway through tends to arrive without any extra time.
Who has to accept the result before anything changes?
Why ask it
The person who asked is not always the person who can block. If finance or legal will challenge the definitions, show them the plan now, when changing a definition costs an hour and not a rerun.
How much is riding on this, in money, hours or risk?
Why ask it
Even a rough figure shows how much analyst time the question deserves and where it sits among the other requests in the queue. When nobody will hazard a guess, that says something about its priority too.
The data
How was this data generated, and who or what is missing from it?
Why ask it
A dataset holds whoever turned up to be recorded: customers who stayed, people who answered the survey, sensors that were switched on. Ask who never made it into the file, because the numbers describe the people in it and those are not always the people in the question.
What does each field actually mean, and who can confirm it?
Why ask it
A column name is a label somebody chose years ago. "signup_date" may be the day the account was created, the day it was verified, or the day the record was migrated from an older system, and only the person who maintains the pipeline can say which.
What does one row represent, and what time window are we using?
Why ask it
One row per order and one row per customer are different tables, even when they share every column name. Mix the grain or the window and you get figures that are each correct and together meaningless, such as a 28-day total set over a 30-day one.
Are we measuring the thing we care about, or a stand-in for it?
Why ask it
Clicks stand in for interest, tickets closed for problems solved, test scores for learning. Name the gap between the stand-in and the real thing, and ask how someone could move one without moving the other.
Did the way this was recorded change during the period we are studying?
Why ask it
A new form, a renamed category, a tracking tag added in June: each one draws a step in the chart that looks like a change in behavior. Ask whoever owns the system for its change history and mark those dates on every time series.
Where are the missing values, outliers, and duplicates, and why are they there?
Why ask it
The reason matters more than the count. Nulls that start at a schema change, or duplicates in only one region, point to a pipeline fault rather than something to impute away.
Which tables are we joining, and what happens to the rows that do not match?
Why ask it
Count rows before and after every join. A total that grows means a key was not unique and something is being counted twice. A total that shrinks means records were dropped without a word, and it matters a great deal which ones.
Is the latest period complete, or are records still arriving?
Why ask it
Returns, refunds and late-synced entries can keep rewriting the most recent weeks. If the last point on the chart always dips, cut the series at the last period you know has settled and say so on the chart.
Is the sample large enough for the smallest group we plan to report on?
Why ask it
Ten thousand rows overall can still mean eighteen in the segment the requester cares about most. Work out the smallest cell before promising a breakdown, and agree a minimum count below which you will not publish a figure.
Can we reproduce one number the organization already reports?
Why ask it
Rebuild a familiar figure, such as last quarter's revenue or headcount, from your own extract first. A mismatch means there is a definition or a filter you do not understand yet, and that is cheaper to learn before you produce a new number than after.
What cleaning rules are we applying, and where are they written down?
Why ask it
Dropping test accounts, capping extreme values and filling blanks are each a decision that moves the answer. Keep them in one commented script or one page, so a reviewer can disagree with a specific rule and you can show what changes without it.
What privacy, consent, or security constraints apply to this data?
Why ask it
Ask which fields are restricted, where extracts may live, and how long they can be kept. The rules differ by country and by employer, so get the answer from whoever handles data protection where you work. Small aggregates can still identify people when a segment has three members in it.
Method
Are we describing what happened, predicting what will, or claiming why?
Why ask it
Each needs a different standard of evidence. A description needs clean counts, a forecast needs a test on data it has not seen, and a claim about cause needs a comparison group or an experiment, so settle which one you are making before you pick a tool.
What is the simplest approach that would be good enough?
Why ask it
Complex models cost more in review, monitoring, and explanation than in fitting. If a segment average answers the question, the sophisticated version has to beat it by enough to earn its upkeep.
What assumptions does this method require, and do they hold here?
Why ask it
Many tests treat every row as independent, which fails quietly when one customer contributes fifty of the orders. For each assumption, ask what the damage is when it does not hold: a wider range can be lived with, a reversed sign cannot.
Which groups will we break the result down by, and why those?
Why ask it
Take the cuts from the decision: the regions that can be priced separately, the plans that can be changed. Slicing by every available column until something stands out is how a team ends up with a finding that is gone by next month.
Should we summarize with the mean, the median, or the whole distribution?
Why ask it
Plot the histogram before choosing. Order values, wait times and incomes often have a long right tail, where a few large cases pull the mean well above what a typical case looks like and the median or a set of percentiles gives the fairer picture.
Should this be a count or a rate, and a rate per what?
Why ask it
Complaints rising by a fifth is bad news in a flat month and good news in a month when orders doubled. Choose the denominator the decision turns on, per order, per customer or per store, and keep it the same on both sides of every comparison.
How are we handling seasonality, trend, and calendar quirks?
Why ask it
This March against last March is a fairer comparison than this March against February, and even then a moved holiday, a leap day or a fifth weekend can shift a monthly total. Fix the comparison period and any adjustment before reading the result.
If we are predicting, how will we test it: a holdout, cross-validation, or a backtest?
Why ask it
Watch for validation that leaks the future into the past: random splits on time series, or features computed after the outcome. Both look excellent in testing and fail in the first live week.
How many comparisons are we running, and how will we account for that?
Why ask it
Test twenty unrelated metrics at a five percent threshold and, on average, one will look significant by chance alone. Name the primary metric in advance and treat the rest as exploratory, or apply a correction and say which one.
Is the effect large enough to matter, and not only large enough to detect?
Why ask it
With enough rows, a difference of a tenth of a percent can pass a significance test. Put the effect in the units the decision is made in, whether dollars, hours or customers, and set it beside what acting on it would cost.
Could someone else rerun every step from the raw data to the final number?
Why ask it
A figure produced by hand edits in a spreadsheet cannot be checked or repeated, including by you in three months. Aim for one script, notebook or documented workbook that runs top to bottom from the raw extract, with the date of that extract recorded.
Checks
What else could explain this pattern?
Why ask it
Have the team name at least two rival explanations before anyone defends the favorite: the season, a marketing push, a change in how events are logged. Each rival points to a check that can be run the same afternoon.
Could the cause run the other way?
Why ask it
Customers who contact support often may cancel more, which does not show that support drives them away: people who are already unhappy are the ones who call. Check which came first in time, and whether a third thing sits behind both.
Did we pick these cases because they were unusually high or low?
Why ask it
The ten worst stores of last quarter will mostly look better next quarter whatever is done to them, because part of any terrible quarter is bad luck. Before crediting the program, compare them with equally poor stores that were left alone.
How much does this number move on its own in an ordinary week?
Why ask it
Chart the metric for the year before anything changed. A three-point rise after the launch is no finding if the line has drifted five points up and down all year with nobody touching it.
Does the answer change if we move the cutoffs, the dates, or the definitions?
Why ask it
Rerun it with the window starting a month earlier, the outlier cap set a little higher and the stricter definition of an active user. A conclusion that flips on one of those rests on a choice you made, and the reader should be told which one.
Does the result hold across segments, or is one group carrying it?
Why ask it
Recompute the headline with the largest customer, the biggest region and the busiest week removed, one at a time. If the effect disappears when one of them goes, the finding is about that customer, region or week, and the report should say so.
What do we see when we run the same test where nothing should have happened?
Why ask it
Apply the method to a period before the change, or to a group the change never reached. If it reports an effect there too, it is picking up something other than what you set out to study.
Does the headline number survive a check by hand?
Why ask it
Pull twenty raw records and work the figure out on paper, or multiply two numbers you already trust and see whether you land in the same range. Cents read as dollars and a filter applied twice are the kind of error this finds in ten minutes.
Which results did we leave out, and why?
Why ask it
Keep a list of every cut and model you tried, including the ones that showed nothing. A reader weighs one positive result very differently once they know it was the single one of twelve attempts that worked.
Who has tried to break this, other than the person who built it?
Why ask it
Hand the query and the draft to a colleague with one instruction: find the mistake. Authors read what they meant to write, so it takes a second pair of eyes to spot the inverted filter or the mislabeled axis.
How much uncertainty is in this number, and how should we express it?
Why ask it
Ranges change decisions that point estimates do not. If the interval straddles the threshold for action, the honest recommendation is to wait or gather more, not to quote the midpoint.
If this is wrong, which way will it be wrong, and who pays for it?
Why ask it
Overestimating demand leaves stock on the shelf, underestimating it loses sales, and the two rarely cost the same. Naming the expensive direction tells you which way to lean when the estimate is close.
Reporting
Who is the audience, and what do they already believe?
Why ask it
A result that confirms the room's view gets accepted with less proof than it should. One that contradicts it needs the method up front, because the first reaction will be to question the data.
How would we explain this to someone who will not read the appendix?
Why ask it
If the method cannot be stated in two plain sentences, it will be either ignored or misquoted. The real test is whether the audience can repeat the caveat, not just the number.
What is the one-slide version a decision-maker needs?
Why ask it
Headline, one chart, the action being asked for. Everything else is appendix, and a headline that needs three caveats to stay true is the wrong headline.
Which chart shows this most honestly?
Why ask it
Start bar charts at zero, label the units, and put the comparison the decision depends on side by side in one panel. Then show the draft to someone outside the project and ask what they think it says. If their reading differs from yours, redraw it.
What are the limitations, and where in the report do we state them?
Why ask it
Put the two that could change the conclusion next to the headline, in plain words: which population this covers and what was not measured. A full list buried on the last page protects the analyst and does nothing for the reader.
What are we recommending, and what would change our mind?
Why ask it
Findings describe and a recommendation commits. State the action, then the specific evidence that would reverse it, such as a retention figure below a named level, so the decision-maker knows how firm the ground is.
At what number do we act, and who takes the first step?
Why ask it
A finding with no threshold and no owner produces agreement in the room and nothing afterward. Settle both while the people are still present.
What question will they ask that we cannot answer yet?
Why ask it
Rehearse the obvious follow-ups: what about the other region, what about last year, how much will it cost. For each, have either a number or an honest "we did not look at that, and here is how long it would take".
Afterwards
How will we find out whether the decision worked?
Why ask it
Before the change ships, write down the metric, the date you will look, and the result you would count as success. Choosing those afterward lets any outcome be described as a win.
What would tell us this finding has gone stale?
Why ask it
A result describes the customers, prices and product of the months it was drawn from. Name the events that would retire it, such as a new pricing plan or a shift in who signs up, and put a review date on any model or dashboard that keeps using it.
Does this need to be run again, and who owns it if so?
Why ask it
A one-off answer and a monthly report need different amounts of engineering. If it will recur, name the owner and schedule it now, or it lives on as a spreadsheet that one person updates from memory.
Where do the code, the extract and the write-up live, and who can find them?
Why ask it
Months later someone will quote the number and someone else will ask where it came from. One folder holding the query, the dated extract and the final document, linked from the report itself, settles that in a minute.
What data do we wish we had been collecting?
Why ask it
The gap that hurt is often small: one timestamp nobody logged, one field left as free text. File the request with whoever owns the system now, since nothing can go back and record what was never captured.
What did this analysis raise that deserves its own look?
Why ask it
The odd segment, the unexplained spike and the join that lost more rows than expected all got parked to meet the deadline. Write them down with a line on why each might matter, so the next project starts from a list and not from a blank page.
What would we do differently next time?
Why ask it
Spend fifteen minutes on this while the work is fresh: where the time went, which check caught the real error, what the requester wishes they had asked for. Those notes are worth most to the next analyst handed a similar brief.
Turning an analysis into a decision
Practical guidance for the conversation itself
Working practices that hold up under review
Take the six groups in order
The tags follow the work: The decision, The data, Method, Checks, Reporting, Afterwards. A small job needs only two or three questions from each, but it needs something from all six. The groups that get skipped are The decision, because the request seemed clear, and Checks, because the deadline arrived.
If you commission analysis and do not do it yourself
The questions to ask about data analytics work that someone else is doing for you sit mostly in The decision and Reporting, plus one from Checks: who has tried to break this. Those are the places where what the requester says changes what the analyst builds.
Write the recommendation first
Draft the sentence you expect to conclude with before you query anything, then treat the analysis as an attempt to falsify it. It exposes early whether the available data can settle the question at all.
Fix the comparison before you see the result
Agree the baseline, the segment definitions, and the success threshold in writing with whoever asked. Choosing them afterward is how honest people produce motivated conclusions.
Show the denominator
Many arguments about a chart turn out to be arguments about who was counted. Put the population definition and sample size on the same slide as the number.
Keep a decision log
Record each judgment call: rows excluded, outliers capped, windows chosen. Six weeks later this is the difference between defending a result and rebuilding it.
What to check before modeling anything
- Row counts by day, looking for gaps, spikes, and the date a pipeline changed
- Distribution of every key field, not just the mean, so bimodality and default values show up
- Join keys: how many rows fail to match, and whether the failures cluster in one segment
- Timestamps: what timezone, and whether event time or ingestion time is recorded
- Definitions of any metric someone else built, read from the query rather than the label
Failures that survive peer review
Leakage that flatters the model
A feature computed after the outcome, or a random split on time-ordered data, produces excellent validation and immediate live failure. Check the timestamp of every feature against the prediction moment.
Survivor-only populations
Cohorts filtered on people who stayed long enough to be measured will show whatever loyalty looks like. Define the cohort at entry, not at outcome.
Simpson reversals
An effect that holds in aggregate and reverses in every segment usually means the segments have different sizes and different base rates. Report both, and lead with the one that matches the decision.
Precision theater
Reporting a percentage to two decimals on a sample of forty invites false confidence. Round to the precision the data supports and say why.
Handing off so the work gets used
Structure for a decision memo
- 1The decision on the table and its owner
- 2The recommendation, in one sentence
- 3The two or three numbers that support it, with ranges
- 4What would change the recommendation, and the threshold at which it changes
- 5What we did not check, and why that is acceptable here
- 6The first action, its owner, and its date
Before you present
- 1Ask a colleague to argue the opposite conclusion from your own charts
- 2Rerun the headline number from raw data by a different route
- 3State the uncertainty out loud and check it does not change the recommendation
- 4Prepare the answer to "what would you do with twice the time"