Questions to Ask About Data
For anyone handed a number, a chart, or a dashboard and expected to act on it: managers, students, and analysts reviewing someone else's work. The six groups run in the order a review goes: where the data came from, what was left out, what to ask about a graph before you trust its shape, how solid the number is, what changes with big data that no person has read through, and the decision it is all supposed to support.
Want questions from the whole vault instead? Try the random question generator.
The questions
Each question, and why to ask it
Source
Where did these numbers come from, and who pulled them?
Why ask it
Where a number came from is the first thing to establish and the thing most often skipped. A figure read off a dashboard, one copied from a hand-built spreadsheet, and one queried from the source system are not equally reliable. Whoever ran the query is also the person who can answer most of what you ask next.
What exactly counts as a user, a sale, or an account in this figure?
Why ask it
Definitions cause more trouble than arithmetic. Active can mean logged in once or used the main feature this week; a sale can be booked, invoiced, or collected. Ask for the definition in one sentence, then check that two people in the room give the same one.
What date range is this, and why that range?
Why ask it
Ranges are chosen, sometimes for sound reasons and sometimes because they flatter. A series that starts immediately after a bad quarter is a presentation decision rather than an analytical one, so have the window widened by a year and see whether the story survives.
Was this entered by a person or logged automatically by a system?
Why ask it
Hand-entered data bends toward whatever the person entering it is judged on: close dates slide to the end of the quarter, and the default option in a dropdown gets picked far too often. Logged data has a different weakness, which is that it silently stops when a tag breaks. Knowing which kind you have tells you which error to look for.
Is this measured directly, or is it estimated, self-reported, or modelled?
Why ask it
A count of invoices, a survey answer about spending, and a forecast filled in for months that have not closed can sit in one column looking identical, and only the person who built the table knows which cells are which. An estimate is not a problem as long as its label travels with the number onto the slide.
Has the way this is counted changed since the series began?
Why ask it
A new tracking tool, a renamed category, or two merged systems will put a step in the line that looks like a change in behaviour. Ask for the dates of every change in method and mark them on the chart. A jump that lands exactly on one of those dates is probably the method and not the customers.
When was this last refreshed, and does the recent data still change?
Why ask it
Three things matter here: the refresh schedule, the lag, and whether late-arriving records rewrite earlier periods. A dashboard that plots the current, unfinished week beside full ones shows a drop at its right-hand edge every time, and the drop fills in once the week closes.
Does this match the number finance has, and if not, why?
Why ask it
Finance stands in here for any figure kept independently: the ledger, the billing system, the official return. A disagreement usually traces to a definition or a timing difference, and finding it tidies up both sides. A gap that has gone unexplained for months suggests nobody has tried.
Left out
How many rows did you start with, and how many did you drop?
Why ask it
Cleaning is where results are quietly made. Ask for both counts and the rule that removed the difference. If a large fraction disappeared, that exclusion rule deserves at least as much scrutiny as the conclusion it made possible.
Who is missing from this data altogether?
Why ask it
Absence never appears in a chart: people who never signed up, those who abandoned the form halfway, anyone without a smartphone, staff who ignored the survey. The records that never entered the system often belong to the group a decision will land on hardest.
Is this a sample or everything, and if it's a sample, how was it drawn?
Why ask it
A full count has no sampling error, while a sample is only as good as the way it was picked: at random, from whoever was nearest, or from whoever chose to take part. Anything built from volunteers overrepresents the enthusiastic and the aggrieved, and the quiet middle goes missing.
What happened to the blanks: were they dropped, filled in, or counted as zero?
Why ask it
Each choice gives a different average. A blank income treated as zero drags the mean down, and dropping every row with a blank can quietly remove the newest customers, whose records are the least complete. The fields with the most gaps are the ones to check first.
Could the same person or transaction be in here more than once?
Why ask it
Duplicates come from two systems feeding one table, from a customer with two email addresses, and from retried payments. They inflate counts unevenly, since heavy users pick up more duplicate records than light ones. Ask what makes a record unique and how that was checked.
Were test accounts, staff, and automated traffic taken out?
Why ask it
Internal testing before a launch and a crawler hitting one page can each look like a surge of interest. The filter matters less than the date it was last updated: a bot that arrived last month is not on a list written last year.
If this came from a survey, how many people were asked and how many answered?
Why ask it
A result from four hundred responses reads differently once you learn that twenty thousand people were invited. Ask how those who answered differ from those who did not on anything known about both: age, tenure, plan, region.
Are we looking only at the ones that lasted long enough to be counted?
Why ask it
Averages taken over current customers, surviving funds, or students who finished the course leave out everyone who quit along the way, and the people who quit tend to have had the worse experience. The honest version of the figure begins with everyone who started.
The graph
What does each axis measure, and in what units?
Why ask it
Read this before the shape of the line. Dollars or thousands of dollars, per day or per month, calendar year or fiscal year: a wrong guess about any of them changes the story. If an axis has no label, that is the first thing to ask for.
What does one bar, point, or line on this graph stand for?
Why ask it
A dot can be a person, a store, a week, or an average of thousands of people, and the right reading depends on which. A scatter of thirty dots where each one is a country says nothing certain about the individuals living in them.
Does the picture show what the title says it shows?
Why ask it
Cover the headline, describe what you see in one sentence, then compare the two. A title such as 'Engagement is surging' does much of the reader's interpreting in advance, and the graph underneath sometimes shows something milder.
Why does the y-axis start there?
Why ask it
A truncated axis can make a two percent change look like a cliff. On a bar chart, where the eye compares lengths, a baseline above zero misstates every comparison on the page. A line chart has more excuse to zoom in, but the same series plotted from zero is still worth a look before you react.
Are there two vertical axes, and how were the two scales chosen?
Why ask it
With one scale on the left and another on the right, the point where two lines cross or seem to move together is set by whoever picked the ranges. Slide one scale and the apparent relationship changes. Ask to see each series on a chart of its own.
Is this showing counts, percentages, or a running total?
Why ask it
A cumulative chart can only go up, so its rise proves nothing; what matters is whether it is getting steeper or flatter. A percentage chart hides whether the whole grew or shrank. Having the same data redrawn in the other form is the quickest cure for either.
Has the line been smoothed or averaged, and over how many periods?
Why ask it
A rolling average removes noise and also delays every turn, so a seven-day average shows a change days after it began and a twelve-month one can bury a bad quarter. The raw points plotted faintly underneath the smooth line give you both views at once.
What is this being compared against: last year, a target, or another group?
Why ask it
A line by itself cannot be good or bad. The same figure looks strong against last year and weak against the plan, and whoever made the graph chose which comparison you see. It is fair to wonder how the two they left off would look.
Which categories were folded into 'Other', and how big is it?
Why ask it
Grouping is a judgment call that changes the ranking. If 'Other' is the second largest slice, the chart is hiding its most interesting part, and the named categories may have been split or combined in a way that puts one of them first.
How many cases are behind the smallest bar?
Why ask it
A satisfaction score of ninety percent means little when it comes from ten respondents in one office. A chart that prints the count under every bar answers this for you; where it does not, treat a bar built on a few dozen cases as a rough indication, however exact it looks beside the others.
Why this type of chart, and can I see the table behind it?
Why ask it
Pie charts make close values hard to compare, stacked areas make every layer but the bottom one hard to read, and maps give large empty regions more ink than small crowded ones. The table behind the picture has none of those problems, and nobody should mind being asked for it.
What happened at this spike, and is it real or a recording error?
Why ask it
The point that sticks out is where the eye goes first, and it has two possible stories: something happened that day, or something went wrong with the counting. A file loaded twice, a one-off bulk order, and a holiday can all draw the same shape. Find out which it was before the spike is explained to the room or trimmed off as an outlier.
How solid
What's the uncertainty on that number?
Why ask it
Every estimate has a range, and a figure quoted to two decimal places implies a precision that rarely exists. If nobody can offer even a rough interval, say out loud that the number is directional, before it hardens into a target in someone's plan.
What does the distribution look like, not just the average?
Why ask it
A mean of forty can come from a crowd around forty or from two clusters at ten and eighty, and those call for opposite actions. Ask for the median and a histogram. When only the mean is offered, that is often because the shape is awkward.
How much of this comes from the largest few records?
Why ask it
A handful of records often carries the total: one enterprise account, one region, one promotional week. Have the figure rerun without the top one percent. If the finding vanishes, you have learned something about a few rows and not about the business.
Is this change bigger than the normal week-to-week variation?
Why ask it
Every series bounces from week to week with nothing behind it, and a move has to clear that bounce before it means anything. The range the figure covered over the last dozen comparable periods is the yardstick. If this change sits inside it, there may be nothing here that needs explaining.
What is this a percentage of?
Why ask it
A conversion rate can be per visitor, per visit, or per person who reached the checkout, and the three can be far apart. Get the top and the bottom of the fraction as two raw counts. A rate that improved because the bottom number shrank is a different event from one where the top number grew.
Is that increase relative or absolute, and what was the starting figure?
Why ask it
Going from two percent to three is a rise of one point and also a rise of fifty percent, and people quote whichever suits their case. With the before and the after as plain numbers, you can do the subtraction yourself.
What else changed over the same period?
Why ask it
Almost nothing moves alone: a price change, a holiday, a release, a competitor's outage, a fix to the tracking code. A timeline of everything that shifted in that window, laid beside the chart, comes before accepting any single cause.
Does the pattern hold inside each group, or only when they are added together?
Why ask it
A total can rise while every segment falls, simply because the mix shifted toward a segment with a higher baseline. Before believing the combined line, ask for the result split by the two or three groupings that matter most, such as region, plan, or new against returning.
Does one of these cause the other, or do they only move together?
Why ask it
Two lines that rise together may share a third driver, such as the season or the growth of the business, or the cause may run the opposite way from the one assumed. Ask what was done to rule those out. If the answer is nothing, 'is associated with' is the most the slide can honestly say.
Did we decide what we were looking for before or after we looked?
Why ask it
Choosing the hypothesis afterwards makes almost any pattern findable, particularly across dozens of metrics. What counts is whether the question was written down first and how many other cuts were tried, and an honest analyst will tell you that number without being pushed.
What did you expect to see that you didn't?
Why ask it
Analysts notice things they cannot explain and often drop them to keep the summary clean. Asking directly brings the anomaly back, and the anomaly is frequently the most useful thing produced by the whole exercise.
Big data
Is this dataset big because it covers everyone, or only because it has a lot of rows?
Why ask it
Size does not fix a slanted sample. Ten million records from the users of one app describe those users with great precision and everyone else not at all. The check is to name the population the data is meant to stand for and compare the two on something known about both, such as age or region.
Was this collected to answer our question, or is it a by-product of people using something?
Why ask it
A lot of big data is exhaust: clicks, swipes, sensor readings, and logs kept for billing or debugging. It records what people did inside one system and nothing about why, or about what they did elsewhere. What the system was built to do decided what got written down, so start there.
Is one row a person, a device, a session, or an event?
Why ask it
The same table gives very different counts depending on the answer. One person with a phone, a laptop, and a cleared browser can appear as several users, and an event log may hold thousands of rows for a single afternoon. Anything called a user count depends on how those rows were rolled up first.
Which sources were joined to build this, and how were the records matched?
Why ask it
Matching on an email address, a name, or a postcode is never complete. Records that fail to match usually drop out without notice, and they are often not a random set: recent movers, people with common names, customers of the older system. Ask for the match rate.
Has anyone read a few hundred of the raw records by eye?
Why ask it
Nobody can read a billion rows, which is how a field full of placeholder dates or a timestamp in the wrong time zone gets through to the summary. A person scrolling a random sample for twenty minutes finds problems that no summary statistic reports.
How big is the difference in real terms, not only whether it is significant?
Why ask it
A significance test asks whether a gap could be down to chance, and with millions of records even a trivially small gap passes. The useful figure is the size of the effect in units someone cares about: minutes, dollars, customers per thousand.
If a model produced this number, what was it trained on and how was it tested?
Why ask it
A score, a forecast, or a segment label is a model's output and inherits the habits of the data it learned from. Two details decide how far to trust it: the period and the people the training data covered, and whether accuracy was measured on cases the model had never seen.
Who does the model get wrong most often?
Why ask it
Overall accuracy averages over everyone, and errors tend to cluster where the training data was thin: new customers, small regions, uncommon cases. Ask for the error rate by group, and for what happens to a person on the receiving end of a wrong call.
How would we find out if a feed stopped or changed its format?
Why ask it
Automated pipelines fail quietly. A source that sends nothing looks like a day with no activity, and a renamed field can become a column of blanks that still loads. A good answer names the checks that run on each load and the person who is told when a count falls outside its usual range.
Would a random sample of ten thousand rows have told us the same thing?
Why ask it
For an average or a rate across a whole population, a well-drawn sample is often enough, and it is far quicker to check. The full dataset earns its cost when the question is about rare events or small groups. The answer shows which kind of question is on the table.
Are we allowed to hold this data, and would the people in it expect us to?
Why ask it
Legality and expectation are separate tests, and a use can pass the first and still surprise the people in the data. The rules differ by country and by sector, so ask whoever handles privacy how they apply here, what people were told when the data was collected, and whether records can be re-linked to individuals. A dataset described as anonymized is worth the same questions.
The decision
What are we going to do differently because of this?
Why ask it
If nobody can name an action, the chart is a status update and can be read as one. If there is an action, the scrutiny can be scaled to it: a reversible change to an email subject line needs far less proof than closing a branch.
What would have to be true for us to be wrong about this?
Why ask it
Ask for the conditions under which the conclusion fails rather than for a confidence level. Good answers are concrete: if the tracking change on the fourth was wrong, if this cohort is unusual, if the response rate is skewed by region.
What decision changes if this number is off by twenty percent?
Why ask it
This sizes the entire discussion. If the action is the same either way, stop analyzing and act. If a modest error flips the decision, the analysis is not yet finished, and that is worth saying before anyone commits budget to it.
If this reading is right, what should we see next month?
Why ask it
A conclusion that predicts nothing cannot be checked later. Write down the expected figure and the date now, while the explanation is fresh, so that a miss gets noticed and is not explained away with a new story.
What is the quickest extra check that could change our minds?
Why ask it
Often it is one more cut of the same data, a call to five customers in the segment, or a look at the raw records for the day of the spike. Naming the check and its cost turns a vague unease about the number into an afternoon of work, or into a decision to go ahead without it.
Who has looked at this and disagreed?
Why ask it
An analysis seen only by the person who made it and the person who wanted the result has not been tested. Ask who reviewed it and what they pushed back on. No objections at all more often means few readers than flawless work.
Could someone else reproduce this from the raw source?
Why ask it
The practical test is whether the query and every step that reshaped the data are saved where another person could run them. Analysis that lives only inside one person's spreadsheet cannot be checked and will not survive that person changing jobs.
When will we look at this number again, and who owns it until then?
Why ask it
Data supports a decision on the day it is read and starts ageing straight away. Agree a date to check again and a person who will do it, so that the figure in the plan does not outlive the conditions that produced it.
How to interrogate an analysis without stalling it
Practical guidance for the conversation itself
How to ask in the room
Start with the group that matches what you were handed
A single chart in a meeting calls for The graph and then How solid. A figure in a report with no picture starts at Source. A model score, or anything described as big data, goes to that group first and then back to Left out. Two or three questions asked well do more than a full audit nobody has time for.
Settle definitions before methods
Most disputes that look statistical are two people using one word differently. Agree what counts as a user, a sale, or a churn before anyone argues about the direction of the trend.
Ask for the numerator and denominator separately
Rates conceal movement. A rising conversion rate on falling traffic is a different situation from a rising rate on steady traffic, and only the two raw counts tell them apart.
Make questioning cheap
If asking about methodology reads as an accusation, people stop volunteering caveats. Ask the same questions of results you like, and say plainly that you are checking the work rather than the person who did it.
Write down the decision first
Agree what action each possible result implies before looking at the analysis. That is what stops a number from being reinterpreted until it supports the plan somebody already had.
Fast checks before you accept a chart
- Read the axis labels, the units, and the range before the shape of the line.
- Count the vertical axes, and check whether the baseline is zero.
- Find the denominator.
- Look for the point where tracking changed, a release shipped, or a system went down.
- Compare the mean against the median where both exist.
- Ask what the same chart looked like a quarter ago.
- Check whether the most recent period is complete.
Common pitfalls
Mistaking precision for accuracy
Three decimal places say nothing about whether the measurement was well designed. A confidently wrong figure travels much further than an honest range does.
Accepting a story built on one cut
If a finding appears in only one segment, one week, or one chart type, it is a hypothesis. Ask what happens under two other reasonable cuts before it becomes a fact in a deck.
Reading significance as importance
A statistically significant effect can be far too small to act on, and a non-significant one can still be the best guide available. Ask about effect size and cost alongside the test.
Letting the dashboard be the analysis
A dashboard shows what somebody once decided to measure. The question you actually have is usually not on it, and answering it means going back to the source data.
Treating a big dataset as a complete one
Millions of rows feel like the whole picture. They are still a record of whoever used one product, carried one device, or lived where the sensors were. Ask who is in the data before asking what it says.
A five-minute review of any analysis
- Ask what decision this is meant to inform.
- Get the definitions and the date range in plain words.
- Ask what was excluded, and who never appeared in the data at all.
- Ask how much of the effect comes from the largest handful of records.
- Ask what would have to be true for the conclusion to be wrong, and whether anyone else could reproduce it.