Bias Questions to Ask
Twenty questions for checking a decision, a dataset or your own judgment for bias, phrased to examine the reasoning rather than to accuse the person. Useful in a hiring panel, a model or metrics review, an editorial call, or on your own before you commit to something.
The questions
Open any question for the note
What had we already decided before we looked at the evidence?
Why ask it
Most bias enters before analysis begins, in the framing of the question and the preferred answer. Naming the starting position out loud makes it possible to test, and a group that claims it started neutral has usually not checked.
Who is in the room for this decision, and who is affected but not here?
Why ask it
Sounds procedural and often turns up the whole problem: the people who will live with the outcome have no representation in it. Listing the absent parties by name or role is more useful than a general commitment to inclusion.
What would have to be true for the opposite conclusion to be right?
Why ask it
Forces the group to construct the other case rather than dismiss it, which surfaces assumptions nobody had stated. If people cannot produce a coherent opposite case, the conclusion has not been examined yet.
Where did this data come from, and who is missing from it?
Why ask it
Datasets inherit the shape of how they were collected: who was available, who answered, who was already a customer. The absences are what produce confident conclusions that do not hold for anyone outside the sample.
What are we measuring, and what does that measure leave out?
Why ask it
A proxy always drops something, and what it drops tends to correlate with who is disadvantaged. Ask what would score well on this measure while being obviously wrong, since that gap is where the bias operates.
Which examples came to mind first, and why those?
Why ask it
The cases that arrive fastest are the memorable, recent or personally connected ones rather than the representative ones. Comparing the first examples against the full list often changes the sense of what is typical.
Would I read this the same way if a different person had said it?
Why ask it
A private question to run on your own reaction, not a challenge to put to someone else. It works because the answer is often a specific person whose identical point you would have accepted without hesitation.
What are we treating as normal here, and normal for whom?
Why ask it
Defaults are the quietest form of bias: a standard schedule, a standard career path, a standard name format. Once the reference case is named, it becomes possible to ask whether it should be the reference.
What is the strongest version of the view we disagree with?
Why ask it
If the group can only reproduce a weak version, they are arguing with a caricature and cannot know whether they are right. Ask someone who holds the view to check the summary rather than assembling it yourselves.
What evidence have we set aside, and on what grounds?
Why ask it
Selection during analysis is more common than selection during collection, and it usually feels like good judgment at the time. You are looking for a rule applied evenly, not a rule that happened to remove the inconvenient cases.
Who benefits from the way this is currently framed?
Why ask it
Framing decides the answer more often than analysis does, and whoever set the frame usually did so for reasons. This is a question about incentives rather than integrity, which is why it is answerable in a meeting.
Are we repeating what we did last time because it worked, or because it is familiar?
Why ask it
Precedent is treated as evidence long after anyone has checked the outcome. Ask what the result actually was, and whether anyone measured it, before it is used to justify the same choice again.
How would this decision look to the person it goes against?
Why ask it
You are testing whether the reasoning survives being explained to the person who loses. If the honest answer is that it would sound arbitrary or insulting, the reasoning needs work rather than better wording.
What am I not being told because I am the one asking?
Why ask it
Seniority filters information, so the version reaching you is already edited for your reaction. Ask who would know the unedited version, and whether they have any reason to volunteer it.
Which of our criteria actually predict the outcome, and which are just conventional?
Why ask it
Requirements accumulate without anyone checking whether they correlate with performance: a degree, an institution, years of experience, a tool. The conventional ones do most of the excluding and none of the predicting.
If we swapped the names on these two cases, would we treat them the same?
Why ask it
A concrete test you can run in the room with real examples rather than a discussion of principles. Disagreement about the answer is itself the finding, and it is usually more productive than any general debate about fairness.
What would we have to see to change our minds, and are we collecting it?
Why ask it
Stating the disconfirming evidence in advance is what separates a decision from a rationalization. If nobody can name it, or nothing in the plan would ever produce it, the conclusion is unfalsifiable by design.
Who reviewed this, and did they have a reason to agree with us?
Why ask it
Review by people who share the assumption checks arithmetic, not premises. Ask whether any reviewer was in a position to say no without cost, since that is what makes a review worth having.
What is the cost of being wrong, and who pays it?
Why ask it
When the downside falls on someone else, scrutiny tends to relax without anyone deciding to relax it. Naming who absorbs the error usually changes how much evidence the group wants before proceeding.
What have we not written down because it would look bad in writing?
Why ask it
The strongest question on this list and the one to save for a group that trusts each other. Anything people are willing to say but not record is usually the part of the decision that would not survive scrutiny.
Using These Questions Without Putting People on Trial
Practical guidance for the conversation itself
How to Raise Bias Productively
Ask about the process, not the person
Where did this data come from is answerable. Are you biased is not, and it turns the conversation into a defence of character. Almost every question worth asking can be pointed at the method instead of the individual.
Ask before the decision, not after
Once a choice has been announced, these questions read as an attack on it and get resisted accordingly. The same questions asked while options are still open are usually welcomed, because nobody has anything to protect yet.
Fix the criteria before you see the options
Writing down what matters, and how much, before candidates or vendors or proposals are in front of you removes the room to redefine the standard around a preference. Whoever runs the meeting should hold that list and read it back.
Give someone the job of arguing the other side
An assigned role makes disagreement safe and expected, and it works better when the person is senior enough to be heard. Rotate it, so it does not become one person's reputation.
Write down the decision and the reason
A short record of what was decided, on what evidence, and what would have changed the answer makes a later review possible. Without it, everyone remembers having been right for good reasons.
Checks That Work Better Than Discussion
- Compare two real cases side by side with the identifying details swapped.
- Check outcomes by group after the fact, not just intentions before it.
- Look at who was screened out at each stage, since the drop-off points are more informative than the final selection.
- Ask how the same question was answered a year ago and whether anyone checked the result.
- Have someone outside the team try to reproduce the conclusion from the same evidence.
- Count how many of the requirements were ever tested against performance.
Common Pitfalls
Treating awareness as a remedy
Knowing about a bias does not reliably stop it operating, which is why structure, checklists and fixed criteria do more than conversation. Use the discussion to change the process, not to conclude that everyone is now alert.
Asking someone to inventory their own biases in public
This produces performance rather than information, and the people most willing to answer are rarely the ones the question was aimed at. Ask about specific decisions they made and how they made them.
Using the language as a weapon
Naming bias to win an argument teaches everyone present that the word is a tactic. Once that happens, the genuine cases stop being raised, which is the more expensive outcome.
Auditing the model and ignoring the people
A biased dataset and a biased review panel produce similar outcomes, and only one of them tends to get formally examined. Point the same questions at the human process that you point at the system.
Stopping at the finding
A conversation that identifies a problem and ends there leaves everyone slightly more cynical. Close with one change to the process and a date to check whether it worked.