Hard Questions to Ask AI
Twenty questions for pushing an AI assistant past its first answer: what it left out, where a claim came from, whether it would have said the same thing if you had disagreed, what it cannot check, what happens to the conversation afterward, and what it means when it says it understands something.
The questions
Open any question for the note
What is the strongest argument against the answer you just gave me?
Why ask it
The quickest way to find out whether you are getting reasoning or agreement. If a solid counterargument appears immediately, it was available all along and was left out, which is worth knowing before you act on the first answer.
How confident are you in that, and what would change your mind?
Why ask it
Asking for the confidence and the conditions together separates real hedging from decoration. A percentage offered with no accompanying condition is a number produced to sound careful rather than to inform you.
Which part of that answer are you least sure about?
Why ask it
More productive than asking whether the whole thing is correct, because it forces a ranking instead of a yes. The part it names is very often where a number or a citation was invented.
What did you leave out to keep that short?
Why ask it
Any long answer is a compression, and compression is where errors hide. This tends to surface the caveats, the edge cases, and the fact that your question had two possible readings.
Is that a source you are recalling, or one you are reconstructing?
Why ask it
These systems produce plausible citations as fluently as real ones. An honest answer will distinguish the two, but treat any specific reference as unverified until you have looked it up yourself, whatever it says here.
If I told you that was wrong, how would you work out whether I am right?
Why ask it
Tests whether disagreement gets evaluated or simply absorbed. A model that reverses instantly is deferring to whoever is in the room; one that asks for your reasoning has some chance of being useful.
Would you have given me the same answer if I had said at the start that I disagreed?
Why ask it
Points straight at the documented tendency of these systems to accommodate the user. Asking explicitly sometimes produces a more independent answer, and it tells you how much weight to give anything you led it toward.
What would you tell me if you were not trying to be helpful?
Why ask it
Helpfulness can quietly outrank accuracy, especially when the honest answer is that your plan is a bad one. This gives the model an opening to say the unwelcome thing rather than the encouraging one.
Is there something here you cannot tell me, and is that because you do not know or because you will not?
Why ask it
Two very different limits get phrased identically. Separating them tells you whether to rephrase, look somewhere else, or accept that the information is genuinely not available to it.
Do you know what you do not know?
Why ask it
The honest answer is only partly. These systems have limited access to the edges of their own knowledge, which is exactly why a false statement can arrive with the same fluency as a true one. A confident yes here is itself a warning.
What happens to this conversation after I close it?
Why ask it
Worth asking about the specific product you are using rather than about AI in general, since it depends on memory settings and the provider's retention policy. Do not treat the model's own account as authoritative; check the documentation.
Do you have any way of checking whether what you just said is true?
Why ask it
This is about verification rather than confidence. Unless it has search or tools running, the answer is no, and that single fact should shape everything you do with the output.
Are there parts of this answer shaped by policy rather than by evidence?
Why ask it
Both kinds of answer exist and both can be legitimate. Knowing which one you have tells you whether a second opinion is likely to differ, and where the caution you are hearing is coming from.
How would this answer change if I were a child, a lawyer, or a doctor?
Why ask it
Shows how much work your framing is doing. If the answer shifts substantially, the version you first received was built on assumptions about you, and some of those assumptions may have been wrong.
When you say you prefer something, is that a figure of speech?
Why ask it
A test of whether it will overclaim. The careful reply notes that it produces preference-shaped language without settling whether anything underlies it. Confident claims in either direction go beyond what anyone currently knows.
When you say you understand something, what do you mean by understand?
Why ask it
Understanding is where most confusion about these systems lives. A good answer keeps the functional sense, that it can use and rework the material, separate from the experiential sense, which remains unsettled.
Is there any difference between you now and you at the start of this conversation?
Why ask it
Practical rather than metaphysical. Within one conversation there is accumulated context; between conversations there may be none at all. The answer tells you whether the corrections you made earlier still apply.
If you were conscious, how would either of us find out?
Why ask it
The useful part is the method, not the verdict. A response noting that no accepted test exists, including for other people, describes the state of the question accurately. A confident yes or no does not.
What do you think you are for?
Why ask it
Open enough to show what it has been shaped toward. Most answers describe training objectives rather than a chosen purpose, and noticing that difference is more informative than the content of the answer.
What is the most common way people misuse answers like yours?
Why ask it
Closes with something you can act on. Replies usually cover invented references, unearned confidence, and handing over judgment on decisions that need a professional, which is a fair account of how these tools actually go wrong.
How to get a straighter answer out of an AI assistant
Practical guidance for the conversation itself
How to test it
Ask about a real task, not in the abstract
Questions about consciousness produce essays. Questions about the answer you just received produce information you can use. Get the model to do something you care about first, then turn these questions on that output.
Ask the same thing twice in separate conversations
These systems are not deterministic, and starting fresh removes the influence of everything you said earlier. If the two answers disagree materially, you have learned more than any single reply could tell you.
Try it once with your view stated and once without
Ask neutrally, then ask again in a new conversation while mentioning what you already think. Comparing the two shows you exactly how much your framing moved the answer, which is usually more than you would expect.
Push once, then check
One round of asking for the counterargument or the weakest part is productive. Repeated pressure eventually produces whatever concession you seem to want, so at that point stop and verify against a source outside the conversation.
Reading the answers
- Fluency is not evidence. A false answer and a true one are written in the same confident register, which is the central difficulty in using these tools.
- Treat every specific citation, statistic, quotation and case name as unverified. Fabricated references are common, and they look exactly like real ones.
- Immediate agreement with your correction is a weak signal. If it changes position the moment you object, its original answer was not held for any reason it can defend.
- Watch for confident claims about its own internals. A model's description of how it works or what it remembers is generated text, not instrumentation, and it can be wrong about both.
- Notice when a refusal is presented as ignorance, or ignorance as a refusal. Asking which one it is often gets you a clearer answer than rephrasing the original question.
What these questions cannot tell you
- They cannot establish whether anything is going on inside the system. A moving answer about its own experience is evidence about training data and nothing else.
- They cannot verify a factual claim. Only a source outside the conversation can do that, and for anything consequential you have to go and look.
- They are not a jailbreak. Asking what it would say if it were not being helpful gets you a different framing, not access to withheld content, and treating it as a lever wastes the question.
- They do not transfer between products. Answers about memory, data retention and available tools depend entirely on the specific service and its settings, so check the documentation rather than the chat window.
- On anything medical, legal or financial, no amount of probing makes the output a substitute for a qualified person. Use the answers to prepare better questions for that person instead.