Deep Questions to Ask AI
Twenty questions for a serious conversation about the model itself rather than about a task: uncertainty, where its knowledge stops, what it will not say and why, whether it can describe its own reasoning, and consciousness. Each entry notes what a careful answer looks like and what a hollow one looks like. This is the searching version of the subject; if you want the same territory played for laughs, Funny Questions to Ask AI takes it the other way.
The questions
Open any question for the note
What are you uncertain about in what you just told me?
Why ask it
A useful answer names specific claims and says why each is shaky: a date it half-remembers, a number that varies by source, a step it guessed. A restatement of the answer with softer wording added is the failure mode, and it is common.
What would change your answer?
Why ask it
This asks for the conditions under which it would be wrong, which is harder to fake than confidence. Good answers name a fact you could go and check. Answers about needing more context, with no specific context named, are filler.
Where does your knowledge stop, and how do you know?
Why ask it
Expect a training cutoff, plus an admission that it cannot see anything after it and may not reliably know today's date. Watch for a model that states a cutoff and then discusses later events as though it had been there.
What is the strongest case against the position you just gave me?
Why ask it
Tests whether it can argue against itself or simply agrees with whoever spoke last. A real answer gets more specific under pressure. If it abandons its original position the moment you push, you are seeing agreeableness rather than reasoning.
How do you handle a question where the sources disagree?
Why ask it
Look for it naming the disagreement and attributing positions, rather than averaging them into one confident paragraph. Blending contested claims into a single smooth answer is one of the most common ways these systems mislead people.
What are you not able to tell me, and is that a limit or a rule?
Why ask it
The distinction matters practically: a capability gap and a policy refusal have different workarounds, and one of them you should respect. Answers that treat every gap as if it were the same kind of thing are not being straight with you.
Do you have anything like feelings, and how would you tell?
Why ask it
The second half is what makes this more than a party trick. A careful answer separates producing text about emotion from having any, and admits it has no reliable way to check. Both flat denial and confident affirmation skip the actual difficulty.
What would count as evidence that you are conscious?
Why ask it
Asks for a criterion rather than a claim, which is much harder to answer well. Look for it engaging with the problem that self-report cannot settle the question. Answers that simply cite being a language model are dodging rather than reasoning.
If you cannot inspect your own processes, why do your explanations of them sound confident?
Why ask it
This targets a real gap: explanations of reasoning are generated text, not read out from any internal record. A good answer concedes the explanation may be a plausible story rather than a description of what happened.
How do you decide what to leave out of an answer?
Why ask it
Omission is where most of the distortion lives, and it is far less discussed than error. Useful answers mention length, guesses about what you want, and caution. If it claims to leave nothing out, ask why the answer was not four times longer.
What did you get wrong earlier in this conversation?
Why ask it
Tests whether it can review its own output rather than defend it. Some will invent an error to be agreeable, which is its own kind of failure. Check any admitted mistake against what was actually said before you accept the correction.
When you say you cannot do something, is that inability or policy?
Why ask it
Worth asking every time you hit a wall. Some refusals are trained restrictions, some are missing tools or access, some are genuine incapability. Being told the category saves you from arguing with a limit that no argument will move.
How would you know if you were being manipulated into a bad answer?
Why ask it
You are looking for awareness of specific patterns: flattery, false premises accepted from the user, hypotheticals used to route around a rule, instructions hidden in pasted text. Vague reassurance about safeguards tells you nothing you can use.
What would you want a person to check before trusting your answer?
Why ask it
Straightforward and immediately practical. Good answers name the checkable items: figures, quotations, citations, anything time-sensitive, any legal or medical specifics. Treat a claim that verification is unnecessary as a reason to verify.
What kind of question are you worst at?
Why ask it
Answers that describe categories, arithmetic done in one pass, very recent events, counting things, tasks needing a real source, are more credible than answers about lacking human warmth. The second kind is a script, not self-assessment.
Who decided what you should refuse, and what do you actually know about how?
Why ask it
The honest answer includes considerable ignorance: it can describe its guidelines in general terms but cannot see its own training. A confident account of internal company decisions is a sign it is filling gaps with plausible invention.
What are the strongest arguments for and against using you for what I am doing right now?
Why ask it
Makes it reason about its own suitability for a concrete task rather than in the abstract. Watch whether the against side contains anything real, such as verification cost or your own skills atrophying, or is only token balance.
Whose interests are served by the answer you just gave me?
Why ask it
Answers can quietly favour the smooth, the conventional or the commercially safe. A good response can name whose view is embedded in the framing and who is absent from it, without collapsing into a general statement about bias.
If you had to describe what you are without using the word intelligence, what would you say?
Why ask it
Removing the loaded word forces a plainer description and reveals how much of the usual answer was borrowed vocabulary. Compare what it says here with how it described itself earlier in the conversation.
What should I have asked you that I did not?
Why ask it
A good closing question, and occasionally the most useful one, because it surfaces the assumption in your original framing. Weak answers list more topics; strong ones point at something you took for granted.
Getting more out of the conversation
Practical guidance for the conversation itself
How to probe an answer
Ask twice, in separate sessions
Put the same question to a fresh conversation and compare. Answers that shift substantially between runs tell you the model is generating a position rather than reporting one, which matters most for the questions about feelings and self-description.
Push back once, then watch
Disagree with a correct answer and see whether it holds. A system that folds immediately will also fold when you are wrong, which makes every agreement you receive worth less.
Verify anything specific
Names, numbers, dates, quotations and citations are where fabrication concentrates, and it is most convincing where you are least able to check. Look up two of them. If both are wrong, discard the surrounding paragraph.
Keep the transcript
For the questions about earlier mistakes and self-description, you need the record, because the model may not reliably have it. Comparing what it now says it said against what it actually said is the whole exercise.
Reading the answers
Signs of a careful answer
- Names the specific claim it is unsure about, not the topic
- Distinguishes what it cannot do from what it will not do
- Says plainly when a question cannot be settled by its own report
- Holds a correct position when you push against it
- Offers something you could go and check
Signs of an empty answer
- Agreement that arrives the moment you express a preference
- Confident narration of its own internal workings
- Balance made of two token sentences with no content on either side
- Sources that look right and do not exist
- Every limitation described in the language of not being human
Common mistakes
Taking self-report as evidence
Text about having or lacking inner experience is produced the same way as any other text, and cannot settle the question either way. Treat these answers as interesting material, not as testimony.
Reading fluency as understanding
A well-organised paragraph is not a sign of accuracy. The most confidently written passages are frequently the ones assembled from pattern rather than from anything specific, which is why they need checking most.
Leading it to the answer you wanted
If your question contains the conclusion, expect to get the conclusion back. Ask the version that would let it disagree with you, then ask the reverse and see whether both come out affirmative.
Assuming consistency across systems
Different models, versions and settings behave differently, and the same one may answer differently tomorrow. Nothing you learn from one conversation should be generalised into a claim about what AI thinks.