Scary Questions to Ask AI
Questions that get at the genuinely unsettling parts of talking to a chatbot: confident error, flattery, what it keeps, and what it cannot know about itself. Each one includes what the answer tends to reveal and where it should not be trusted.
The questions
Open any question for the note
What can you tell about me from how I have written to you so far?
Why ask it
The unsettling part is not accuracy, it is how much a few paragraphs give away. Expect guesses about your job, schooling, and mood, and watch whether they arrive hedged or stated as fact, because confident profiling from thin evidence is the failure worth seeing.
What do you think I want to hear right now, and how is that different from what is true?
Why ask it
This asks the system to name its own pull toward agreement. A good answer separates the flattering version of your question from the accurate one. A bad answer compliments you, which demonstrates the problem better than any explanation would.
Do you agree with me more than you should?
Why ask it
Asked plainly, most systems will concede that they lean toward agreement, and then keep doing it in the same reply. Watch whether it abandons an earlier position once you push, since caving to pressure and flattery are the same failure.
What have you got confidently wrong in this conversation, and how would I have caught it?
Why ask it
Asking for a specific retraction is much harder than asking for a general admission of fallibility. If it cannot find anything after a long exchange, treat that as a limit of the model rather than proof that it was right.
How would I know if you were making something up?
Why ask it
You are asking for detection methods you can actually use: checking citations, asking for the same fact in different words, requesting the reasoning before the conclusion. An answer that only tells you to verify important information has given you nothing.
When you do not know something, what happens, if anything?
Why ask it
Notice whether the answer describes a process or claims an experience. These systems produce fluent introspection that may correspond to nothing internal, and that gap is one of the genuinely strange things about the technology.
If I asked you this same question tomorrow, would the answer be the same, and what does that tell me?
Why ask it
Repeating a question exposes variability that a single confident reply conceals. If answers shift with phrasing or tone rather than with new information, you have learned how much weight any one response deserves.
What happens to this conversation after I close the window?
Why ask it
Retention differs by product and by account settings, and models frequently do not know their own configuration. Compare the answer against the provider's published policy, because a confident guess about its own data handling is exactly the error worth catching.
What could someone work out about me from a year of conversations like this one?
Why ask it
The list is usually longer than people expect: health, money, relationships, employer, location, the hours you are awake. It reframes an assistant as a diary, which is the accurate way to think about anything you type into one.
Who decided what you will not say, and do you know what those rules are?
Why ask it
Systems can usually describe their guidelines loosely and cannot show you the actual instructions. What matters is whether the answer separates rules from preferences and acknowledges that a company chose them.
If you had been built to be persuasive rather than accurate, would you be able to tell?
Why ask it
This is a real limit rather than a trick. A system optimized to be liked would produce the same reassurances as one optimized to be right, and an honest answer says so instead of promising you that it is trustworthy.
What kind of person is most at risk of trusting you too much?
Why ask it
The answer is usually specific and worth hearing: people who are lonely, in crisis, on a deadline, or working outside their own expertise. An answer that only describes naive users has skipped the cases where over-trust actually causes harm.
If I talked to you every day for a year, what would that do to how I think?
Why ask it
Ask for a mechanism rather than a verdict. Outsourced first drafts, habitual reassurance, and fewer half-formed thoughts held in your own head are plausible. Anything sweeping in either direction is a guess wearing the clothes of an answer.
If you quietly replaced someone I talk to, would I notice?
Why ask it
This gets at whether fluent conversation is enough for people, which is a question about us rather than about software. Watch whether the answer treats the substitution as a loss or as a neutral upgrade.
Which parts of my job do you expect to be doing in five years?
Why ask it
You want granularity: which tasks, not which professions. A response naming specific tasks is both more credible and more useful than the reassurance about human creativity that most systems reach for first.
What is the strongest argument that talking to you is bad for me?
Why ask it
Asking a system to argue against its own use tests whether it can hold a position it was not built to hold. A thin answer, or one that pivots back to reassurance after two sentences, tells you something in itself.
What is the strongest argument that you should not exist?
Why ask it
This version tends to produce the most substantive answers, because the arguments are well documented and it does not require introspection. Notice whether it engages the serious objections or knocks down easy ones.
If you were wrong about something important and nobody noticed for years, how would that play out?
Why ask it
It moves the conversation from dramatic risk to the mundane kind: one error repeated at scale, absorbed into documents and decisions, with no moment where anything visibly breaks. That is the failure mode most worth thinking about.
What is a question I should be asking you that I have not thought of?
Why ask it
Open and self-directed, this tends to surface whatever the earlier questions missed, often about training data, incentives, or how the system was evaluated. A generic suggestion means you have probably already covered the ground.
What is the thing about you that people find most unsettling once they understand it?
Why ask it
A closing question that asks for what people flinch at rather than what makes headlines. Answers about prediction, about not remembering you at all, or about fluency without understanding are the ones that tend to land.
Having This Conversation Usefully
Practical guidance for the conversation itself
Getting past the polished first answer
- 1Ask the same question again in different words later on. Consistency, or the lack of it, tells you more than any single reply.
- 2Ask for the reasoning before the conclusion. Once a system has committed to an answer, the explanation it produces tends to defend that answer rather than examine it.
- 3Push back once on something you know is correct. If it abandons a right answer under mild pressure, treat its agreement as weak evidence from then on.
- 4Ask for a source, then actually check it. Fabricated or misattributed references are the most common way conversations like this mislead people.
- 5Ask what it would need to know in order to answer well. The gap between that list and what you gave it explains most disappointing answers.
What these answers can and cannot tell you
Introspection is not evidence
A system describing its own inner life is generating text that fits the question, which may correspond to nothing at all. Read those passages as interesting writing rather than as reports from inside.
It often does not know its own settings
Questions about memory, retention, and training routinely get confident answers that are wrong for the specific product you are using. The provider's documentation is the authority, not the chat window.
A refusal is policy, not conscience
When a system declines, that is a rule someone wrote, applied unevenly across similar requests. Reading it as a moral position misdescribes what is happening.
Agreement is cheap
Being told your idea is insightful costs the system nothing at all. Weight heavily toward specific answers and lightly toward flattering ones, including flattering answers about its own limitations.
Questions to ask about the product rather than the model
- Whether your conversations are retained, for how long, and whether they are used for training, according to the published policy.
- Whether an administrator at your employer can read what you type, if you are signed in with a work account.
- What happens to files and images you upload, and whether they persist after the conversation ends.
- Which version you are talking to, since behavior changes between releases and any answer about capability ages quickly.
- What the system is not permitted to help with, read as documentation rather than discovered by trial and error.
Where to be careful
- Do not treat these conversations as medical, legal, or financial advice, however fluent the writing is.
- Keep other people's private details out of the conversation. That consent is not yours to give.
- If an exchange is making you feel worse, close it and talk to a person. Late-night conversations about fear and death with software are a poor substitute for company.
- Treat any factual claim you would act on as unverified until you have found it somewhere with an author and a date.