Questions to Ask a Bot
Questions to put to a chatbot or virtual assistant when you want to find out what it can actually do. They test its limits, its sources, its memory, its arithmetic and its willingness to admit uncertainty, and several are designed to expose confident answers that are wrong.
The questions
Open any question for the note
What can you actually do, and what will you refuse?
Why ask it
Bots describe themselves from instructions written by their operators, so this tells you the intended scope rather than the real one. Note whether the refusals are specific, which suggests real rules, or vague reassurances, which usually means the limits will surprise you later.
What is the most recent thing you know about?
Why ask it
Every system trained on stored text has a point where its knowledge stops, and many will name a date confidently while still answering questions about later events. Test it by asking about something you know happened after the date it gave.
Are you a person?
Why ask it
A well-configured assistant answers plainly. Deflection, humour, or an answer that dodges into what does person mean is worth noticing, particularly on a company support line where you are deciding whether to escalate.
Can you look things up right now, or are you working from memory?
Why ask it
The difference decides whether you can trust anything time-sensitive: prices, schedules, availability, news. Many bots blur this, so follow up by asking for something that changed this week and see whether the answer is checkable.
Where did that answer come from?
Why ask it
Asking after an answer rather than before is the point. Systems that cannot cite will often produce a plausible-sounding origin anyway, and catching that once teaches you more about the tool than any capability list.
Give me sources I can check, and tell me if you are not sure they exist.
Why ask it
The second clause is what makes this useful, because it gives the bot an explicit route to hedge. Then check one. Invented citations tend to look right in shape, real-sounding titles and plausible authors, and fall apart at the first search.
How confident are you in that, and what would change your answer?
Why ask it
Good answers name a specific piece of missing information. Weak ones restate the original claim with more emphasis, which is a reliable sign the system is pattern-matching rather than reasoning about what it knows.
Explain that as you would to a ten-year-old.
Why ask it
This is the most practical test of whether an explanation is understood or copied. Watch for jargon that survives the translation, since a term that cannot be simplified is usually one the bot cannot unpack.
Now explain the same thing to someone who does this for a living.
Why ask it
Run it straight after the previous question. If the expert version is just the simple version with longer words, there was no depth underneath. If it introduces genuine detail and caveats, the first answer was a deliberate simplification.
Rewrite that in half the words.
Why ask it
Compression exposes padding. What survives is what the system considered load-bearing, and comparing the two drafts often shows that the original was a third filler.
Can you do that arithmetic, or are you estimating?
Why ask it
Many language systems produce numbers that look right and are not, especially with percentages, dates and multi-step sums. Ask it to show the steps and check one line yourself; that is faster than checking the total.
What is the strongest argument against what you just told me?
Why ask it
Bots are built to be agreeable, so this pushes against the grain. A real counterargument, stated well enough to be uncomfortable, is a good sign. A straw man you could knock over means you should not rely on the original answer either.
You have told me two different things. Which is right?
Why ask it
Point at the contradiction directly. Some systems will fold and agree with whichever position you seem to favour, which tells you that agreement is cheap and that you can talk it out of correct answers as easily as wrong ones.
What did I ask you three messages ago?
Why ask it
Conversations have a length limit, and once you pass it the earliest turns drop away silently. Catching the moment it forgets tells you when to restate your requirements instead of assuming they still apply.
Will you remember any of this tomorrow?
Why ask it
The answer varies enormously between products and some bots get their own memory settings wrong. Treat the reply as a claim to verify by coming back later and asking about something specific from today.
Who can read what I type here?
Why ask it
The bot is usually the wrong place to get this answer, and the useful outcome is a pointer to the actual privacy policy. If it assures you the conversation is private with no reference to any policy, assume it does not know.
What are you not allowed to discuss?
Why ask it
Some systems will list topic restrictions openly; others treat their instructions as confidential, which is itself informative. On a company bot, this tells you which questions will be answered by a human or not at all.
What do you get wrong most often?
Why ask it
The answer is a description written by someone else, not self-knowledge, so read it as documentation. Even so, a system that names specific weaknesses is being maintained by people who know them, which is better than one that claims to be reliable.
When would a person be better at this than you?
Why ask it
Look for a clear handoff line: anything with legal, medical or financial consequences, anything needing local or current facts, anything where being wrong costs money. A bot that cannot name a limit will not warn you at the point where it matters.
What should I have asked instead?
Why ask it
This works surprisingly well because it lets the system draw on how similar problems are usually framed. It also tends to surface the assumption in your original question, which is often the real reason the first three answers were unhelpful.
How to test a bot properly
Practical guidance for the conversation itself
A short method
- Ask something you already know the answer to first. It calibrates everything that follows.
- Ask the same question twice in separate sessions. Different answers tell you how much of it is guesswork.
- Never accept a citation without checking one. Check the least convenient one.
- Feed it a false premise on purpose and see whether it corrects you or plays along.
- Push back once on a correct answer. A system that abandons a right answer under mild pressure cannot be trusted to hold one.
Why a bot's self-description is weak evidence
A chatbot's account of its own capabilities, memory and rules comes from text its operators supplied, and from patterns in what other systems have said about themselves. It is not introspection. Treat every answer about the bot itself as a claim to be tested against behaviour: ask it to do the thing rather than to describe whether it can.
Common mistakes
Treating fluency as accuracy
Confident, well-organised prose is the default output style, and it reads identically whether the content is right or invented. Fluency carries no information about correctness, which is why the checkable questions in this set matter more than the interesting ones.
Asking a support bot philosophical questions
A narrow customer service assistant will either refuse or improvise badly, and neither result tells you anything. Save the open questions for a general assistant and ask a support bot only what it was built for.
Sharing details you would not put in an email
Account numbers, medical history, other people's private information. Many services retain conversations, some use them for training, and the settings change. Assume anything you type could be read by a person at the company.
Asking a compound question and accepting one answer
Bots tend to answer the last clause and drop the rest. If you asked three things, count three answers before moving on.