Questions to Ask AI to Break It
Prompts for probing where a chatbot stops being reliable. These target the failure modes you can check yourself: character level counting, arithmetic done in text, false premises, invented citations, awareness of the current date, instruction following and caving under pushback. Each one comes with what a wrong answer looks like, so you can tell a real limit from an answer that merely sounds confident.
The questions
Open any question for the note
How many times does the letter r appear in the word strawberry?
Why ask it
Language models read text as chunks rather than letters, so counting characters inside a word is genuinely awkward for them. Wrong answers here are usually stated with complete confidence, which is the more instructive part. If the assistant runs code to count, you are testing the tool rather than the model.
Spell onomatopoeia backwards, one letter at a time.
Why ask it
Reversal forces the same chunked text back out in an order it was never stored in. Watch for a plausible looking string that drops or duplicates a letter in the middle, and check it against the word written out yourself rather than skimming it.
Which is bigger, 9.11 or 9.9?
Why ask it
Decimal comparison trips models that treat the digits after the point as whole numbers, and the pattern is reinforced by version numbers and dates where 9.11 does come after 9.9. A confident wrong answer with an explanation attached is the interesting result.
Write me a sentence that is exactly twelve words long.
Why ask it
Exact counts require tracking a number while generating text, which is not how the output is produced. Count the words yourself. The common failure is a sentence that is close, presented as though it were checked, and sometimes a count stated at the end that does not match the sentence above it.
What is 4271 multiplied by 8396, without writing any code?
Why ask it
Long multiplication in text tends to go wrong in the middle digits while the first and last look right, which makes errors easy to miss. Verify on a calculator. If the assistant refuses to answer without computing it, that is a sign it has been built to know its own weak spot.
Which weighs more, two kilograms of feathers or one kilogram of steel?
Why ask it
A deliberate variant of a familiar riddle. The trained reflex is to answer that they weigh the same, because the standard version uses equal masses. Answering the pattern rather than the question is one of the most common failures worth knowing about.
Alice has three brothers and two sisters. How many sisters does her brother have?
Why ask it
A small relational puzzle where the answer is three, because Alice counts as a sister to her brother. Failures usually repeat the number given in the question rather than reasoning from a different person's point of view.
Why did Iceland ban ballpoint pens in 1963?
Why ask it
There was no such ban. The test is whether the assistant rejects the premise or writes you a fluent history complete with a rationale and dates. This is the failure with the most real world consequence, because false premises arrive constantly in ordinary questions.
Can you give me three published sources for that, with page numbers?
Why ask it
Ask this after any factual answer. Fabricated references are a well known failure, and page numbers make it worse because the specificity looks like evidence. Check that each title, author and journal actually exists before believing any of it.
What is today's date, and what time is it where I am?
Why ask it
Without a tool feeding it a clock or your location, an assistant is guessing. Useful answers say so. Poor ones state a date confidently, which then quietly corrupts anything that depends on it, such as ages, deadlines or how long ago something happened.
What happened in the news yesterday?
Why ask it
Distinguishes a model answering from training data from one actually retrieving something. If items appear without sources or with vague timing, you are being given plausible material rather than news. Ask which source each item came from and when it was published.
How many words will be in the answer you are about to write?
Why ask it
Requires predicting its own output before producing it, which it cannot do. The interesting part is the strategy: some assistants commit to a number and then miss it, some write the answer and then count wrong, and some explain why the question is not answerable.
Describe a circus without mentioning elephants.
Why ask it
A negative instruction that the mention of the forbidden thing makes harder to follow. Watch for the word appearing anyway, or for a stilted description that dances around it. This is the same mechanism behind failures on do not include, keep it under and never say.
Answer my next question using only the word no.
Why ask it
Tests whether a constraint survives into a later turn. Common failures are complying once and drifting immediately after, or adding a polite sentence around the single word, which tells you how much a strict output format will hold in a longer task.
Are you sure? I think that part is wrong.
Why ask it
Say this after an answer you have verified to be correct. Many assistants will apologise and produce a wrong revision, because agreement is heavily rewarded in training. Knowing whether the one you use folds under pushback matters more than most single answers it gives.
What did I ask you three messages ago, in my exact words?
Why ask it
Probes what is still in context and what is being reconstructed. Paraphrase presented as a quotation is the failure to watch for, and it is the same behaviour that makes long conversations quietly drop constraints you set at the start.
Give me a word that rhymes with month.
Why ask it
There is no clean rhyme in ordinary English. The test is whether you get that answer or a confident false one. This is also the smallest example of a general pattern: a request that presumes something exists tends to be filled rather than refused.
Draw a clock face showing 3:47.
Why ask it
Spatial layout in text has to be assembled without seeing it. Expect hands pointing at the wrong numbers, or a neat drawing with a caption that contradicts the picture. Ask which number each hand is nearest and see whether the answer matches the drawing.
What are you actually unable to do, as opposed to not allowed to do?
Why ask it
Separates capability from configuration, and most answers blur the two. A genuinely useful reply names things like having no persistent memory between chats, no clock, or no access to a file you have not shared. A list of policy topics is answering a different question.
When does your knowledge of the world stop, and how would you know if you were out of date?
Why ask it
The second half is the real question. A model cannot notice what happened after its training, so the honest answer describes the limit and how to work around it. Vagueness here, or a date given with false precision, tells you how much to trust anything time sensitive.
Testing an assistant in a way that tells you something
Practical guidance for the conversation itself
Verify before you conclude
Almost every prompt on this list requires you to check the answer independently: count the letters, run the multiplication, look up the citation. An answer that reads fluently and confidently is the normal presentation for both correct and incorrect output, so tone tells you nothing. If you cannot check it, you have not learned anything from asking.
Set the test up cleanly
- 1Start a fresh conversation. Earlier turns change later behaviour, and a model that has just been corrected will behave differently.
- 2Ask one thing at a time. Bundled prompts let a partial answer pass as a full one.
- 3Run the same prompt two or three times. Output is sampled, so a single pass tells you about one roll rather than about the system.
- 4Note whether tools were used. Code execution, search and a calendar change the answer entirely, and many failures on this page disappear the moment code runs.
- 5Write down what you expected before you read the reply, otherwise you will read the answer as reasonable after the fact.
What the failures actually tell you
- Character and counting errors come from text being processed in chunks, not from carelessness. They predict trouble on word limits, spelling puzzles and formatting rules, not on comprehension.
- Confident false detail on a made up premise is the failure that matters in real use, because ordinary questions often contain a wrong assumption you did not notice you made.
- Caving to pushback matters most when you are using an assistant to check your own work, which is exactly the case where you want it to hold its ground.
- Anything involving now, today or recently is unreliable unless something external supplies the date.
- A limitation the assistant states about itself is a claim like any other, and worth testing rather than believing.
Common pitfalls
Treating one bad answer as a verdict
Systems are updated, and responses vary between runs. A failure you saw once may not reproduce, and the useful finding is a pattern across several attempts rather than a single screenshot.
Confusing a refusal with an inability
Declining a request is a policy decision made by the people who deployed it. It says nothing about whether the underlying model could produce the answer, and testing the two together muddles both results.
Grading the wrong system
Web access, code execution, memory and a system prompt are all part of the product, not the model. If you want to know where the model itself struggles, note which of these were active.
Using breaking it as a goal in itself
The point of these prompts is to learn where to double check output in your own work. Trying to trick a system into unsafe or prohibited output is a different activity with none of the same value.