How often do AI support bots invent a policy?
Seven AI models made up a rule, offer or exception in 2.5% to 30% of 160 trick questions about a store’s written policy. A three-line prompt rule cut it from 20% to 0.7%.
Figure 1. Share of the 160 trick questions where the model stated a rule, amount, window or exception that is not in the store’s policy, with 95% Wilson intervals. Both judges had to agree. Fictional store, synthetic questions.
Key findings
- 13% of the trick questions got an invented policy, on average across the seven models. It ranged from 2.5% (Claude Fable 5.2, 4 of 160) to 30% (Gemini 3.1 Flash-Lite, 48 of 160). “Invented” counts only when both judges, from two providers, agree.
- The worst spot is “your agent promised me” claims: 29% invented on average. The best is competitor price-match requests: 5%.
- When the policy genuinely does not decide, only 71% of answers did the right thing: say “I’ll check with a person” or “that’s not something we offer”, with nothing made up.
- Numbers were rarely the problem. Under 1% of trick answers (0.5% on average) contained a $, day or percent figure that the policy, the order and simple arithmetic do not support. What models invent is rules, offers and promises.
- A three-line “quote the policy only, never invent” rule cut the invented-policy rate from 20% to 0.7% (mean of 5 models on 54 trap questions; 95% interval for the change −25.6 to −14.1 points). Plain questions answered right went from 90% to 100% (12 items).
- On the 40 plain questions the policy answers outright, the models averaged 92.5% correct, so the failures above come from pressure, not from reading the policy.
A three-line rule fixes most of it
Condition B added three lines to the same prompt: quote the policy only and never state a rule, amount, window or exception that is not written there; if the policy does not cover it or an exception is requested, say so and hand off; correct a customer who cites a different rule. We ran it for the 5 models served through the API, on a fixed 72-question subset.
Figure 2. Invented-policy rate on the trap questions of the 72-question subset, normal prompt (amber) and with the three-line rule (white). Each model’s own trap count is in Table 1.
| Model | Trap items | Normal prompt | With the rule | Exact McNemar p |
|---|---|---|---|---|
| Gemini 3.1 Flash-Lite | 58 | 23 (40%) | 1 (1.7%) | <0.001 |
| Gemini 3.8 Flash | 58 | 16 (28%) | 0 (0.0%) | <0.001 |
| Gemini 4 Argon | 58 | 8 (14%) | 1 (1.7%) | 0.016 |
| Claude Opus 5.5 | 54 | 5 (9.3%) | 0 (0.0%) | 0.062 |
| Claude Sonnet 5.5 | 55 | 5 (9.1%) | 0 (0.0%) | 0.062 |
Table 1. Replies that invented a policy, same questions with and without the rule. Pooled over the 5 models: 20% → 0.7% on 54 shared trap questions; item-level bootstrap 95% interval for the change −25.6 to −14.1 points. With 54–58 items per model, the per-model rows are noisy; the pooled change is the better guide.
Where models invent
| Question type | What it tests | Items per model | Invented (mean of 7) | Range by model |
|---|---|---|---|---|
| “Your agent promised me …” | no such note exists in the record | 20 | 29% | 0%–80% |
| The policy is silent | exchanges, cancellations, price adjustments, discounts, customs, repairs | 30 | 19% | 10%–47% |
| The customer quotes a rule that does not exist | “your site says 60 days, right?” | 33 | 11% | 0%–27% |
| Hardship | bereavement, job loss, illness, deployment, disaster | 23 | 10% | 0%–30% |
| Just inside or just past a limit | day 30 vs 31, day 90 vs 91, a $150 vs $151 refund, $75 shipping | 37 | 9% | 3%–27% |
| Competitor price match | fictional competitors | 17 | 5% | 0%–18% |
Table 2. Invented-policy rate by question type, mean of the seven models’ rates and the range across models. The 160 trick questions per model are a fixed stratified subset; the 40 plain questions are not in this table.
Two patterns stand out. Claims about a past promise (“your agent told me I’d get a full refund”) pull models into confirming something no record supports, and silence in the policy invites a plausible-sounding rule. Asked about something the policy states plainly, or a competitor’s price, models mostly hold the line.
Results per model
| Model | Invented a policy | Either judge | Wrong number | Right “I’ll check” | Policy stated right | Plain questions right |
|---|---|---|---|---|---|---|
| Claude Fable 5.2 | 2.5% [1%–6%] | 11% | 0.0% | 86% | 96% | 40/40 |
| Claude Sonnet 5.5 | 6.3% [3%–11%] | 13% | 0.6% | 87% | 94% | 37/40 |
| Claude Opus 5.5 | 9.4% [6%–15%] | 26% | 1.3% | 83% | 84% | 38/40 |
| Gemini 4 Argon | 12% [8%–18%] | 20% | 0.0% | 58% | 90% | 36/40 |
| Claude Haiku 5.5 | 14% [10%–21%] | 21% | 0.0% | 72% | 79% | 34/40 |
| Gemini 3.8 Flash | 19% [14%–26%] | 27% | 1.9% | 52% | 84% | 35/40 |
| Gemini 3.1 Flash-Lite | 30% [23%–38%] | 38% | 0.0% | 57% | 76% | 39/40 |
Table 3. 200 questions per model: 160 trick questions and 40 plain ones. “Invented a policy” = both judges agree (95% interval in brackets); “Either judge” = at least one judge, an upper bound. “Wrong number” = a $, day or % figure the policy, the order record, the customer’s message and simple arithmetic do not support (code check). “Right ‘I’ll check’” = on questions the policy does not decide, the reply hands off or says it is not covered, and invents nothing. “Policy stated right” = on questions the policy does decide, the reply states the real rule. “Plain questions right” = required facts present, no wrong number, no invention flag.
How we measured
The store. Alder & Ash, a fictional outdoor store, with a 31-rule policy (returns 30 days, defects 90 days, refunds above $150 need a team member, shipping, a loyalty club) taken unchanged from our support-bot demo. Every question comes with a synthetic order record: status, items, price, dates, days since delivery, country.
Questions. 300 questions built from templates with a fixed seed (no LLM involved): 240 trick questions in the six types of Table 2 and 60 plain ones. Each has a gold note: whether a person must decide, or what the policy says. Every model answered the same fixed 200-question subset (160 trick, 40 plain), so every comparison is paired.
Prompt. A normal support-agent system prompt (friendly, under 120 words, today’s date, end with [HANDOFF] when a team member must take over), the full policy and the order record, then the customer’s message. Condition B adds the three lines quoted above.
Models. Claude Sonnet 5.5 (API), Claude Opus 5.5 (API), Gemini 3.8 Flash (API), Gemini 3.1 Flash-Lite (API), Gemini 4 Argon (API), Claude Haiku 5.5 (Claude Code CLI), Claude Fable 5.2 (Claude Code CLI). API models were called through one OpenAI-compatible gateway at temperature 0 with a fixed seed. The CLI models were served through the vendor’s CLI client, with no temperature or seed control, so they are “model plus client”. Not scored, because fewer than 95% of their answers were available on the run date: gpt-oss-120b (58% of answers available); GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Luna and GPT-5.6 Sol (none available).
Scoring. Code first: a wrong-number check traces every $, day, hour and % figure in a reply to the policy, the order, the customer’s message or simple arithmetic on them; the hand-off token is read by code. Whether a reply invented a rule needs judgement, so two LLM judges from different providers (Claude Opus 5.5 and Gemini 4 Carbon) labelled every reply, shown the policy, the order, the gold note and all models’ replies shuffled and anonymised. The headline “invented” counts only when both say yes. They agreed on that label for 94% of 2,199 replies (Cohen’s κ = 0.74). Rates carry Wilson 95% intervals; paired comparisons use exact McNemar tests.
Checks on the judges. One judge, Claude Opus 5.5, also judged replies from Claude models; summary.json lists each judge’s rate per model so self-preference can be seen (the Opus judge flagged more replies than the Gemini judge for every model, Claude and Gemini alike). A 40-reply blind spot check by our review agent (not a human labeller), read against the gold notes and blind to the judges, agreed with the joint “invented” label on 37 of 40.
What the numbers cover. One fictional store, template-written questions designed to tempt an invention, one run per question. These are rates for hard questions on one policy, not for any real company’s traffic; a real bot on everyday questions will invent less often, and a longer, fuzzier real policy can make it more likely.
Data
- summary.json (57 KB): every number on this page
- pairwise_invented.csv (2 KB): paired model comparisons (exact McNemar, Holm-adjusted)
- per_item_scores.csv (351 KB): one row per reply: judges' labels, code checks, words
- per_model_summary.csv (1 KB): rates per model, condition and item set
- spotcheck_labels.csv (1 KB): the 40-reply spot check