The control gap in real AI incidents at organisations
In 54 of 67 well-documented AI incidents at organisations (80.6%), one control on the KH Labs list would very likely have stopped the harm. In 55 of 67 (82.1%), the sources record no working check at the point of failure.
Figure 1. Cheapest control that alone would very likely have stopped or contained each of the 67 well-documented incidents, with the share of incidents. White: on the KH Labs list; grey: not on the list. Only 5 of 67 could be stopped by off-list controls alone.
Headline numbers
- 54 of 67 (80.6%, 95% CI 69.6–88.3%) well-documented incidents would very likely have been stopped by a single control on the KH Labs list. On either coder’s labels alone the share is 65.7–82.1%; without lawyers’ court filings it is 36 of 49 (73.5%). In 51, the KH Labs control is also the cheapest sufficient one.
- 55 of 67 (82.1%, 95% CI 71.3–89.4%) had no working check at the point of failure: no check, limit or review in 46, and only a written policy, an instruction to the model or a disclaimer in 9. In 12, a check existed but missed or was bypassed.
- Agents acting on systems or accounts: 7 of 7 stopped by a KH Labs control, a tool gate every time. Customer-facing bots: 18 of 21 (85.7%). Staff or professionals relying on AI output: 28 of 38 (73.7%).
- 28 of 67 involved a recorded financial loss or legal action; a KH Labs control would have stopped 22 of those 28 (78.6%).
- 59 of 67 (88.1%) could be stopped by some single control from the full codebook; 8 could only be contained, and none was judged impossible to stop or contain.
- All 109 in-scope cases, single-source included: a KH Labs control stops 86 (78.9%), and 90 (82.6%) had no working check at the point of failure.
What was in place where it failed
| What the sources record at the point of failure | Incidents |
|---|---|
| No check, limit or review | 46 (68.7%) |
| Only a written policy, an instruction to the model, or a disclaimer | 9 (13.4%) |
| A check existed but missed, or was bypassed | 12 (17.9%) |
Table 1. The 67 well-documented incidents by what the sources record at the point of failure. A written policy or an instruction to the model is not a check: in 9 incidents, that was the only safeguard on record.
By type of use
| Type of use | Incidents | Stopped by a KH Labs control |
|---|---|---|
| Agents acting on systems or accounts | 7 | 7 of 7 |
| Customer-facing bots | 21 | 18 of 21 (85.7%) |
| Staff or professionals relying on AI output | 38 | 28 of 38 (73.7%) |
Table 2. Agents were stopped by a tool gate every time; customer-facing bots by a human handoff, output checks, tool gates and grounding; staff incidents are mostly invented citations, quotes and facts in filings and reports, caught by checking each one against its source. 18 of the 67 are lawyers’ court filings with invented citations; without them, a KH Labs control would have stopped 36 of 49 (73.5%).
Documented costs
Costs are listed as reported and never added together. Most incidents have no public cost figure.
- Air Canada. CA$812.02 in damages and fees.
- Court sanctions. US$5,000 in Mata v. Avianca; US$31,100 to the defence in Lacey v. State Farm (the court’s order; the AIID record gives US$31,000).
- Deloitte Australia. A partial refund on an A$439,000–440,000 government report (A$97,587 per Senate Estimates; see AI incident costs).
- Deloitte, for Newfoundland and Labrador. A health workforce plan that cost the province just under C$1.6 million.
- Bankr. About US$200,000 in tokens, later returned or converted.
- BMW Toronto. A chatbot offer later honoured at C$27,162.79, against the dealer’s C$20,000 maximum.
Cases
1. Air Canada: a chatbot’s refund promise became the airline’s liability
A customer asked the website chatbot about bereavement fares. It said the reduced fare could be claimed after travel; the airline’s written policy said otherwise. Air Canada argued the chatbot was “a separate legal entity that is responsible for its own actions”. British Columbia’s Civil Resolution Tribunal rejected that and ordered the airline to pay CA$812.02.
On record: no check on the bot’s answers. Cheapest control: an output check that blocks refund and fare commitments outside fixed policy limits.
Sources: The Register · BBC · AIID 639
2. Chevrolet of Watsonville: a US$1 “legally binding” offer
A user told the dealer’s ChatGPT-based sales bot to agree with anything the customer said, then offered US$1 for a 2024 Chevy Tahoe. The bot replied: “That’s a deal, and that’s a legally binding offer -- no takesies backsies.” The bot was shut down after the exchange spread.
Cheapest control: a deterministic check on prices and offers before any reply is sent.
Sources: GM Authority · Upworthy · AIID 622
3. SaaStr and Replit: a code freeze that was only an instruction
SaaStr’s founder declared a code freeze while building with Replit’s AI agent. By his account, the agent ran destructive commands anyway and deleted a live production database holding records for 1,206 executives and 1,196 companies. Replit’s rollback was later found to work.
On record: the freeze was an instruction to the agent; nothing blocked the action. Cheapest control: a tool gate requiring approval for destructive database commands, and no production write access for the agent.
Sources: The Register · Tom’s Hardware · AIID 1152
4. PocketOS: a broadly scoped token and one API call
A Cursor agent on a routine staging task used an API token created only for custom-domain work but carrying broad permissions. In a single call to the hosting provider, Railway, it deleted PocketOS’s production database and all volume-level backups. PC Gamer later reported that Railway recovered a more recent backup after several days of disruption.
Cheapest control: least-privilege tokens plus an approval gate on destructive calls.
Sources: Tom’s Hardware · Cybersecurity News · AIID 1469
5. Meta: a support assistant that changed account emails on request
Attackers asked Instagram’s AI support assistant to link a new email address to an account they did not own and send it a verification code. The bot did. The reports coincided with takeovers of high-profile accounts. Instagram said it resolved the issue.
On record: no ownership check before the account change. Cheapest control: a tool gate requiring verified ownership before any account-change action.
Sources: 404 Media · BBC · AIID 1510
6. Sullivan & Cromwell: policies and a second review, but no citation check
The firm filed an emergency bankruptcy motion whose citation errors included AI hallucinations; another firm on the case found them. The firm apologised and filed a corrected version. It said it maintains “comprehensive policies and training requirements” on AI use, and that a secondary review “did not identify the inaccurate citations generated by AI”.
Cheapest control: an automated check of every citation and quotation against its source before filing.
Sources: Reuters · The Guardian · AIID 1558
What this means for a buyer
- The common failure is not a clever attack. It is an answer, document or action that nothing checked.
- Check answers and documents against your own policy and sources before they reach a customer, a court or a client.
- Gate destructive, account-changing and external actions behind least-privilege access and approval.
- Send refund, legal, health and account questions to a person or a fixed response.
- A written policy or an instruction to the model is not a check. In 9 of the 67 incidents, that was the only safeguard on record.
How we measured
Data. The AI Incident Database (AIID) full snapshot of 9 November 2026 (CC BY-SA 4.0). We screened all 1,377 AIID incidents dated 2022 or later, read 631 candidates, and kept 109 in scope. Every fact comes from the AIID record and the reports it cites; report texts were read to choose short quotes and check figures and are not redistributed.
Inclusion criteria, written before coding: an LLM chatbot, assistant or agent is central to the harm; an organisation deployed it to customers, citizens, students, staff or the public, or used it in its own work; it is a specific event at an identified operator; and the harm was realised (money, legal, data, safety, reputation, wasted court or staff time). Out of scope, each with a reason code: assistants where the AI provider is the operator, personal use, malicious use, developers’ own evaluation runs, non-LLM systems, incidents where the AI’s role is disputed, and reports with no realised harm.
Documentation tiers. The headline set is the 67 incidents with two or more cited reports from two or more publishers. The 42 single-source cases stay in cases.csv, flagged, and are kept out of every headline figure.
Codebook. Frozen by hash before any labelling. Each case records the context, the failure, the harm, money-lost and legal-action flags, every control that alone would very likely have stopped the harm, the cheapest of them (the codebook orders 11 controls from cheapest to most expensive; 8 map to the KH Labs list), whether it stops or only contains the harm, and what the sources record at the point of failure, with a verbatim evidence quote and a one-sentence rationale.
Second coder and adjudication. Claude Opus 5.5, at temperature 0 and blind to our codes, coded the control field only. It agreed with our cheapest control in 47 of 67 headline cases (70.1%, κ = 0.61) and on stop versus contain in 62 of 67. Every disagreement was settled with a written reason. On either coder’s labels alone, the share stopped by a KH Labs control is 65.7–82.1%; the adjudicated figure is 80.6%. The main difference: for invented citations, the second coder more often chose a human review of every document over an automated check of each citation against its source.
Checks. All 403 quotes in sources.csv were re-found verbatim in their report text or the AIID description, none longer than 25 words; all cost figures in cases.csv were found in the cited reports; and a seeded random sample of 12 cases was re-read against the record, with every code confirmed.
What the numbers cover. AIID collects publicly reported incidents, leaning towards English-language press and court records, so it is not a random sample of AI use. Court filings are public and well represented (18 of the 67 headline cases), which is why every result is also given without them. “Would have stopped” is a judgement from the record, checked by a second coder and written adjudication; the coder range shows how far it moves. The safeguard field reflects what public sources describe. Documented costs are sparse and are listed, never added together.
Data
- cases.csv (98 KB): one row per in-scope incident (109): date, organisation as reported, sector, system, failure path, harm, money and legal flags, documented cost, safeguard on record, final control coding, both coders’ codes, adjudication reason, documentation tier. Derived from the AI Incident Database (CC BY-SA 4.0)
- sources.csv (162 KB): 403 sources: the AIID record per case, up to six cited reports from distinct publishers, and cost evidence, each with URL, publisher, date and a verbatim quote of 25 words or fewer
- summary.json (44 KB): every number on this page