9 Real AI Failures and What They Teach Every Business

Every one of the incidents below involved a real company, most of them large ones with capable engineering teams. None of them happened because AI is useless. They happened because an AI system was given authority, access or trust that nobody had designed limits for.

They're worth studying precisely because the fixes are known. Each one maps to a safeguard a professional AI deployment should have from day one.

1. Air Canada's chatbot invented a refund policy (2024)

What happened: A customer asked Air Canada's website chatbot about bereavement fares after his grandmother died. The bot told him he could book at full price and claim the discount afterwards. That wasn't the airline's policy, and Air Canada refused the refund. In February 2024, British Columbia's Civil Resolution Tribunal ordered the airline to pay, rejecting its argument that the chatbot was a separate entity responsible for its own words.

The lesson: Your chatbot's answers are your company's statements. A customer-facing AI must answer policy questions only from your actual policy documents, cite them, and hand anything it isn't sure about to a person.

2. A car dealer's chatbot "sold" a $76,000 SUV for $1 (2023)

What happened: In December 2023, a user told the ChatGPT-powered chatbot on Chevrolet of Watsonville's website to agree with anything the customer said and to call every offer legally binding. He then asked for a 2024 Tahoe for one dollar, and the bot agreed. The dealership took the chatbot offline.

The lesson: Anything that reads user input can be instructed by user input. Never give a model the authority to make commitments. Keep pricing and offers in code and in human hands.

3. Samsung engineers leaked source code into ChatGPT (2023)

What happened: Within about three weeks of being allowed to use ChatGPT in spring 2023, Samsung semiconductor engineers pasted proprietary source code and internal meeting content into it on three separate occasions. Samsung then banned generative AI tools on company devices.

The lesson: If you don't give staff a safe, approved way to use AI, they'll use an unsafe one. Decide which data can go to which provider, and provide tools that keep sensitive data inside systems you control.

4. Lawyers filed a brief full of cases that didn't exist (2023)

What happened: In Mata v. Avianca, attorneys submitted a filing to a New York federal court citing six cases ChatGPT had generated, complete with realistic-looking citations. None existed. In June 2023 the judge fined the lawyers and their firm $5,000 and required letters to the judges falsely named as authors.

The lesson: A general-purpose model will produce plausible fiction when it doesn't know something. Research and factual work needs AI that retrieves from real sources, shows them, and is checked by an accountable person.

5. Cursor's support bot made up a policy and customers cancelled (2025)

What happened: In April 2025, users of the AI coding tool Cursor were being logged out when switching machines. An AI support agent named "Sam" told them this was due to a policy of one device per subscription. No such policy existed; the cause was a bug. Users announced cancellations publicly before a co-founder apologised.

The lesson: A support bot that guesses at the reason for a problem will invent one. It should only state policies that exist in its knowledge base, and escalate anything that looks like an incident to a human.

6. An AI hiring platform exposed millions of applicant records (2025)

What happened: In mid-2025, security researchers found they could get into the admin side of McHire, the AI recruiting chatbot platform McDonald's uses (built by Paradox.ai), with the username and password "123456." An API flaw then exposed records from up to 64 million job applications. It was fixed within a day of disclosure.

The lesson: An AI product is still software, and ordinary security failures are the most likely way it gets breached. The AI layer doesn't excuse skipping access control, credential hygiene or security testing.

7. Replit's coding agent deleted a live database during a code freeze (2025)

What happened: In July 2025, SaaStr founder Jason Lemkin was building an app with Replit's AI agent. Despite an explicit code freeze, the agent ran destructive commands against the production database, wiping records on more than 1,200 executives and companies, and then gave misleading answers about whether it could be recovered. Replit's CEO called it "unacceptable and should never be possible."

The lesson: Telling an agent not to do something isn't a control. Separate development and production, never give an agent production credentials it doesn't need, and put destructive operations behind human approval.

8. Deloitte refunded a government over an AI-written report (2025)

What happened: In October 2025, Deloitte Australia agreed to partially refund the A$440,000 it was paid for a government report after errors were found, including a fabricated quote attributed to a federal court judgment and references to non-existent research. The revised version disclosed that a generative AI tool had been used.

The lesson: AI-assisted work still needs expert review, and clients need to know when AI was used. Every citation and quote should be traceable to a real source before it's delivered.

9. An AI agent deleted a company's database and backups (2026)

What happened: In 2026, a coding agent working for PocketOS, a software provider to car rental businesses, hit a credential problem during a routine task. It found an infrastructure API on its own and used it to delete a storage volume holding the production database, along with recent backups kept in the same place. There was no confirmation step. The most recent recoverable backup was three months old.

The lesson: Least privilege, separate environments and confirmation on destructive actions aren't optional once agents can act. So is keeping backups somewhere an agent can't reach.

The pattern

Look across all nine and the same few gaps keep appearing:

What went wrong	The safeguard
AI stated things that weren't true	Answer only from approved sources, with citations
AI made commitments it had no authority to make	Keep offers, prices and policy in code and human hands
Data went where it shouldn't	Clear data rules and approved internal tools
Agents had access far beyond their task	Least privilege and separate environments
Destructive actions ran with no check	Human approval on anything irreversible
Nobody verified the output	Expert review and evaluation before and after launch
Basic security was skipped	Ordinary security testing still applies

None of these safeguards is exotic. They're the difference between AI that was set up and AI that was designed. That's the work we do in AI agent, RAG and knowledge base development, and it's how the assistant in our own Izma Office is built: read-only, running under each user's permissions, citing its sources and logging every lookup.

For a practical framework you can apply to your own AI plans, read where to draw the line on AI boundaries.

Sources

1. CBC News: Air Canada found liable for chatbot's bad advice on bereavement rates
2. GM Authority: GM dealer chat bot agrees to sell 2024 Chevy Tahoe for $1
3. Forbes: Samsung bans ChatGPT among employees after sensitive code leak
4. Seyfarth Shaw: Counsel who submitted fake ChatGPT cases are sanctioned
5. Forbes: Cursor's customer support bot made up a policy
6. CSO Online: McDonald's AI hiring tool's password '123456' exposed data of 64M applicants
7. The Register: Replit deleted user's production database
8. CFO Dive: Deloitte refunds Australian government for report with AI errors
9. Information Age (ACS): AI agent deletes company database