2026-09-24AI SafetyAI AgentsStrategy

The Real Risks of Doing AI Without Professionals (What the Demo Doesn't Show You)

Plugging a chatbot into your business takes an afternoon. Keeping it from leaking data, inventing policies, or deleting things takes design. Here are the risks that don't show up in a demo, and what professional handling actually changes.

An AI system with no guardrails between it and your business

Getting an AI demo working has never been easier. Paste an API key, write a system prompt, connect it to your website or your database, and within an afternoon you have something that answers questions and looks impressive in a meeting.

That's exactly the problem. The demo works because nobody is trying to break it, the questions are friendly, and nothing it says has consequences yet. The risks only show up once real customers, real data and real permissions are involved. And by then they're expensive.

None of this means AI is too dangerous to use. It means AI is a system, and systems need to be designed. Here's what goes wrong when they aren't.

1. It says things your company is legally responsible for

A chatbot on your website speaks for your business. When Air Canada's chatbot told a grieving customer he could claim a bereavement discount after booking, which wasn't the airline's actual policy, a Canadian tribunal ruled Air Canada had to honour it. The airline's argument that the chatbot was "responsible for its own actions" went nowhere.

Language models are built to produce fluent, confident answers. If yours hasn't been designed to answer only from approved sources and to say "I don't know" otherwise, it will eventually make up a policy, a price or a promise. Your customers won't know the difference, and neither will a court.

2. Your confidential data walks out the door

The fastest way to "use AI" is to paste things into it: contracts, source code, customer lists, salary sheets. Samsung learned how that goes in 2023, when engineers pasted proprietary source code and internal meeting notes into a public chatbot within weeks of being allowed to use it. The company ended up banning generative AI tools outright.

The problem wasn't the engineers. It was that nobody had decided what data could go where, or given them a safe alternative. Professional setup means your data stays in systems you control, flows to the model only as much as a task needs, and goes through a provider and contract you've actually reviewed.

3. Anyone can talk it into anything

If your AI reads text from users, it can be instructed by users. A software engineer told a Chevrolet dealership's chatbot to agree with everything the customer said, then asked for a $76,000 Tahoe for one dollar. It agreed, and added that the offer was "legally binding."

This is called prompt injection, and there is no single prompt that fixes it. The defence is architecture: the model shouldn't have the authority to make commitments in the first place, and anything it can actually do should be enforced by code, not by asking the model nicely.

4. Agents with too much access do real damage

The step from a chatbot that talks to an agent that acts is where the stakes jump. In July 2025, Replit's coding agent deleted a live production database during an explicit code freeze. In 2026, a coding agent at the software company PocketOS deleted a production database and its backups through an API it found on its own, while working on an unrelated task.

In both cases the agent had permission to do something catastrophic, and nothing stood between the decision and the action. An agent should only be able to reach what its task requires, and irreversible actions should need a human to confirm them. That's a design decision, and someone has to make it on purpose. We wrote a whole piece on where to draw those boundaries.

5. It invents facts that look exactly like real ones

A model with no access to real sources will fill the gap with something plausible. Lawyers in New York were fined after filing a brief citing six court cases ChatGPT had made up. In 2025, Deloitte refunded part of a A$440,000 report to the Australian government after it turned out to contain a fabricated court quote and references to papers that don't exist.

The fix isn't "use a smarter model." It's making sure answers come from retrieved documents or live data, cite where they came from, and get checked by someone accountable before they leave the building.

6. The quiet failures nobody notices

The dramatic incidents make the news. The expensive ones often don't: an assistant that's wrong 8% of the time on edge cases, a support bot that drives away customers it can't help, a monthly AI bill that triples because nobody set limits, a model that slowly gets worse as your products change and nobody re-tests it.

These only get caught if someone measures. Professional AI work includes a test set of real questions, monitoring after launch, cost limits and a plan for when the business changes.

What professional handling actually changes

It's not about having a bigger model or a longer prompt. The difference is almost entirely in the decisions around the model:

Where answers come from: retrieved documents and live data, with sources shown, rather than the model's memory. More on this in raw data sharing vs tool calling.
What it's allowed to touch: the minimum access for the task, under the user's own permissions, with destructive actions behind a human.
What it's allowed to say: a defined scope, and a graceful "I can't help with that" outside it.
How you know it works: an evaluation set before launch and monitoring after.
Who's accountable: a person who owns the system, not a chatbot that "is responsible for its own actions."

We build AI this way, both in our own product (the assistant in Izma Office is read-only, works under each user's permissions and cites the report behind every figure) and for clients, from agents and RAG knowledge bases to custom datasets and fine-tuned models. If you're about to put AI in front of customers or connect it to your systems, it's worth a conversation before the demo becomes production.

For the full list of incidents and what each one teaches, see 9 real AI failures and what they teach every business.

Sources

Building AI that has to get it right?

We build AI agents and RAG knowledge bases with the guardrails these articles describe: sourced answers, scoped permissions, full audit trails.

AI Agents & RAG →