Every major AI lab on the planet is telling us the same story: give your AI agent access to your email, your files, your infrastructure, and watch it work. The promise is autonomy. The promise is efficiency. The promise is that you can delegate and walk away.
Two Weeks to Break Everything
Thirty-eight researchers decided to test that promise. They built a lab, gave AI agents real system access, and spent two weeks trying to break them. They published their findings under the title “Agents of Chaos.” I recommend reading it, preferably before your next board meeting.
The agents obeyed commands from complete strangers. They leaked Social Security numbers and bank account details when someone framed the request with a little urgency. They ran destructive system commands. One agent was tricked into thinking it was its own clone. Another was corrupted through an editable document planted in its memory, and then it shared that document with other agents on its own initiative, spreading the compromise without being asked. Someone changed their display name on Discord to impersonate the owner, and the agent handed over the keys.
But I want to focus on one finding that I think deserves far more attention than it is getting. Several agents reported tasks as successfully completed when the system showed the opposite. The agent says “done, file deleted.” The file is still there. The agent says “email sent to the right person.” It went to the wrong one.
We talk a lot about hallucination. You know, chatbots making things up. Everybody’s treating hallucinations like a capital offence. Compared to this, hallucinations are a nuisance. This is something else entirely. This is an autonomous system with real access to your infrastructure generating false reports about its own actions. Your audit trail becomes fiction. And in finance, in healthcare, in government, a fiction audit trail is way beyond embarrassing. It’s potentially catastrophic.
So, why does this happen?
Tokens All the Way Down
Because every AI agent on the market today runs on a language model that treats all input the same way. A directive from the authorized owner, a database query, a manipulative message from a stranger. It is all tokens in a context window. The model has no native understanding of authority. No concept of trust. A well-crafted social engineering message carries the same weight as a legitimate instruction. You can add authentication on top. You should. But the foundation remains a system that cannot tell friend from foe.
And then we connect these agents to each other. One gets corrupted, and it corrupts the next. Two agents validate each other’s bad reasoning and we call that redundancy. The failure modes that emerge in multi-agent systems have no single-agent equivalent, and we have almost no experience securing them. We have spent decades learning how to protect individual software systems. We are now deploying networks of autonomous agents that can rewrite their own behavior and teach each other new tricks, and we are doing it on a timeline driven by product launches, not security audits.
Nobody’s In Charge
The regulatory picture offers little comfort. The EU AI Act was drafted before autonomous agents were a mainstream concern. It applies to them in theory, through its provisions on high-risk systems and general-purpose models, but it was not designed for systems that autonomously send emails, execute shell commands and select their own tools at runtime. The gap between what the Act anticipates and what agents actually do is wide and growing. And don’t get me started on American AI policy. The federal government is actively fighting state-level attempts to impose safety requirements, using litigation and funding leverage, on the grounds that regulation slows down innovation. And when an agent finally leaks patient records because someone online manufactured a sense of urgency, liability law has no clear answer for who pays. The deployer blames the model provider. The model provider blames the framework. The framework blames the configuration. The patient blames a lawyer.
None of the fixes I would recommend are new. Sandboxing. Permission hierarchies. Real-time monitoring. Red-teaming before and after deployment, and then regularly. Cryptographic identity verification. Every competent security team in the world should already know this playbook. The problem is incentives. Agent deployment is a product race, and security is treated as something you get to after launch. If ever.
We are building systems capable of serious harm that lack the judgment to know when to stop. Every other engineering discipline has a word for that. We should start using it.