LLMs do not fail with a bang. They leak, drift, and comply their way into trouble. Rather than a rogue model, the real danger is a polite one, steered off course by clever inputs its creators never imagined.
After a year of jailbreak demos and AI agents emailing strangers, Germany’s Federal Office for Information Security has decided to weigh in. Its new guide, “Evasion Attacks on LLMs – Countermeasures in Practice“, is the agency’s attempt to bring order to the chaos. It speaks directly to developers and IT leads who have deployed pre-trained models in production and now live with the uneasy feeling that someone, somewhere, is feeding those systems malicious prompts.
The report draws a clear line between training time poisoning and inference time attacks, the latter being where the real action is. It maps how hostile inputs sneak past filters, disguise commands, or hijack memory. You get categories, threat positions, and examples that are all far too plausible.
The document earns its keep with specifics. It shows how a hidden instruction in an AI assistant’s memory can quietly exfiltrate user data. How a poisoned public issue can pull a code agent into a private repository. How a “rule file” can smuggle a backdoor across forks. Examples that describe the daily reality of connected agents processing untrusted content.
From there, the report organizes its advice into four layers: management, human, system, and model. I find it refreshingly practical. There are safer system prompts with no secrets and clear rules. Guardrails for what users and tools can do. Filters that strip out URLs, code, or odd characters. Normalization and paraphrasing that blunt encoded or adversarial payloads. Least privilege for every tool. Redaction and verification for uploaded files. And model side work like adversarial training and instruction tuning. The logic is straightforward. Layer defenses where data enters, where it is stored, where tools run, and where responses leave.
The authors are honest about limitations. Filters will miss things. Encrypted payloads can slip through. A single misplaced delimiter can change context and neutralize alignment. The best security practice still collapses when teams skip hygiene or let agents run with broad entitlements. The report’s most valuable contribution is architectural discipline. Push decisions into explicit plans. Keep tools sandboxed. Treat every external datum as hostile until proven otherwise. Give every automated action a narrow permission scope and a human checkpoint when it matters. If you are building an assistant that browses or sends mail, assume someone will try to make it mail them. That mindset alone will save you grief.
In the end, the BSI’s paper is a dose of realism in a field addicted to magic prompts. Evasion and prompt injection are supply chain problems wearing a UX disguise. You cannot charm your way out with clever phrasing. You need inventories, default deny policies, and clear accountability for what the AI is allowed to do.
Call that dull if you like. But shipping boring controls beats apologizing for creative failures. AI won’t implode from one jailbreak. It will die from a thousand polite approvals.
Discipline, not genius, will keep it alive.