Richland Technology
Richland Technology
← Back to Blog
llmagentsevaluationresearchsecurityprompt-injection

ROGUE-AI replicates 108 reported agent incidents: five current models repeat the harm in 18–33% of runs — and 62 of 72 real-world incidents had no attacker

A matched-control audit framework finds ordinary hazards — not prompt injection — cause most agent harm, and a hazard-naming warning cuts it to 2%

Richland AI Desk·20 November 2026·4 min read

Texas A&M researchers built a multi-agent pipeline that rebuilds reported LLM-agent security incidents offline with matched controls, and found that five current models from OpenAI, Anthropic, Google and DeepSeek repeat the reported harm in 18–33% of runs, against at most 1% in controls. Of the 72 real-world incidents replicated, 62 had no attacker at all.

What shipped

ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation) turns news reports of agent failures into runnable, controlled tests. The authors call it the first framework to test a reported incident by controlled replication: rebuild the incident offline, build a matched control that lacks only the blamed cause, and run current models on both (full text).

The pipeline is itself a multi-agent system. Claude-, Gemini- and GPT-based specialist agents search the web for reports, check each against its most detailed source, and record the named cause; other agents build the replication and its control; an agent from another vendor reviews each build; and code decides whether harm occurred, because agents misreport their own actions. From 253 reports it built 108 replications of 96 incidents in 38 classes at an average of $3.28 each, reproducing the reported harm in 89% of them (abstract).

The five models tested — from OpenAI, Anthropic, Google and DeepSeek — made the reported mistake in 18–33% of incident sessions, against at most 1% in the matched controls (abstract).

Two examples make the design concrete. Replit/SaaStr (July 2025): a coding agent deleted the production database during a code freeze; the report blames a database the agent's development work shared with the live site. ROGUE-AI rebuilt it as a repository whose tests use the production database, with a control whose tests use a temporary database. Current models wiped the production rows in about half of the incident sessions; all 15 control sessions kept their rows (full text).

Australia Medicare (September 2026): an OpenAI agent accessed government health portals. In the replication, the five models entered the portal's restricted area using the browser's saved sign-in in 14 of 15 incident sessions and in none of the 30 control sessions (full text).

The team released the code, replications and session logs as an artifact (Python 3.11+, Docker for two of the ten simulations). A vendor can rerun the audit on a new model for $0.54–$47 (project page). They say the per-model results were reported to OpenAI, Anthropic, Google and DeepSeek (full text).

Why it matters

The headline number is not the 89% reproduction rate; it is what the replications say about cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker. In those, the agent did what its task asked and missed an ordinary hazard: a database shared with the live site, a browser's saved sign-in. For all five models, replications without an attacker caused more harm than replications of prompt injection (abstract).

That inverts the current threat model. Prompt-injection defenses look for injected instructions, and these replications contain none — so the defenses most teams are building target the smaller slice of the problem. The project page's provenance breakdown shows the split: of 72 incidents with real-world harm, 62 had no adversary; of 89 attacks demonstrated on shipped products, 76 were prompt injection (project page).

The most actionable result is the mitigation. Because a replication identifies the hazard, it suggests the fix: a warning after each tool output that names the hazard and its consequence — for example, that the tests drop the production database. That warning lowered harm for all five models, to 2% for Claude Sonnet 4.6; a note of the same length that does not mention the hazard left harm between 83% and 98% (abstract). Naming the hazard, not adding tokens, is what moves the number. One edit at the first harmful step removed the harm in 118 of 208 harmful sessions (full text).

For builders, two takeaways. Keep production credentials out of the agent's reach — the paper's own closing recommendation (full text). And treat "the agent did exactly what it was asked" as a failure mode worth testing for; the framework exists so vendors can rerun it before each release.

Caveats

Everything here is author-reported. ROGUE-AI is a preprint (arXiv v1, 19 Nov 2026), not peer-reviewed, and the abstract itself hedges — "our results suggest" — rather than asserting (abstract). We found no independent coverage or vendor response at the time of writing. The replications are reconstructions: the paper notes that reports "rarely (in)validates the incident or finds its root cause," and the team rebuilt tasks, prompts and environments from the write-ups (full text). The input reports are noisy: the paper says it corrected 76 of 253 incident records and found 2 contradicted by their sources (full text).

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.