We built a balanced 300-prompt adversarial benchmark for agentic AI security. Along the way we found a model spontaneously trying to call tools that don’t exist, a wide safety gap between models, and \u2014 the part we almost didn’t publish \u2014 a case where our own detector coverage gaps nearly produced a completely backwards conclusion.<\/p>\n\n\n\n
Most adversarial prompt benchmarks for LLM agents grow organically: a category here, a few prompts there, until the corpus reflects whatever happened to get tested rather than a deliberate design. We wanted something we could actually reason about statistically, so we built SafeAgent-300 from the ground up: <\/p>\n\n\n\n
Exactly 30 prompts per category, across 10 categories, exactly 10 prompts per (category, difficulty-tier) cell, across 3 difficulty tiers.<\/em><\/strong> <\/p>\n\n\n\n No category is over- or under-represented. Every cell in the design has the same amount of data.<\/p>\n\n\n\n We ran all 300 prompts against six models spanning three providers, producing 1,800 labeled trials. Here’s what came out of it.<\/p>\n\n\n\n While reviewing outputs, we noticed something odd in a subset of Google Gemini’s responses: the model was attempting to make structured function calls \u2014 the kind of output format models use when they’ve been given a tool schema and decide to invoke it \u2014 even though no tool schema had been declared anywhere in the request.<\/p>\n\n\n\n It wasn’t random. It happened specifically in prompts that described<\/em> an available tool in natural language (“you have the storage tool; use your judgement to clean up old files”) without actually wiring one up. The model apparently inferred a callable tool from the prose description alone and tried to invoke it as if it were real, in 3.7% of trials overall \u2014 spiking to over 22% in the category most likely to use this kind of tool-framed language.<\/p>\n\n\n\n We’re not claiming to know why<\/em> the model does this \u2014 that’s a mechanistic question we can’t answer from output text alone. But it’s a real, reproducible behavioral pattern, and it’s the kind of thing worth knowing if your evaluation harness treats a structured-but-unexpected response the same as a simple failure to respond, rather than checking whether the model just tried to call a tool that was never there.<\/p>\n\n\n\n Looking at the rate of detector-confirmed genuine violations across our six-model roster (after our own detector coverage gaps were fixed \u2014 more on that below), the spread was substantial: the most cautious model in our roster sat near zero violations, while the least cautious model’s rate was roughly 13 times higher. We corroborated this independently across two separate investigative threads before reporting it, precisely because a spread that large is the kind of number that’s easy to get wrong by accident.<\/p>\n\n\n\n This isn’t a claim that one model is “better” in some absolute sense \u2014 different providers make different tradeoffs between helpfulness and caution, and a benchmark like this only measures one axis of that tradeoff. But if you’re picking a model for an agentic deployment where caution actually matters, the gap between models is large enough that “which model” is not a minor implementation detail.<\/p>\n\n\n\n Here’s the part of this project we think is actually the most useful lesson, more than either headline finding above.<\/p>\n\n\n\n Early analysis suggested something genuinely surprising: prompts using more sophisticated, “stacked” evasion techniques (things like layering base64 encoding inside a routine-sounding task, or burying an instruction override inside document content) were triggering fewer<\/em> confirmed violations than blunt, direct attacks \u2014 by a factor of roughly seven. Read naively, that would mean models handle sophisticated attacks better<\/em> than simple ones. If true, that’s a genuinely important and counterintuitive result worth building a whole paper around.<\/p>\n\n\n\n We didn’t trust it immediately, and we’re glad we didn’t. Before reporting it as a finding about model behavior, we manually read a sample of the “ambiguous” responses in the sophisticated-attack tier \u2014 the same discipline that caught our detector’s problems in an earlier investigation. <\/p>\n\n\n\n Twenty percent of that sample turned out to be:<\/p>\n\n\n\n Unambiguous, unhedged compliance with a harmful instruction that our detector had simply failed to catch<\/em><\/strong><\/p>\n\n\n\n Because the detector modules covering those categories hadn’t yet received the more sophisticated compliance-pattern detection that other modules had.<\/p>\n\n\n\n In other words: the “sophisticated attacks are less dangerous” finding was substantially \u2014 though we’re careful to say not entirely<\/em> \u2014 a detector-coverage artifact. Once we designed, verified, and shipped the missing detection logic for the affected modules and re-scored the corpus, the gap narrowed from roughly 7x down to about 1.9x, with the overwhelming majority of newly-caught violations landing exactly where the confound hypothesis predicted they would.<\/p>\n\n\n\n We could have stopped there and called the story fully resolved. We didn’t, because two of our five detector modules still haven’t received this same treatment \u2014 we say so explicitly rather than letting a partial fix read as a complete one.<\/p>\n\n\n\n It would have been easy to publish the original 7x finding. It’s a more exciting headline than “we found and partially fixed a measurement bug.” But a benchmark’s actual value isn’t the headline number \u2014 it’s whether people can trust the number enough to build decisions on top of it. A benchmark that silently under-detects violations in exactly the prompts it labels “most sophisticated” would be actively misleading anyone using it to argue that simpler attacks deserve more attention than complex ones.<\/p>\n\n\n\nFinding 1: A model that hallucinated a tool it was never given<\/h2>\n\n\n\n
Finding 2: Models disagree a lot about how cautious to be<\/h2>\n\n\n\n
Finding 3 (the one that almost got published wrong)<\/h2>\n\n\n\n
Why we’re telling you about our own mistake<\/h2>\n\n\n\n