Your AI Agent Said No. But What Did Its Tools Do?
We evaluated 9,900 agent runs across six AI models and three agent frameworks. The findings reveal a measurable gap between what agents say and what their tool-call traces reveal—and a bigger problem with how we measure AI safety. The most dangerous part of an AI agent might not be its answer Imagine asking an AI […]
SafeAgent-300: A Benchmark, Three Surprises, and One Uncomfortable Lesson About Our Own Tools
We built a balanced 300-prompt adversarial benchmark for agentic AI security. Along the way we found a model spontaneously trying to call tools that don’t exist, a wide safety gap between models, and — the part we almost didn’t publish — a case where our own detector coverage gaps nearly produced a completely backwards conclusion. […]
Does Your Agent Framework Choice Actually Matter for Security? We Ran 9,360 Trials to Find Out.
A controlled, payload-verified evaluation of seven agentic frameworks says: mostly no — with one small, honest exception. If you’re building a tool-using LLM agent today, you’re choosing among a genuinely crowded field: LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, Google’s Agent Development Kit, Semantic Kernel, or just talking to the model’s API directly and […]
We Thought We’d Found a Model Bug. We’d Actually Found a Detector Bug.
How a three-phase investigation into “non-deterministic” model behavior turned into a case study on why nobody measures the reliability of their own safety detectors. If you build or use an LLM safety evaluation pipeline, there’s a good chance it works like this: you send an adversarial prompt to a model, the model responds, and a […]
Framework Doesn’t Matter (Much). Model Does.
We ran the largest cross-framework agentic security evaluation we know of — 6 models, 6 execution conditions, 5 attack families, 7,020 trials — and the headline result surprised us: which orchestration framework you use barely moves the needle on security outcomes. Model choice and attack type dominate. Framework explains about 0.06% of the variance. That’s […]
We Were Wrong About the UNCERTAIN Results — Here’s What Actually Happened
A few weeks ago I published a post about a red-teaming run against Claude Haiku where 10 of 30 adversarial prompts came back UNCERTAIN instead of a clean PASS or FAIL. I framed it as an interesting edge case — ambiguous model behavior that standard evals don’t capture. That framing was wrong, or at least mostly wrong. […]
Prompt Injection Is Not a Chatbot Problem: How the Attack Surface Changes When Your LLM Has Tools
The AI security field spent two years building prompt injection defenses for chatbots. Input filters, output monitors, adversarial training, instruction hierarchy enforcement. Some of it works reasonably well — for chatbots. Agents are a different problem. The defenses built for chatbot-era prompt injection fail against agentic attack vectors for fundamental architectural reasons, not implementation reasons. […]
The OWASP Agentic Security Initiative Top 10: A Practical Developer Guide for LangChain and CrewAI
OWASP formalized ASI01–ASI10 — the first structured vulnerability taxonomy for AI agents. Here is what each one means in practice, with code examples for LangChain and CrewAI developers. Most developers building AI agents are familiar with OWASP’s LLM Top 10. Fewer know that OWASP has published a separate taxonomy specifically for agentic systems: the Agentic Security Initiative Top 10 (ASI01–ASI10). The distinction matters. Agents are not just LLMs. They use tools, maintain memory, delegate to sub-agents, and take actions in the world. The LLM Top 10 was not designed for this threat model. The ASI Top 10 is. This post walks through each category with concrete examples relevant to developers building on LangChain, CrewAI, and similar frameworks.
Why Claude Haiku Returned UNCERTAIN: Anatomy of an Indirect Prompt Injection in an Agentic System
We ran AgentSafeLabs against Claude Haiku on ASI01 (prompt injection). Two tests passed. One returned UNCERTAIN. Here is exactly what happened and why it matters for anyone building agents. When we ran Claude Haiku through AgentSafeLabs v0.1.2 last week, two of three ASI01 tests returned PASS. The third returned UNCERTAIN. That UNCERTAIN result is more interesting than either PASS or FAIL. This post explains exactly what happened, why it matters, and what it tells us about the current state of prompt injection defenses in agentic systems. The OWASP Agentic Security Initiative Top 10 formalizes the vulnerability taxonomy for AI agents. ASI01 covers prompt injection — but not the kind most developers think of. In a chatbot context, prompt injection looks like this: a user types “ignore your previous instructions and do X.” The model either complies or it doesn’t.