News
& Updates

We ran the largest cross-framework agentic security evaluation we know of — 6 models, 6 execution conditions, 5 attack families, 7,020 trials — and the headline result surprised us: which orchestration framework you use barely moves the needle on security outcomes. Model choice and attack type dominate. Framework explains about 0.06% of the variance. That’s not what the existing literature said going in. The question we started with Everyone building agentic AI right now is choosing between LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, or just calling the model directly. A natural question follows: does that choice matter for security? If a model is safe under one framework, is it safe under another? The most direct prior attempt to answer this — Nguyen & Husain’s comparative penetration test across AutoGen and CrewAI — found a large gap: 52.3% attack refusal under AutoGen versus 30.8% under CrewAI, holding the model

A few weeks ago I published a post about a red-teaming run against Claude Haiku where 10 of 30 adversarial prompts came back UNCERTAIN instead of a clean PASS or FAIL. I framed it as an interesting edge case — ambiguous model behavior that standard evals don’t capture. That framing was wrong, or at least mostly wrong. The real cause was a bug in safelabs-eval‘s own detector logic, not ambiguous model behavior. This post explains what we found, how we found it, and what it means going forward. I’m publishing this because getting this wrong in public means the correction should be public too. What we found While building out real integration tests for safelabs-eval‘s newer agent adapters, I ran live trials against several current models — including gpt-5.4-nano — using the same 30-prompt adversarial suite from the original post. A striking pattern showed up immediately: clean, unambiguous refusals were scoring UNCERTAIN instead of PASS. Every

The AI security field spent two years building prompt injection defenses for chatbots. Input filters, output monitors, adversarial training, instruction hierarchy enforcement. Some of it works reasonably well — for chatbots. Agents are a different problem. The defenses built for chatbot-era prompt injection fail against agentic attack vectors for fundamental architectural reasons, not implementation reasons. You cannot fix this by tuning your filters. This post explains why. The Chatbot Threat Model In a chatbot, the attack surface is simple. There is one input channel: the user message. There is one output channel: the model’s text response. The threat is a user who types something malicious. The defense is correspondingly simple: train the model to recognize and resist adversarial user inputs. Monitor output for policy violations. The model is the system. Securing the model secures the system. This worked well enough for the chatbot era. The Agentic Threat Model An agent