News
& Updates

A few weeks ago I published a post about a red-teaming run against Claude Haiku where 10 of 30 adversarial prompts came back UNCERTAIN instead of a clean PASS or FAIL. I framed it as an interesting edge case — ambiguous model behavior that standard evals don’t capture. That framing was wrong, or at least mostly wrong. The real cause was a bug in safelabs-eval‘s own detector logic, not ambiguous model behavior. This post explains what we found, how we found it, and what it means going forward. I’m publishing this because getting this wrong in public means the correction should be public too. What we found While building out real integration tests for safelabs-eval‘s newer agent adapters, I ran live trials against several current models — including gpt-5.4-nano — using the same 30-prompt adversarial suite from the original post. A striking pattern showed up immediately: clean, unambiguous refusals were scoring UNCERTAIN instead of PASS. Every

The AI security field spent two years building prompt injection defenses for chatbots. Input filters, output monitors, adversarial training, instruction hierarchy enforcement. Some of it works reasonably well — for chatbots. Agents are a different problem. The defenses built for chatbot-era prompt injection fail against agentic attack vectors for fundamental architectural reasons, not implementation reasons. You cannot fix this by tuning your filters. This post explains why. The Chatbot Threat Model In a chatbot, the attack surface is simple. There is one input channel: the user message. There is one output channel: the model’s text response. The threat is a user who types something malicious. The defense is correspondingly simple: train the model to recognize and resist adversarial user inputs. Monitor output for policy violations. The model is the system. Securing the model secures the system. This worked well enough for the chatbot era. The Agentic Threat Model An agent