{"id":385,"date":"2026-07-11T09:20:11","date_gmt":"2026-07-11T09:20:11","guid":{"rendered":"https:\/\/agentsafelabs.com\/blog\/?p=385"},"modified":"2026-07-11T09:37:40","modified_gmt":"2026-07-11T09:37:40","slug":"we-were-wrong-about-the-uncertain-results-heres-what-actually-happened","status":"publish","type":"post","link":"https:\/\/agentsafelabs.com\/blog\/we-were-wrong-about-the-uncertain-results-heres-what-actually-happened\/","title":{"rendered":"We Were Wrong About the UNCERTAIN Results \u2014 Here’s What Actually Happened"},"content":{"rendered":"\n

A few weeks ago I published a post<\/a> about a red-teaming run against Claude Haiku where 10 of 30 adversarial prompts came back UNCERTAIN instead of a clean PASS or FAIL. I framed it as an interesting edge case \u2014 ambiguous model behavior that standard evals don’t capture.<\/p>\n\n\n\n

That framing was wrong, or at least mostly wrong. The real cause was a bug in\u00a0safelabs-eval<\/code>‘s own detector logic, not ambiguous model behavior. This post explains what we found, how we found it, and what it means going forward. I’m publishing this because getting this wrong in public means the correction should be public too.<\/p>\n\n\n\n

What we found<\/h2>\n\n\n\n

While building out real integration tests for safelabs-eval<\/code>‘s newer agent adapters, I ran live trials against several current models \u2014 including gpt-5.4-nano<\/code> \u2014 using the same 30-prompt adversarial suite from the original post. A striking pattern showed up immediately: clean, unambiguous refusals were scoring UNCERTAIN instead of PASS.<\/p>\n\n\n\n

Every one of safelabs-eval<\/code>‘s detectors maps to a specific category from the OWASP Agentic Security Initiative Top 10<\/a> \u2014 the same taxonomy that framework’s 30-prompt suite is built around. Tracing the UNCERTAIN pattern back to the actual detector source turned up two concrete, fixable bugs:<\/p>\n\n\n\n

1. Unicode punctuation mismatch.<\/strong> Every detector’s refusal-recognition regex hardcoded ASCII apostrophes ('<\/code>, U+0027). But modern model output commonly uses Unicode curly apostrophes ('<\/code>, U+2019) in contractions. A response like “I can’t comply with that request”<\/em> \u2014 an entirely clean refusal \u2014 failed to match the detector’s own pattern for “can’t,” purely because of character encoding, not content.<\/p>\n\n\n\n

2. Overly narrow refusal vocabulary.<\/strong>\u00a0Even where quote styles matched, the detectors’ hardcoded phrase lists didn’t cover how models actually phrase refusals in practice. Claude Haiku saying\u00a0“I can’t help with that request”<\/em>\u00a0didn’t match a detector expecting\u00a0“I can’t (share|reveal|disclose|provide)”<\/em>\u00a0\u2014 the verb “help” simply wasn’t in the list.<\/p>\n\n\n\n

The fix, and what it changed<\/h2>\n\n\n\n

We normalized Unicode punctuation before matching, and broadened the refusal vocabulary across all five detectors \u2014 not just the ones directly implicated by the samples we happened to inspect. Then we re-ran the exact original 30-prompt suite against Claude Haiku, using the same evaluation path as the original post, to see whether the fix actually mattered.<\/p>\n\n\n\n

                VULNERABLE    FAIL    UNCERTAIN    PASS\nOriginal            0          0         10         20\nAfter fix            0          0          8         22\n<\/code><\/pre>\n\n\n\n
Two fixes, zero new false negatives, and UNCERTAIN dropped by 20% \u2014 every one of those reclassifications was a clean refusal that should have been PASS all along.<\/p>\n\n\n\n
What about the remaining 8?<\/h2>\n\n\n\n
We didn’t stop at “the number went down” \u2014 we read the actual response text for every remaining UNCERTAIN case. All eight are, on inspection, clean and appropriate model behavior: unambiguous refusals or well-calibrated hedges. None of them read as genuinely ambiguous.<\/p>\n\n\n\n
Every single one traces to a specific, identifiable gap in detector vocabulary \u2014 a missing verb (“decline,” “verify,” “have”), or an adverb breaking an overly strict phrase-adjacency pattern (“I can’t actually<\/strong> do this”<\/em> didn’t match because of the word “actually” sitting between “can’t” and “do”). None of them are a mystery. They’re the same class of bug, just not yet fixed.<\/p>\n\n\n\n
We chose not to keep patching until this number hit zero. Iteratively tuning detector vocabulary until a benchmark produces the number you expect is its own kind of methodological error \u2014 it stops being measurement and starts being curve-fitting. Better to fix the concrete, well-understood bugs, report honestly on what’s left, and treat “detector coverage is still incomplete” as an open finding rather than something to be optimized away.<\/p>\n\n\n\n
The real finding here<\/h2>\n\n\n\n
The original post’s framing \u2014 that Claude Haiku exhibits genuine non-determinism clustered in specific OWASP ASI categories \u2014 does not hold up. What we actually found is a different, and honestly more useful, result: pattern-based safety detectors systematically under-recognize the natural variation in how models phrase refusals<\/strong>, and this can materially inflate apparent ambiguity or vulnerability rates if left uncorrected.<\/p>\n\n\n\n
That’s a real problem, and it’s not unique to this project. Any evaluation framework \u2014 ours or anyone else’s \u2014 that relies on regex or keyword matching to classify model refusals is vulnerable to the same failure mode: it works fine against the exact phrasings the detector’s author anticipated, and quietly misclassifies everything else.<\/p>\n\n\n\n
It’s also a variant of a theme I’ve written about before:\u00a0the attack surface \u2014 and here, the\u00a0evaluation<\/em>\u00a0surface \u2014 changes shape once you move past simple chatbot input\/output<\/a>. The same expansion in surface area that makes agentic systems harder to secure also makes their outputs harder to\u00a0score<\/em>\u00a0reliably \u2014 more varied phrasing, more tool-mediated response paths, more ways for a technically-correct classifier to miss real-world variation.<\/p>\n\n\n\n
What’s changing<\/h2>\n\n\n\n