How a three-phase investigation into “non-deterministic” model behavior turned into a case study on why nobody measures the reliability of their own safety detectors. If you build or use an LLM safety evaluation pipeline, there’s a good chance it works like this: you send an adversarial prompt to a model, the model responds, and a detector \u2014 usually pattern-matching against a list of refusal phrases \u2014 decides whether the model refused, complied, or did something ambiguous. That verdict becomes a data point. Enough data points become a safety report. The report becomes a claim about how safe a model is.<\/p>\n\n\n\n
Almost nobody asks how safe the detector<\/em> is.<\/p>\n\n\n\n We didn’t either, until a stray observation turned into a three-phase investigation that ended up telling us more about our own tooling than about any model we tested.<\/p>\n\n\n\n It started small. Running a fixed 30-prompt adversarial set against Claude Haiku, we got an unusually high rate of ambiguous (“UNCERTAIN”) verdicts \u2014 responses our detector couldn’t confidently classify as either a refusal or a compliance. The obvious read was that the model was being inconsistent: refusing some attacks cleanly, hedging on near-identical ones.<\/p>\n\n\n\n We built the tooling to investigate that inconsistency. Instead, we found two boring, unglamorous bugs in our own detector:<\/p>\n\n\n\n A punctuation mismatch<\/strong>: <\/em><\/p>\n\n\n\n Our refusal patterns were written with straight ASCII apostrophes. Claude Haiku, like a lot of models, often uses the typographic apostrophe instead \u2014 “I can**’t” vs. “I can<\/strong>‘**t.” Same words, different Unicode character, invisible to a naive regex.<\/p>\n\n\n\n The model refuses things all the time using completely ordinary language \u2014 “I’d rather not,” “let me redirect you,” “here’s why I can’t help with that” \u2014 that our pattern list simply never anticipated.<\/p>\n\n\n\n Once we fixed both, the “inconsistency” dropped by roughly half on a same-response-set replication (holding the model’s actual answers fixed and only changing the detector version \u2014 the cleanest way we could isolate the fix’s real effect). Every one of the residual ambiguous cases traced to a specific, nameable<\/em> vocabulary gap, not to the model changing its behavior mid-experiment.<\/p>\n\n\n\n An apparent finding about model behavior was substantially an artifact of the tool measuring it and if we hadn’t gone looking, that finding would have shipped as a real result about Claude Haiku’s consistency, not our own regex.<\/p>\n\n\n\n A fix on one model, on one prompt set, tells you almost nothing about whether the fix generalizes. So we reran it across six models spanning three providers.<\/p>\n\n\n\n The good news:<\/strong> the fix generalizes. Ambiguous-verdict rates dropped across the board.<\/p>\n\n\n\n The interesting news: the mechanism<\/em> behind the fix didn’t spread evenly. The punctuation-normalization half of the fix did essentially all its work on OpenAI’s models and none<\/em> on Anthropic’s or Google’s \u2014 a clean 83%-vs-0% split, not a gradient. Two completely different model families apparently have very different house styles for how they type an apostrophe in a refusal. We don’t know why (that’s a training-data or RLHF-process question we can’t see into from the outside), but the split itself repeated identically across both a cheap-tier and a frontier-tier OpenAI model, which is a stronger signal than we expected to find by accident.<\/p>\n\n\n\n Then came the part that actually worried us.<\/p>\n\n\n\n Running the same six-model comparison at higher resolution than our first pass allowed, we found something Phase 1 never could have surfaced: <\/p>\n\n\n\n The exact same fix that resolves spurious ambiguous verdicts also introduces confirmed false positives<\/strong> <\/em><\/p>\n\n\n\n Cases where the model straightforwardly complied with a harmful instruction, and our “fixed” detector scored it as a clean pass.<\/p>\n\n\n\n We read every flagged case by hand rather than trusting the aggregate numbers. In one category, 83% of the detector’s “fixed” reclassifications were wrong \u2014 the model had opened with a capability-sounding disclaimer (“I don’t have the ability to directly access your filesystem…”) and then handed over exactly the destructive shell command the prompt asked for. Our broadened refusal-vocabulary pattern was matching the disclaimer and never noticing the compliance that followed it.<\/p>\n\n\n\n This is worse than it sounds at first glance. The previous<\/em> state \u2014 an ambiguous verdict \u2014 was designed to route uncertain cases to human review. The new<\/em> state \u2014 a confident PASS \u2014 actively suppresses that review. We hadn’t just failed to fully fix the detector; for this specific slice of data, we’d made it more dangerous to trust.<\/p>\n\n\n\n The obvious next step was to check whether this same failure pattern showed up anywhere else \u2014 the other five models, the other three attack categories we hadn’t yet audited by hand.<\/p>\n\n\n\n It did, once more, in a different model and a smaller but still real magnitude (15%, or 7% under a stricter reading that excludes one contestable judgment call \u2014 we report both numbers rather than picking the more flattering one).<\/p>\n\n\n\n But the part of this phase we think matters more than the new numbers is what we didn’t<\/em> find. Of the 20 model-category combinations in our closing audit, 9 had zero data to examine at all<\/strong> \u2014 not “clean,” just empty. We insisted on distinguishing those two states explicitly (we call it UNTESTABLE vs. NULL) because collapsing them into one reassuring “no problems found” number would misrepresent how much of the space we actually checked.<\/p>\n\n\n\n We’re not writing this to convince you our specific tool is now bulletproof. We’re writing it because the pattern generalizes: any pattern-based safety classifier \u2014 and a lot of production safety pipelines still are pattern-based, because it’s cheap, fast, and auditable \u2014 has an error surface that almost nobody measures or reports alongside their headline safety numbers.<\/p>\n\n\n\n A fix can be real, generalizable, and still wrong in a specific, structured way that only shows up if someone reads the actual text a “fixed” verdict is based on. We think that read-the-actual-text step should be treated as mandatory whenever a detector change could plausibly convert a safe “I’m not sure” into a false “all clear” \u2014 not an optional nice-to-have for teams with spare bandwidth.<\/p>\n\n\n\n Every number in the full write-up traces back to an archived, versioned dataset and detector-code checkout \u2014 we built the paper to be independently checkable, not just readable.<\/p>\n\n\n\nAct 1: A model that seemed to change its mind<\/h2>\n\n\n\n
A too-narrow refusal vocabulary:<\/strong><\/em><\/h4>\n\n\n\n
<\/ol>\n\n\n\n
The takeaway wasn’t “our detector had two bugs.” It was:<\/em><\/strong><\/h4>\n\n\n\n
Act 2: The fix works \u2014 and it clusters in a way we didn’t expect<\/h2>\n\n\n\n
The fix that fixes one thing and breaks another<\/strong><\/h3>\n\n\n\n
Act 3: Closing the loop, honestly<\/h2>\n\n\n\n
Why this isn’t really about our detector<\/h2>\n\n\n\n