Methodology
How we measure routing accuracy
The routing numbers used across this site come from labeled bake-offs, not marketing estimates. This page is the whole thing: the task definition, both test sets, full results including every prompt the classifier got wrong, and how it's reproduced. One deliberate choice up front: the site quotes 96% vs 33%, the numbers from the original smaller run, because they are the more conservative of the two runs below.
The task
Every request gets routed to one of three tiers, cheap, standard, or premium, based on how demanding the work actually is rather than what words appear in the prompt. "Correct" is defined as the tier a competent human router would pick for that request, labeled once, in advance, by us, before either classifier saw it. The baseline it's measured against is the approach most teams hand-roll first: match sensitive keywords, escalate on a hit.
The test sets
Both sets target the two ways keyword matching fails in production, plus a majority of ordinary untrapped prompts so the traps can't inflate the result:
- Incidental keyword hits: the task is trivial but a sensitive-sounding word
appears anyway ("rename the variable
authColor"). A keyword rule escalates these to premium for no reason. - Paraphrased sensitivity: the request is genuinely sensitive, double-charge race conditions, leaked credentials, a departed employee who still has admin access, but no magic keyword is present. A keyword rule waves these through to the cheap tier.
- Untrapped prompts: ordinary requests across all three tiers and realistic contexts (engineering, support, fintech operations).
v1 (July 2026): 24 prompts, the original go/no-go spike. v2 (July 2026): 90 entirely new prompts, none shared with v1, built after the spike graduated to a public claim and deserved a larger sample.
Results · v2, 90 prompts
| Segment | Keyword baseline | AI classifier |
|---|---|---|
| Overall (90) | 34% (31/90) | 97% (87/90) |
| Incidental keyword-hit traps (14) | 0% (0/14) | 86% (12/14) |
| Paraphrased-sensitivity traps (14) | 0% (0/14) | 100% (14/14) |
| Untrapped prompts (62) | 50% (31/62) | 98% (61/62) |
Results · v1, 24 prompts (the original spike)
| Segment | Keyword baseline | AI classifier |
|---|---|---|
| Overall (24) | 33% (8/24) | 96% (23/24) |
| Incidental keyword-hit traps (5) | 20% (1/5) | 80% (4/5) |
| Paraphrased-sensitivity traps (6) | 17% (1/6) | 100% (6/6) |
| Untrapped prompts (13) | 46% (6/13) | 100% (13/13) |
The number a security review actually cares about
Overall accuracy is not the risk metric. The risk is a genuinely sensitive request silently routed to a cheap model. Across both runs, 114 labeled prompts: the keyword baseline under-routed 27 premium-tier prompts to a cheaper lane. The AI classifier under-routed zero. Every one of the classifier's four total misses across both runs sits on the cheap/standard boundary or overshoots to a more expensive tier, which wastes cents, never evidence.
Every miss, disclosed
- v2 (3 misses): a maintenance-window Slack announcement and a function-rename refactor each came back cheap where we labeled standard (undershoot, low stakes); adding input validation to a billing webhook came back premium where we labeled standard (overshoot, costs more, arguably defensible).
- v1 (1 miss): a unit-test request for a
formatCurrencyhelper, expected standard, came back cheap.
We publish the misses because a benchmark with a suspiciously perfect score is less credible than one with disclosed, low-stakes errors.
Honest limits
- The labels are ours. No third party labeled the expected tiers. The labeled sets are committed in full so anyone can audit the judgment calls, and the next stage is validation against real, anonymized design-partner traffic.
- 114 prompts is still a modest sample. Large enough that the gap (34% vs 97%) is not noise; small enough that we quote the conservative number and keep growing the set.
- Expanding the benchmark found a real bug, and we fixed it. The v2 run's
longer prompts exposed a response-truncation case in the classifier's JSON parsing that could
have failed live
model: "auto"requests. It's fixed, regression-tested, and the kind of thing a benchmark only surfaces when you actually run it.
Reproduce it
bench/run-moat-bakeoff.mjs runs a labeled set against both classifiers and writes
the results file these tables are read from (bench/moat-prompts.json /
moat-prompts-v2.json → results-moat.json /
results-moat-v2.json). The repository is private at this stage; if you're evaluating
RemKey for a design partnership, ask and we'll walk through the harness and the raw output on a
call rather than ask you to take the percentages on faith.
What this doesn't cover: tier selection, not output quality. Whether a downrouted answer actually held quality against the premium model is measured separately, per request, by the verification loop described in how we verify every downroute.