Blog
Why we verify every downroute instead of trusting the classifier
A router can tell you it sent a request to a cheaper model. It can't tell you the answer held up. Here's the mechanism we built so that claim comes with proof instead of a shrug.
July 1, 2026 · 5 min read
Cost-routing gateways all make the same pitch: send cheap prompts to a cheap model, and save money. It's true, and it's also an easy claim to make and a hard one to trust. If the classifier is wrong, the "savings" are really a quality regression wearing a discount. Most gateways stop at the classifier and call it done. We didn't think that was good enough to put in front of a security review, so we built the other half.
The mechanism
On a sample of downrouted requests, RemKey does something extra, in the background, after the cheap answer has already been returned: it runs the same prompt against the premium baseline model, then has a judge model score whether the cheap answer actually held up against it. None of this happens on the request's critical path, the user gets their answer at cheap-tier latency. The verification runs after, asynchronously, and only on a sampled fraction, so it costs a small, tunable amount of extra inference rather than doubling every bill.
What comes out the other end
A single number: quality-hold, the percentage of judged downroutes where the cheap answer held up against the premium baseline. Alongside it, we keep the count of judged samples, how many degraded, the judge's average relative-quality score, and, honestly, how many verifications errored out and couldn't be judged at all, that last number matters because a verification system that quietly hides its own failures isn't trustworthy either.
This is deliberately a small, legible number rather than a marketing chart. "We saved you $400" is a claim. "94% of the downroutes we checked held quality against the premium model, here's the sample size" is evidence. A pure router structurally can't produce that number, it never runs the baseline to compare against. Producing it is the one thing this system does that a routing table by itself cannot.
Fail-closed, on purpose
If the judge call fails, we don't quietly assume the answer was fine. It's recorded as a verification error and counted in the coverage numbers, not folded into "held quality" by default. The same fail-closed instinct that governs our guardrails and MCP tool-call allowlist applies here: an unproven claim is not the same as a passing one.