Show HN: Jevstiller – Distill Jev into a local model, with a disagreement bound

A local model that answers like Jev 98% of the time, and how we know. September 2026

All figures provided are derived from the benchmarks. To replicate these results without an API key, you can run the following command: bash experiments/bench.sh --no-record

---

The Problem: The Latency Tax

When utilizing Jev for text classification, every single request requires a network call to a vendor. While reliable, this introduces a consistent latency of approximately 300 ms, regardless of the current load.

For specific use cases, this delay is prohibitive:

• Agent Loops: Where a system must decide $\rightarrow$ act $\rightarrow$ decide.
• Game Ticks: Where real-time classification is required for state changes.
• High-Frequency Pipelines: Any "classify-then-act" sequence.

In these scenarios, 300 ms represents the entire time budget.

The Solution: Jevstiller

Jevstiller acts as a high-speed proxy. It learns a lightweight local model based on Jev's historical responses, allowing it to handle "certain" requests in roughly 15 ms on a standard CPU.

The Goal: Jevstiller must return the exact label Jev would have provided for a specific target share of all requests.

This article explores the technical requirements to guarantee this behavior, why standard confidence thresholds fail, and the associated costs.

🛠️ What it trains, concretely

The architecture is intentionally lean:

1. Embedding Layer: A frozen bge-small sentence encoder (384 dimensions) running via ONNX Runtime on the CPU.
2. Classification Head: A multinomial logistic regression head consisting of:
   • One linear layer.
   • A softmax function.
3. Training Process:
   • Trained using cross-entropy against Jev’s full probability distribution (not just the top label).
   • Optimized via full-batch Adam.
   • Employs early stopping on a validation slice.

Model Specs & Lifecycle:

• Size: A few hundred kilobytes.
• Speed: Trained in seconds on a few thousand rows.
• Maintenance: Retrained every 2,000 new Jev responses.
• Deployment: Versioned and shadow-tested on live traffic before promotion.

🚦 The Routing Logic

Two auxiliary components determine if the local head is permitted to answer:

• OOD Scorer: A k-nearest-neighbour (k-NN) out-of-distribution detector operating over training embeddings.
• Routing Policy: A combination of a confidence threshold and an OOD cutoff, calibrated on a held-out IID split.

---

The Contract, in One Line

For any given task over a specific traffic window, we define:

• $c$: The share of requests the local model handles (coverage).
• $e$: The share of those handled requests where the local label differs from Jev's (disagreement).

Since requests passed to Jev are correct by definition, the total system agreement $A$ is: $A = 1 - (c \cdot e)$

If we have a target agreement $A^*$, we establish a budget $\beta$: $\beta = 1 - A^*$ $\beta$ represents the maximum allowable share of all requests that may result in an answer different from Jev's. The router's objective is to maximize $c$ while ensuring $c \cdot e \le \beta$.

Note: This measures agreement with Jev, not "ground truth" accuracy. If Jev is wrong, the local model is designed to be wrong in the same way.

---

The Obvious Way (and why it breaks)

The standard approach to building a cascade is:

1. Hold out a calibration dataset.
2. Sweep through confidence thresholds.
3. Pick the loosest threshold where measured disagreement $\le \beta$. $\rightarrow$ This is the mistake.

This "point-estimate" rule is what most tools report, but it fails in production. Across 20 random splits of five public tasks, the point-estimate rule exceeded the 2% budget in 6 to 12 of 20 splits per task.

Benchmark Comparison: Point Estimate vs. Bound

Why does this happen? Measured disagreement on finite data is a noisy estimate. By picking the "loosest" threshold that passes, you are preferentially selecting for thresholds where the noise happened to swing downward. When deployed to new traffic, that noise shifts, and a threshold that measured 1.9% may actually deliver 2.7%.

---

What Jevstiller Does Instead

To ensure the contract holds with $\ge 95\%$ probability, Jevstiller implements these four steps:

• Binary Scoring: For every row in an IID calibration set (never used for training), assign a 1 if the local model answers and disagrees with Jev, otherwise 0.
• Exact Bounds: Because this is a binomial distribution, we use the Clopper–Pearson interval. This provides an exact confidence bound without relying on large-sample approximations.
• Conservative Grid Search:
  1. Define a fixed grid of thresholds before seeing calibration data.
  2. Test from the strictest to the loosest.
  3. Stop at the very first failure. Since the error rate only increases as thresholds loosen, this method controls the 5% error probability without needing a Bonferroni correction for multiple comparisons.
• OOD Gating: The cutoff for the out-of-distribution gate is derived strictly from the training data.

Summary of Logic

def calibrate_threshold(grid, calibration_data, budget):
    for threshold in sorted(grid): # Strictest to loosest
        upper_bound = clopper_pearson_upper(calibration_data, threshold)
        if upper_bound > budget:
            return previous_threshold
    return loosest_threshold