Needle against a commercial decision API, by training stage

The short version

An LLM answers a question by writing a sentence one word at a time. If all you wanted was "refund or deny", that is slow, you pay for every word, and sometimes the sentence comes out in a shape your code cannot read. Needle does not write anything. It reads the situation and the options you gave it and, in one pass, says how likely each option is. It is a small model, 395M parameters, a BERT-style encoder rather than a chatbot, trained only to make those choices.

We tested it against a commercial API that does the same job, on the same requests. Inside the kinds of decisions it was trained for it picks the right option more often, in about a third of the time. It knows very little about the world, so on trivia-style questions the commercial API wins easily.

Throughout, that product is called "the reference API". The full experiment record is the marker encoder lab note; the paired statistics are on the benchmarks page.

What Needle is

• A 395M-parameter bidirectional encoder (ModernBERT-large backbone). Each candidate option gets a [MASK] marker; the marker's final hidden state scores that option. One forward pass scores every option in every question.
• Zero output tokens. No sampling. Deterministic: identical prompts never flip the winner (0% on 48 repeats; the reference API flips 2.1%).
• Trained in stages, A through L, on synthetic data labelled by executable oracles. No training example is labelled by copying the reference API's answers. A frozen panel of 1,000 cached reference requests was excluded from all training.
• Later stages train on about 1,700 task families written by an LLM and labelled by running model-written code, so labels are program outputs, not model opinions.

How it was measured

• Paired protocol: identical requests sent to both models. Cross-field constraints sent to neither. Scored against oracle or official labels.
• Statistics: exact McNemar on discordant fields and a request-clustered bootstrap for the 95% interval. Evidence files are published for stages E, G and K.
• Reference API version 1.13.0, called once per request, no retries, none failed.
• Latency: Needle is measured in-process on our GPU; the reference includes HTTP to its provider. Not apples to apples.
• One seed for most stages. Stage K was re-run with a second seed and reproduces within noise on every panel except the small out-of-family holdout (about plus or minus 4 points).

Headline numbers

Field accuracy, Needle against the reference API.

• Stage E, blind test, 1,180 fields: 89.7% vs 85.8%, 95% interval +1.3 to +6.6 points. Negative log likelihood 0.240 vs 0.331. Latency p50/p95 41/63 ms vs 109/192 ms.
• Stage G, blind test: 98.2% vs 85.8%. Policy development 99.8% vs 78.3%. Paired counterfactual 99.9% vs 76.7%. New propositions and layouts generated before G trained: 97.9% vs 88.2%.
• Stage G, order stability: reordering the choices flips the winner 0% of the time vs 8.3% for the reference (15.6% on low-evidence cases). The first-listed option wins 33.3%, which is unbiased.
• Stage K, the current reference checkpoint: blind 97.0%, new wording 98.1%, unseen task families 80.4% vs 96.1%, synthesized held-out families 93.4%.
• Latency at stage G on policy-length prompts: 44/71 ms vs 118/190 ms. On short prompts 20/35 ms vs 102/160 ms.

Where it loses, and by how much

• General knowledge. MMLU 41.0% vs 88.7%, HellaSwag 82.8% vs 92.6%, WinoGrande 73.8% vs 92.6% at stage G. MMLU is bounded by what a 395M model learned in pretraining: better-matched data moved it 1.6 points, a bigger backbone moved it 17 to 29 points with no training at all.
• Task families it never saw. Stage I 66.6%, stage K 80.4%, against the reference's 94 to 96%. An untrained 7B model scores the same as stage I here, so the reference's lead is breadth of training, not size.
• Calibration on novel wording was worse than the reference's at stage E, with accuracy tied.
• Unconstrained policy actions at stage E: 60.0% vs 63.3%. Both sit near 60% without the typed constraint.
• Stage K traded order stability for breadth: the reordered-choices panel fell from 97.8% to 82.9%. Stage L recovered order (90.0%) and gave back out-of-family accuracy (80.4% to 71.6%). At this data volume the two trade off.
• Hand-written task templates only improved the holdout that informed them. The model learns the generator, not rule-reading in general. That is why stage K uses LLM-synthesized families with executable labels: +16 points on the clean holdout where templates gave nothing.

What this does not show

• That Needle is a general reasoner or a drop-in replacement for the reference API. It is a narrow model.
• Calibration. Probabilities on general-knowledge tasks are overconfident; the temperature is fitted on judgment-family splits only.
• A fair latency comparison. Ours is in-process; theirs crosses the internet.
• Anything from a stage other than the one you are reading about. Every number above names its stage.