Benchmarks
Measured, not marketed
Every number below comes from one matched run: the same 599 requests sent to Needle and to a leading commercial decision API, scored against oracle labels. Neither model trained on them.
- 89.7%
- field accuracy, blind test
- vs 85.8%
- 41 ms
- median latency
- vs 109 ms
- 63 ms
- p95 latency
- vs 192 ms
- 0.240
- negative log likelihood
- vs 0.331, lower is better
Blind test, by panel
Field accuracy on 1,180 fields the models had never seen. Paired difference on the whole set: +1.3 to +6.6 points, 95% interval.
| Panel | Fields | Reference | Needle |
|---|---|---|---|
| Judgment, support, semantic, reasoning tests | 484 | 91.7% | 99.8% |
| Policy conditions: majority, override, veto | 240 | 92.5% | 100% |
| New propositions, layouts and policy text | 216 | 90.3% | 88.9% |
| Policy actions, unconstrained | 240 | 63.3% | 60.0% |
| All blind fields | 1,180 | 85.8% | 89.7% |
What this does not show
Needle is not a general reasoner. On 99 public validation questions from HellaSwag, MMLU and WinoGrande it scores 37.4% against the reference's 96.0%. It is trained for typed decisions over stated evidence and policies, and it should be judged there.
Novel wording is a tie. On propositions, layouts and policy text unlike anything in training, the two models are statistically indistinguishable, and the reference is better calibrated.
One seed, one run. The reference was called once per request with no retries. Needle's latency is measured in-process on our hardware; the reference's includes HTTP to its provider.
The full evidence file, paired statistics (exact McNemar and a request-clustered bootstrap) and the evaluation code ship with the model repository. We will publish them alongside general availability.