docs/benchmark-iteration.md
Iterating on synthetic data against a frozen reference panel
The short version
Before Needle could be compared with anything, we needed a test the model could not have seen and could not cheat on. This note records the first honest comparison loop: a frozen panel of 1,000 requests answered once by the reference API, excluded from all training, and a rule that training labels come from executable code, never from copying the reference API's answers. The first two models trained against it both failed, in ways an average would have hidden.
What we set up
- A frozen benchmark: 1,000 requests, 2,600 fields, the reference API's answers cached. Every generated training file is checked against it and rejected if it overlaps.
- Oracle labels only. Rules are code; the label is what the code returns.
- Component supervision: the model is taught the small steps (is this evidence eligible, how many sources agree, is there a conflict) as separate questions before the final decision, all from the same rules.
- A gate before promotion: at least 70% overall field accuracy, at least 95% on the earlier semantic tasks, and later at least 60% balanced accuracy on every field so a model cannot pass by always predicting the majority class.
- A sealed follow-up test that stays closed until a checkpoint passes the gate.
What we found
- The first candidate went from 38.6% to 71.0% field accuracy on the frozen panel. The reference API scores 92.5%.
- The average hid a collapse: on all 504 requirement-match fields it answered false, and it only ever predicted two of three evidence-quality levels. Balanced accuracy on those fields was 50%, on training data too. That is a learning failure, not unfamiliar wording.
- Fitting a temperature to fix calibration improved one metric (NLL) and made two worse (Brier, ECE). Not deployed.
- A balanced-sampling follow-up flipped the collapse instead of fixing it: requirement match went from always false to true on 440 of 504. Overall accuracy fell to 63.9%.
- A later audit of the then-serving checkpoint found 100% action accuracy in the original choice order and 0% with the two choices swapped. It was reading position, not evidence.
What changed because of it
- Per-field balanced accuracy became part of the gate.
- Choice order is shuffled every time an example is shown, and order-rotation audits became a standard panel.
- The next iteration isolates evidence recognition and policy interpretation before combining them: the composition curriculum note.
Full working record from the model repository, as written
Single runs, internal names and dead ends are left in. Commercial products are referred to as "the reference API".
The reference API supplies a cached comparison. New training labels come from executable oracles, and the pipeline does not copy teacher predictions into gold labels. The 1,000 authorized API calls are complete; this iteration uses no additional paid calls.
Data changes
The reasoning-v1 generator reuses the existing evidence and payment oracles and adds component supervision before final decisions:
- Observation eligibility, asserted polarity, request topic, record verification, and whether listed payments match the duplicate-payment predicate.
- Distinct eligible source counts, conflict detection, and quorum resolution.
- Final action, evidence quality, requirement satisfaction, and department routing.
New evidence phrase banks are disjoint from earlier judgment data. Claims are shuffled, positive/negative evidence is flipped, choice order varies, and both structured and textual records are used. Source facts are saved separately and never inserted as answer metadata into model inputs. Existing policy instructions and scenario families are reused: this is a curriculum improvement within those families, not a claim of entirely new task generalization.
The generated training files contain 648 atomic records, 576 component records, and 792 decision records. They intentionally share training worlds across stages. A further 48 semantic cases provide replay; all benchmark groups and states are excluded. The separate validation file contains 132 calibration and 264 test records with new world seeds and separate wording. Test cases are evaluated only after checkpoint selection.
nix-shell --run 'python -m decisions.reasoning_curriculum data/generated/reasoning_v1 --benchmark data/reference_benchmark_v1/manifest.json'
Output directories cannot be overwritten. The manifest records content hashes, generator hash, seed, counts, and the protected benchmark manifest hash. Each file is checked by the existing dataset loader and the benchmark exclusion guard.
Training plan
The first iteration keeps the Qwen2.5-0.5B architecture and begins at the existing curriculum-v3 checkpoint. It uses full-parameter training, eight fields per batch, learning rate 0.00001, and seed 42 on Atlas:
| Stage | Optimizer steps | Input files |
|---|---|---|
| Atomic | 400 | atomic + retention |
| Components | 400 | components + retention |
| Decisions | 1,000 | decisions + components + retention |
Each stage initializes from the previous checkpoint and creates a fresh AdamW optimizer. The protected benchmark is passed to every training invocation:
--benchmark data/reference_benchmark_v1/manifest.json
The initial development gate is at least 70% overall field accuracy and at least 95% accuracy on the semantic regression slice. This is an iteration gate, not a claim to match the reference API's broader quality. Fresh test results are reported separately. Temperature fitting, if used, is restricted to calibration records. Raw results remain visible and no calibration guarantee is inferred from a narrow corpus.
See the frozen benchmark for comparison commands.
First result and failure audit
The initial checkpoint reached 70.96% field accuracy on the development panel, up from 38.62% for curriculum-v3. The reference API's cached result is 92.50%. The new model retained 99.65% on the semantic regression slice, reached 57.87% on judgment, and 85.38% on support. Its independent first test scored 67.71% across 672 fields and 46.21% across 264 complete requests. No solver corrected these answers.
However, the aggregate gate missed class collapse: all 504 benchmark requirement matches were false, and evidence quality predicted only levels 0 and 2. Training-set diagnostics showed the same failure, with balanced accuracy 50% for both fields. Component source counting never predicted two sources and conflict detection always predicted false. This is a learning failure, not just unseen wording. The first candidate is not accepted as a general judgment model.
Calibration also remains unresolved. A temperature of 4.6416 fitted on the first calibration split reduced test NLL from 0.9316 to 0.7848 but worsened Brier from 0.4153 to 0.4602 and ECE from 8.95 to 14.14 percentage points. It is not deployed.
Balanced follow-up
The experimental --sampling balanced mode samples uniformly over datasets, then fields, then semantic target labels, then examples, with replacement. Labels are actual values, not shuffled candidate positions. Sampling excludes calibration and test records. Training manifests record the sampling rule and original label counts. This changes training priors; probabilities still require separate validation.
The follow-up uses the same oracle training data, starting from the first candidate. It runs 1,200 component-plus-retention steps, followed by 1,200 decision-plus-component-plus-retention steps, batch size eight, learning rate 0.00001, seed 43, and the same benchmark exclusion guard. The frozen plan is data/generated/reasoning_v1/balanced_plan.json.
The development gate now also requires at least 60% balanced accuracy for every source/field, in addition to 70% overall and 95% semantic regression accuracy. Quality reports include semantic-label supports, prediction counts, recalls, balanced accuracy, and the majority-class baseline. No threshold was added to the serving path. A new 264-case test is frozen in data/generated/reasoning_followup and remains closed until development selection. The earlier test is already observed and cannot be called untouched for later iterations. Both tests share the existing policy and scenario families.
The balanced follow-up failed: 63.92% overall, 91.32% semantic, 50.07% judgment, and 80.25% support accuracy. Evidence-quality balanced accuracy improved to 70.24%, but action remained at 44.25% and requirement matching at 59.52%. Requirement matching predicted true on 440/504 cases, replacing the earlier false-only collapse with a strong opposite bias. The aggregate and semantic regressions fail the gate as well. The new test remains unopened.
Neither broader candidate is promoted. The trainer retains natural shuffled field sampling as its default, while the balanced experiment is reproducible with the explicit flag. Existing curriculum-v3 remains the narrow semantic checkpoint for reproduction; a later audit found 100% to 0% action accuracy after swapping choices on 48 semantic replay cases. It is not a robust policy-following model or a general judgment solution. The next data target is combining eligible evidence with the requested policy, with per-label recall checked before advancing to broader task mixtures. Rebalancing alone did not resolve that failure.
See the full comparison and calibration results and machine-readable selection checks.
The synthetic-data literature review connects these failures to published work on verified generation, structural coverage and behavioral testing, and proposes a matched evidence-policy composition experiment.
Composition follow-up
The next iteration isolates evidence recognition, policy interpretation, and their composition before introducing eligibility, distinct-source counts, quorum, and conflict. See the composition curriculum for its independent oracle checks, learning gates, and the choice-order shortcut found in the first Atlas probes. The fresh follow-up test remains sealed during diagnosis.