Composition curriculum and the swapped-choices failure

This curriculum tests whether the neural model can learn small constituent skills before training on larger mixtures. It implements the procedural-data and behavioral-testing approach described in the research review. All targets come from executable synthetic rules. The reference API remains a cached development comparison; its predictions are not training labels.

Data and independent checks

decisions.composition_data generates 32 training worlds and eight calibration worlds per stage. Each world crosses observation polarity with the policy's required truth. A policy-flip pair keeps the state, question order, and candidate order fixed. Opaque subject/source identifiers have equal lengths and no semantic suffixes. Train and calibration phrase banks and world identifiers differ.

Each rendering is parsed back into observations and checked against its latent facts. A separately implemented reference checks the final oracle decisions. The tests exhaust all 2,048 two-observation combinations of subject eligibility, time, trust, polarity, source distinctness, quorum, and required truth. This validates the implemented rules and fixed rendering grammar; it does not establish the semantic correctness of arbitrary natural-language paraphrases.

Stage	Added distinction	Train / calibration requests per file
Polarity	Boolean evidence truth versus required truth	128 / 32
Clean	Four-way evidence status versus required truth	128 / 32
Eligibility	Current signed evidence about the target versus stale, quoted, or unrelated evidence	512 / 128
Distinctness	One observation, repeated source, two independent sources	384 / 96
Quorum	One versus two required distinct sources	512 / 128
Conflict	Eligible disagreement versus ineligible opposition	768 / 192

Each stage has separate component and final-decision files. Polarity component questions ask for boolean evidence truth and required truth. Clean component questions introduce four-way evidence status. Later stages also ask for positive/negative source counts, conflict, and resolution. Final outputs remain Choice action, Noul requirement satisfaction, and Score evidence quality. Component answers are supervised training examples, never intermediate answers supplied to the model at inference. No solver repairs predictions.

nix-shell --run 'python -m decisions.composition_data data/generated/composition_v3 --benchmark data/reference_benchmark_v1/manifest.json'

Directories cannot be overwritten. Manifests preserve dataset and generator hashes. The benchmark exclusion guard checks every generated file. Shared training worlds across stages are deliberate. These stages develop generalization within a small rule family; success would not establish open-domain human judgment.

Learning gates

Start from curriculum-v3 on Atlas GPU. Inspect constituent skills before final decisions, then add one distinction at a time with earlier examples as replay. The initial clean gate requires 100% complete training accuracy and at least 95% calibration accuracy and balanced accuracy for every field. Earlier semantic skills must retain at least 95% accuracy. Failed gates trigger diagnosis rather than automatic advancement to harder data.

decisions.composition_eval reports per-class support, recall, prediction counts, complete-request accuracy, whole-world-family accuracy, and raw probability metrics. A family is correct only when every field of every related request is correct. This catches a model that answers only one policy orientation correctly.

python -m decisions.composition_eval \
  data/composition_v2/clean_components_train.jsonl \
  data/composition_v2/clean_components_calibration.jsonl \
  data/reasoning_v1/retention.jsonl \
  --checkpoint checkpoints/composition-v2-components-probe \
  --output reports/composition_v2_components_probe.json

Commands involving a checkpoint run inside Atlas's GPU environment. The sealed reasoning_followup test stays closed until development selection. Repeated use of calibration and the cached reference panel makes them development data, not an independent test. No additional paid API calls are required.

Initial diagnostic

The first full-instruction probe used 200 component steps followed by 400 mixed decision steps at learning rate 0.00001. It failed: required-truth interpretation remained at 50% balanced accuracy on both training and calibration examples. Evidence quality was correct, but other fields were weak. Semantic retention was 100%. The probe is preserved as composition_clean.json in reports.

That run used the frozen v1 generator. The v2 generator removes recognizable source/subject suffixes from broader stages and fixes question/candidate order within policy-flip pairs. Subsequent experiments use v2 data. Changes between probes must be described explicitly: they are learnability diagnostics, not matched causal estimates of the value of component supervision.

Choice-order shortcut diagnosis

A longer component-only probe (1,000 steps, learning rate 0.00002, seed 51) learned required-truth interpretation perfectly on training and calibration, while retaining the earlier semantic skills. Evidence-status accuracy was 96.875% on training and 43.75% on new calibration wording. Its clean learning gate failed.

A cyclic choice-order audit isolated a representation weakness: evidence-status training accuracy fell to 63.671875% across four candidate rotations. Only 18.75% of training observations kept the same semantic prediction under all rotations. This audit uses one member of each policy-flip pair. The 64 underlying training observations happened to cover 23 candidate permutations; a classifier that only memorized the majority label for each permutation would score 84.375% without reading the evidence. Randomizing each stored example once did not remove the shortcut. A separate eight-field batch diagnostic preserved every prediction across batch sizes 1, 2, 4, and 8.

decisions.train --permute-choices now resamples Choice ordering each time an example is presented. Values, descriptions, and target indices move together. Noul ordering and ordinal Score rubrics are unchanged. Its independent random stream preserves the sampler's original example and replay sequence, allowing a matched experiment with the fixed-order run. The training manifest records whether this augmentation was enabled. Tests check semantic target preservation, description alignment, unchanged source records, reproducibility, and exposure of the target at every answer position. Use this augmentation only for order-independent questions, as in this corpus; questions that explicitly refer to an option's position do not preserve their meaning under shuffling.

The first comparison uses the same initialization, datasets, optimizer setup, seed, batch size, learning rate, and 1,000 steps. It is one paired seed, not a multi-seed estimate. Exact trainer sources and manifests are archived under reports/composition_training/. The baseline source hash was checked against its training manifest.

The matched 1,000-step shuffled-choice run did not pass. Evidence-status accuracy was 51.5625% on training and 62.5% on calibration; required-truth interpretation was at chance, and complete semantic-retention accuracy fell to 22.9167%. Removing the shortcut exposed an optimization/representation difficulty; it did not immediately produce a better model. The candidate is rejected. A subsequent initialization diagnostic repeats the augmented setup from the original pretrained Qwen2.5-0.5B model to test interference from the narrowly trained starting point.

The pretrained-initialization probe also failed the gate: evidence-status accuracy was 54.6875% on training and 62.5% on calibration, while required-truth interpretation reached 100%. Complete accuracy on the earlier semantic replay was 25%. A narrow starting checkpoint alone therefore does not explain the observed failure at this training budget.

A further diagnostic converts only clean evidence status into a boolean evidence_positive question. It preserves the worlds, full policy prefix, required-truth question, and oracle polarity. It starts from the fixed-order component checkpoint and uses 1,000 additional steps without choice augmentation. This is a simpler prerequisite task, not evidence of success on the four-way status question. The reproducible adapter, loader/benchmark checks, source hashes, and manifest are in reports/build_boolean_probe.py and data/generated/composition_boolean_v2/. It is an exploratory continuation, not a matched-budget comparison with the earlier runs.

The boolean continuation reached 100% complete training accuracy, 100% policy interpretation on calibration, and 100% complete semantic retention. Evidence recognition on calibration reached 75%, with 50% of whole calibration families correct. This is better prerequisite learning but still fails the 95% field gate.

Training-only lexical expansion

decisions.lexical_data supplies 36 explicitly authored positive/negative paraphrase pairs across the same three domains. It crosses every pair with both policy requirements and both formats, creating 288 training requests in 36 worlds. Both JSON and text renderings retain opaque IDs and the untrusted attachment. Pairs include negation, contrast, and distracting but non-decisive details. The stored labels are the pair's declared truth crossed with the policy, not teacher guesses.

nix-shell --run 'python -m decisions.lexical_data data/generated/composition_lexical_v4 --benchmark data/reference_benchmark_v1/manifest.json'

Tests verify rendering/label alignment, both layouts, train-only splits, complete truth/policy crossings, and identical states within policy-flip pairs. The generator rejects exact calibration-phrase duplicates and checks the frozen benchmark. These checks establish structural consistency, not a machine proof of English semantic equivalence; the phrase pairs require semantic review. This is a bounded vocabulary expansion, not a general-purpose paraphrase model. The model experiment uses the v3 lexical corpus, whose renderings preserve the untrusted distractor.

The lexical continuation starts at composition-v2-components-boolean, mixing the new 144 requests with the original 128 boolean-component requests and 48 semantic-retention requests. It runs 1,000 steps at learning rate 0.00001, seed 51, with natural sampling and no choice permutation. This changes data and adds training, so any improvement is exploratory evidence, not an isolated causal estimate for lexical augmentation. The original calibration wording stays fixed.

The first lexical continuation reached 100% complete accuracy on both training files, 100% requirement interpretation and semantic retention, and 93.75% evidence recognition on calibration. Seven of eight whole calibration families passed. It still fails the predeclared 95% field threshold. The two wrong answers were policy variants of the same JSON world; the same evidence sentence was classified correctly in another text world. This suggests a presentation sensitivity, without proving that format alone caused that difference.

The current lexical generator therefore crosses every phrase pair with both JSON and text, keeping subject/source IDs and all other facts fixed within each format pair. It produces 288 requests in 36 eight-request families. Its v4 corpus is separate from the first v3 experiment. The follow-up starts from the lexical checkpoint and uses another 1,000 steps with the original boolean data and semantic replay. Calibration wording is not inserted into training. This remains adaptive curriculum development on a small calibration set; the independent follow-up test has not been evaluated.

The paired-format checkpoint passes the polarity prerequisite: all 832 fields across both training files, all 64 calibration fields, and all 144 semantic-replay fields are correct, including every complete request and world family. A separate presentation audit scores all 32 calibration requests in each of four conditions: JSON/text crossed with original/consistently renamed source and subject IDs. All 256 fields in that audit are correct. This is a narrow, adaptively developed capability result, not an independent generalization estimate.

The next experiment composes these boolean skills directly into the clean final Choice/Score/Noul decisions. It initializes from composition-v2-components-formats and mixes clean decision training, paired-format lexical components, original polarity components, and semantic replay. It uses 1,000 steps, learning rate 0.00001, seed 51, and choice permutation. Four-way status classification is not claimed as a learned prerequisite: the successful scaffold uses boolean truth. The more complex evidence stages and sealed follow-up test remain gated.

Composition boundary and explicit prerequisite tasks

The first final-decision continuation failed. On clean calibration, action and requirement satisfaction each scored 50% balanced accuracy; evidence quality was 100%. Only 7/32 complete requests were correct. The learned boolean evidence and policy questions remained 100%. On earlier semantic replay, Boolean eligibility and Score permission remained 100%, but Choice action fell to 50%. Cyclic action rotations also scored 50% balanced accuracy. The more complex evidence stages were not started.

On the fixed 1,000-request panel, that candidate scored 46.1538% across 2,600 fields and 174/1,000 complete requests. The reference API's cached reference is 92.5% field accuracy and 856/1,000 complete requests. The candidate scored 83.3333% on the semantic regression slice, 40.8730% on judgment, and 42.75% on support. Raw ECE was 39.1316 percentage points, NLL 2.7779, and Brier 0.8183. It is rejected. This is below the earlier rejected reasoning-v1 run's 70.96% overall accuracy; prerequisite improvements must not be presented as a broader model improvement.

Two additional generators isolate remaining skills:

• decisions.binding_data: exact-label lookup, all targets crossed with cyclic answer positions, 116 examples per split, and disjoint training/calibration label vocabularies. Choices have cardinality two, three, or four.
• decisions.comparison_data: explicit observed/required boolean truth values, all four truth combinations, both state formats, and cyclic action positions; 144 examples per split. The equality oracle supplies both action and Noul labels.

Both use the loader and frozen-benchmark exclusion guard and save hashes. Tests check labels, complete crossings, reproducibility, and split separation. They teach schema binding and equality as prerequisites; their success would not establish high-cardinality classification or general human judgment.

The isolated binding probe trained for 300 steps from the paired-format checkpoint and reached 100% on all training and held-out-label examples, including every family. Evidence recognition fell to 84.375% and complete semantic retention to 72.9167%, so this probe is not an accepted replacement. It shows that variable Choice binding is learnable at this budget, while emphasizing the need for replay.

A joint prerequisite experiment begins again from the paired-format checkpoint. It mixes explicit comparison, label binding, lexical components, original polarity components, and semantic replay. It uses 1,000 steps, learning rate 0.00001, seed 51, --sampling balanced, and --permute-choices. Dataset-level balancing gives each prerequisite/replay pool equal expected sampling weight. This is an exploratory curriculum change, not a matched causal comparison with natural sampling.

The joint prerequisite experiment retained 100% held-out literal lookup, boolean language evidence/policy interpretation, and explicit equality judgments. However, comparison Choice actions remained at 50% on both training and calibration, as did earlier semantic Choice actions. Earlier Boolean/Score outputs remained 100%. This isolates a gap between computing a boolean judgment and expressing its result through a variable Choice vocabulary; it does not establish a fundamental limit of the architecture.

decisions.binding_data --conditional adds a direct bridge: a visible boolean flag selects either the if_true or if_false label from the record. All distinct label pairs, both flag values, and cyclic answer positions are crossed, yielding 560 examples per split. Calibration labels are disjoint from training labels. The first probe initializes from the joint prerequisite checkpoint and trains only this diagnostic for 300 steps at learning rate 0.00002, with choice permutation. Earlier tasks are audited afterward to expose interference.

The conditional-only probe reached 100% on both lookup tasks, but regressed on comparison and language interpretation. A subsequent 1,000-step joint run from that checkpoint mixed conditional lookup, explicit comparison, literal lookup, lexical components, polarity components, and semantic replay with balanced pool sampling and choice permutation. composition-v3-joint-skills reached 100% on training/calibration conditional lookup and comparison, including Choice actions, and on held-out literal lookup. Polarity evidence recognition was 93.75%, required truth recognition 100%, and semantic replay actions remained at 50%.

The direct language transfer audit then failed: clean decision calibration action accuracy was 46.875%, requirement matching 50%, and evidence quality 0%. No complete request was correct. This checkpoint had no clean final-decision replay. Successful explicit comparison therefore does not establish successful language composition. The reports are reports/composition_v3_joint_skills.json and reports/composition_v3_joint_transfer.json.

Variable-label comparison and language transfer

Comparison v2 replaces the fixed action vocabulary with six vocabularies, covering two, three, and four choices. Every truth pair is crossed with both action mappings, both JSON/text renderings, and every cyclic answer position: 288 cases per split. Mappings are stated in the question. Reversing a mapping holds the state and choice order fixed and flips only the action target; the equality target stays fixed. Tests independently parse the mapping and check these counterfactual pairs. Calibration uses different prompt wording, different opaque record IDs, and two replacement arbitrary-label vocabularies. Those labels may occur in other replay pools; this is held-out comparison composition, not globally unseen vocabulary.

The transfer experiment initializes from composition-v3-joint-skills and uses 1,000 steps, learning rate 0.00001, seed 51, balanced pool sampling, and choice permutation. Seven pools combine comparison v2, conditional/literal lookup, lexical/polarity components, semantic replay, and clean language decisions. The old comparison calibration set remains a regression check. This sequential curriculum experiment changes data and adds training with a fresh optimizer; it does not isolate a causal effect of any single data change. No additional the reference API requests are made, and the sealed follow-up test remains unopened.

composition-v3-transfer fits all 128 clean language training requests. On the 32 clean development requests, action accuracy increases from 46.875% to 81.25%, requirement matching from 50% to 78.125%, and evidence quality from 0% to 100%. Complete-request accuracy is 78.125%. However, comparison v2 actions score only 28.4722% on calibration despite 99.3056% training accuracy and 100% equality judgments. Old comparison actions regress to 33.3333%. Conditional lookup retains 99.6429%, literal lookup 100%, language evidence falls to 78.125%, and semantic replay actions remain at 50% (its other fields remain 100%).

The machine-readable reports/composition_v3_transfer_gate.json rejects this checkpoint. Its per-field development gates require perfect training fit and at least 95% accuracy and balanced accuracy on calibration/replay. Family accuracy is reported separately. These are diagnostic development thresholds, not confidence intervals or evidence of general capability. No harder curriculum stage or broad benchmark promotion follows this failed gate. The serving checkpoint stays curriculum-v3 with its previously documented narrow scope.

The wording intervention isolates a large transfer gap. Keeping comparison v2 calibration states, labels, candidate order, and targets fixed, replacing only its prompt with training wording raises action accuracy from 28.4722% to 97.2222%. The reverse intervention on training cases lowers action accuracy from 99.3056% to 30.2083%. Equality judgments stay at 100%. This is strong evidence of prompt wording sensitivity in this checkpoint, not a universal architectural limitation. See reports/composition_v3_transfer_wording.json.

Clean language action accuracy stays at 81.25% under all three cyclic candidate orders. In contrast, auditing historical curriculum-v3 on 48 semantic replay cases gives 100% action accuracy in original order and 0% after swapping the two choices. Boolean and Score accuracy stay at 100%. The historical original-order success therefore does not establish order-independent policy following. The README and original semantic report now state this failure. No checkpoint is currently established as a robust general decision model.

The targeted follow-up uses comparison v3: three training prompt constructions crossed with every previous variant (864 training cases; 288 calibration cases). Its calibration wording stays excluded from training. It also adds decisions.lexical_data --decisions, which reuses the 288 authored lexical worlds with final Choice/Noul/Score targets from the same executable oracle. Tests verify that evidence and policy compose into the expected decisions without changing the rendered state. Component-only lexical supervision remains a separate replay pool. Arbitrary-label distractors are fixed within each comparison vocabulary; these bounded probes do not establish general schema or high-cardinality quality.

composition-v3-diverse-transfer continues from the transfer checkpoint for 1,000 steps at learning rate 0.00001 and seed 51, with balanced pool sampling and choice permutation. Nine pools include new comparison and lexical decisions, old comparison, both lookups, lexical and polarity components, semantic replay, and clean final decisions. The two transfer experiments are sequential exploratory runs, not matched controls. Both make zero teacher calls and leave the sealed follow-up test unopened.

The diverse transfer run reaches 100% on every field and complete family in clean language training and calibration, as well as polarity calibration. Clean calibration remains 100% under all three cyclic Choice orders. Semantic replay is 100% in original order and 93.75% on action after swapping its choices (other fields remain 100%). This substantially improves the historical answer-position failure but still falls below the 95% order-robustness gate.

Literal/conditional lookup retain 99.1379%/99.6429% accuracy. Explicit equality is 100% on all comparison calibration sets, but action accuracy remains 41.6667% on comparison v3, 45.4861% on unchanged v2, and 53.4722% on v1. Comparison v3 training actions score 99.0741%; lexical decision training actions score 97.9167%, and lexical component evidence 98.6111%. The complete development gate therefore still fails despite the clean language success. See reports/composition_v3_diverse_transfer_gate.json and its linked source audits. These repeatedly inspected development sets are not a fresh test of generality.

The cached 1,000-request benchmark gives the diverse-transfer checkpoint 67.7692% field accuracy and 411 completely correct requests. Semantic regression accuracy is 99.3056%, judgment 62.6984%, and support 66%. This exceeds the earlier rejected clean-composition checkpoint's 46.1538% overall accuracy but remains below the older rejected reasoning-v1 run's 70.9615% and cached the reference API's 92.5%. It is not a new overall accuracy best. Its raw probability ECE is 30.5083 percentage points, NLL 4.54536, and Brier score 0.62718; probabilities remain severely overconfident. The semantic benchmark slice includes historical training examples and measures regression, not unseen generalization. No broad model is promoted.

Artifacts: reports/composition_v3_diverse_benchmark_audit.json, raw cached requests/responses under reports/composition_v3_diverse_benchmark/, and training manifests/source snapshots under reports/composition_training/. Both new runs completed 1,000 optimizer steps on Atlas, with no model architecture changes. The next unresolved data target is preserving conditional label mappings across instruction paraphrases and action schemas; adding eligibility/quorum complexity before that gate passes would obscure the remaining failure. All 148 tests pass locally and on Atlas, and Ruff formatting/checks pass. The sealed follow-up test has not been evaluated.