A bidirectional marker encoder, measured against the reference API

MarkerEncoder packs one question, its candidates, and the state into a single ModernBERT-large sequence. Each candidate is preceded by a [MASK] marker whose final hidden state is the candidate feature; markers attend to the full state in both directions. The existing scalar readout and constrained joint head are reused. Inputs over the token limit raise rather than truncate.

ROCm attention gradients

On Atlas (ROCm, torch 2.8, transformers 5.17) ModernBERT's sdpa attention gives a correct forward pass and a wrong backward pass: gradient norm 836,172 against 40.7 with eager on the same request, growing about 100,000× from the last layer to layer 0. Training under sdpa stays at chance. The encoder loads with attn_implementation="eager".

Training

experiments/train_curriculum.py: 16 requests per update, linear warmup and decay, backbone learning rate 0.00001, head learning rate 0.001, seed 97. A cold start on the prose corpus stays at chance, so stages run as a curriculum, each from the previous checkpoint. Every file passes the frozen reference panel exclusion; labels come from oracles only.

Stage	Updates	Data
A	400	joint_fit_message_v1 truth tables
B	1,500	rlcd_synth_v3 + random-list and joint-fit replay
D	2,000	B's data, half of samples from reasoning_v1 and the benchmark-clean train cases of judgment_v1 (72) and support_v1 (80); label smoothing 0.1

One joint temperature is then fitted on 432 calibration-split requests (reasoning_v1, judgment_v1, support_v1, semantic_v1) and stored in the checkpoint. Stage D fits 0.858. An earlier unsmoothed stage C needed 4.30.

Atlas result

One seed. The cached panel is a development panel, not a blind test; reasoning test splits were not evaluated. No new reference API calls.

cached reference panel, 2,600 fields	Marker D	Marker C + temperature	Qwen replay-plain	Reference API
Field accuracy	96.54%	91.92%	49.71%	92.50%
NLL	0.157	0.250	—	0.219
Brier	0.068	0.128	—	0.119
ECE, 10 bins	0.014	0.029	—	0.019
Complete requests correct	944	865	—	—
Latency p50 / p95 ms	37 / 53	37 / 54	58 / 88	115 / 187

Marker D reaches 100% on policy development, paired counterfactual (640 requests each, NLL 0.032) and random-list retention, against 80.6% policy development for Qwen replay-plain. Reasoning calibration split: 99.70% field accuracy, NLL 0.039. Remaining cached-panel errors are all in judgment_v1: action 94.2%, evidence quality 95.2%, requirement match 92.7% (balanced 89.4%). Public validation stays at chance (35.7% over hellaswag, mmlu, winogrande); this model is not a general reasoner.

reference latency includes HTTP; local latency is serial in-process scoring.

Wording data and blind test

Stage D's calibration-split errors on judgment_v1 were polarity errors on unseen sentences, mostly under a "must be false" requirement, not counting errors. The existing banks hold 3 propositions with 2–4 train phrasings each, and every split uses its own evidence layout. decisions/wording_data.py generates 3,312 train-only cases: 15 propositions (12 new domains), 6–8 new phrasings per polarity, 5 layouts, 3 policy paraphrases, new injection strings, and single-sentence assertion drills. Labels come from the judgment oracle. A test asserts the bank shares no sentence with any existing train, calibration or test phrase. Stage E is stage D's recipe plus this file; temperature 0.736.

cached reference panel, 2,600 fields	Marker E	Marker D	Reference API
Field accuracy	99.65%	96.54%	92.50%
NLL	0.024	0.157	0.219
Brier	0.005	0.068	0.119
ECE, 10 bins	0.013	0.014	0.019
Complete requests correct	991	944	—
Latency p50 / p95 ms	41 / 57	37 / 53	115 / 187

Test splits (experiments/evaluate_blind.py, up to 256 requests per file, joint accuracy). D was frozen before its single test run. E was selected on the cached panel and calibration splits only, but its run is the second look at these files. The reference API was not run on them.

Test file	Marker E	Marker D	Qwen replay-plain
judgment_v1 (192)	95.3%	83.9%	27.6%
reasoning_v1	97.3%	98.4%	18.8%
reasoning_followup	99.2%	99.6%	24.2%
support_v1 (60)	100%	100%	18.3%
semantic_v1 (48)	97.9%	97.9%	33.3%
10 rlcd_synth_v3 policies	100% each	100% each	48.0–100%

E keeps 100% policy development, counterfactual and retention accuracy.

Matched reference comparison on new requests

599 new paid the reference API (v1.13.0) calls (no retries, none failed) on requests neither model trained on, scored against oracle or official labels. Both models receive the same payload; cross-field constraints are not sent to either, so local numbers here are lower than the constrained evaluations above. Paired statistics come from experiments/compare_cached_panel.py: exact McNemar on discordant fields and a request-clustered bootstrap. Evidence.

Panel	Fields	Reference acc.	Marker E acc.	Only the reference API / only E correct	McNemar p	Reference NLL	E NLL
Cached development	2,600	92.5%	99.7%	9 / 195	1e-46	0.219	0.024
Blind test, all	1,180	85.8%	89.7%	71 / 117	1e-3	0.331	0.240
— judgment, support, semantic, reasoning-followup tests	484	91.7%	99.8%	1 / 40	—	—	—
— unseen_v1 (new propositions, layout, policy text)	216	90.3%	88.9%	22 / 19	—	0.240	0.376
— policy condition (majority, override, veto)	240	92.5%	100%	0 / 18	—	—	—
— policy action, unconstrained	240	63.3%	60.0%	48 / 40	—	0.688	0.795
Public validation (HellaSwag, MMLU, WinoGrande)	99	96.0%	37.4%	59 / 1	1e-16	0.180	1.310

Blind-test accuracy difference, 95% interval: +1.3 to +6.6 points; complete requests correct 390 against 363 of 500. At 75% coverage the marker model is 98.8% accurate against the reference API's 93.2%. Latency p50 / p95: 41 / 63 ms local against 109 / 192 ms for the reference API over HTTP.

What this shows:

• Inside the trained task families the marker model beats the reference API on held-out wording, in accuracy and probability quality.
• On unseen_v1 the two are statistically tied on accuracy, and the reference API is better calibrated. The reference API is ahead on action (91.7% against 81.9%, p = 0.17).
• The policy action field is only solved locally with the typed constraint that ties it to condition; without it both models are near 60%.
• On general knowledge and commonsense questions the marker model is at chance and the reference API is at 96%. This is a narrow model, not a the reference API replacement.

decisions/unseen_data.py generates unseen_v1 (test-only; a test asserts no sentence or proposition is shared with any training generator). decisions.teacher takes --split and --ungrouped; collecting a non-training split is an explicit opt-in. The new collections are frozen as data/reference_blind_v1 and data/reference_public_v1 and must never become training sources.

Stages F and G: public train splits, constraint dropout, policy paraphrases

Stage F continues E for 6,000 updates with three changes. decisions/public_train_data.py samples 10,000 rows per task from the official HellaSwag, WinoGrande and MMLU auxiliary train splits in the frozen typed format (validation rows and states over 1,500 characters excluded); these take about 45% of samples. --constraint-dropout 0.5 removes the typed constraints from half the requests, because the blind comparison showed policy action was only solved through its constraint. wording_v2 widens the bank to 31 propositions.

F regressed on unseen_v1 action (81.9% to 75.0%). Every error was a review case (conflict, quorum shortfall, repeated source): the review rule was known only in the wording of four policy texts. Stage G (2,500 updates from F) trains on wording_v3: six more policy paraphrases plus "do kept observations disagree" and "is it resolved" drills over all propositions. unseen_v1 and order_v1 were used for that diagnosis and are development sets for G; unseen_v2 (new propositions, XML layout, new policy text) was generated before G trained.

All panels below are paired with the reference API (v1.13.0) on identical requests, constraints sent to neither, G temperature 0.631. Evidence.

Panel	Fields	Reference acc.	G acc.	Only the reference API / only G	Reference NLL	G NLL
Cached development	2,600	92.5%	99.8%	—	0.219	0.012
Policy development	1,280	78.3%	99.8%	0 / 276	0.599	0.012
— action	640	65.0%	99.7%	0 / 222	0.939	0.016
Paired counterfactual	1,280	76.7%	99.9%	0 / 297	0.585	0.014
Random-list retention	1,024	89.7%	100%	0 / 105	0.287	0.007
Blind test	1,180	85.8%	98.2%	18 / 164	0.331	0.073
— unseen_v1 (development for G)	216	90.3%	91.7%	16 / 19	0.240	0.301
unseen_v2	432	88.2%	97.9%	9 / 51	0.355	0.053
Order rotations (development for G)	1,008	93.9%	97.0%	30 / 61	0.198	0.114
HellaSwag validation	256	92.6%	82.8%	37 / 12	0.387	0.695
WinoGrande validation	256	92.6%	73.8%	60 / 12	0.379	0.941
MMLU validation	256	88.7%	41.0%	127 / 5	0.823	2.311

Latency p50 / p95 ms, the reference API over HTTP against local: public 102 / 160 vs 20 / 35; policy 118 / 190 vs 44 / 71.

Choice-order and repeat stability (decisions/order_data.py, 96 unseen-proposition cases × 3 rotations of the action choices, 48 identical repeats):

	Reference API	Marker E	Marker G
Identical prompt, winner flips	2.1%	0%	0%
Reordered choices, winner flips	8.3%	3.1%	0%
— low evidence (absent, conflict, shortfall, repeated source)	15.6%	6.3%	0%
— resolved evidence	4.7%	1.6%	0%
Noul shift under reorder	0.035	0.004	0.001
First-listed option wins (unbiased 33.3%)	32.3%	33.0%	33.3%

The reference API's order effect concentrates where the state gives no dominant signal and is not a first-position prior, consistent with the evidence-alignment reading. Local inference is deterministic and training shuffles choice order every sample.

Remaining gaps: MMLU is bounded by what a 395M encoder knows; HellaSwag and WinoGrande trail the reference API by 10 and 19 points after about 1.5 epochs of a 10,000-row sample. Calibration on unseen_v1 is still worse than the reference API's. Public-task probabilities are overconfident (the temperature is fitted on judgment-family calibration splits only).

Why MMLU stays low: two probes

Stage G fits its own MMLU auxiliary training rows (93% on a 300-row sample) but scores 41.0% on validation: humanities and social 49%, STEM 39%, law and professional 29%, moral scenarios 24%. The format and optimisation work; the gap is generalisation. The auxiliary split is reading comprehension and grade-school science, not the recall MMLU tests.

Data probe: stage H continues G for 3,000 updates on closed-book questions (decisions/knowledge_train_data.py: MedMCQA, SciQ, CommonsenseQA, plus auxiliary rows of at most 400 characters; validation questions excluded). Headroom probe: experiments/probe_zero_shot.py scores untrained Qwen2.5 Instruct backbones on the same 256 validation rows per task.

Model	MMLU	HellaSwag	WinoGrande
Marker G, 395M, fine-tuned	41.0%	82.8%	73.8%
Marker H, knowledge data	42.6%	76.6%	78.1%
Qwen2.5 0.5B, zero-shot	39.8%	36.7%	48.4%
Qwen2.5 1.5B, zero-shot	56.6%	59.4%	53.5%
Qwen2.5 7B, zero-shot	68.4%	77.3%	68.4%
the reference API (v1.13.0)	88.7%	92.6%	92.6%

Better-matched data moves MMLU by 1.6 points, inside the ±3 point noise of 256 rows; backbone size moves it by 17 and 29 points with no training. MMLU is bounded by knowledge stored at pretraining. HellaSwag and WinoGrande are skills that fine-tuning teaches: the 395M encoder beats the untrained 7B model on both. Stage H is a probe, not a promoted checkpoint; its judgment and policy panels were not run.

Dense causal backbone: Qwen2.5-1.5B with LoRA

experiments/initialize_causal.py wraps Qwen2.5-1.5B-Instruct in the packed-prefix candidate scorer and joint head. decisions/lora.py is a dependency-free LoRA (rank 16 on every linear layer, merged into the weights on save; 6 GB while training). Gradients are healthy under SDPA on this path: the truth-table stage reaches 100% by update 120. One mixed stage follows: 3,000 updates of 16 requests, learning rate 0.0001, stage G's data plus the knowledge sets, label smoothing 0.1, constraint dropout 0.5; temperature 0.341. Same paired protocol as above.

Panel	Fields	Reference API	Marker G (395M)	Qwen 1.5B LoRA
Cached development	2,600	92.5%	99.8%	99.8%
Policy development	1,280	78.3%	99.8%	97.0%
Paired counterfactual	1,280	76.7%	99.9%	98.0%
Random-list retention	1,024	89.7%	100%	100%
Blind test	1,180	85.8%	98.2%	98.3%
— unseen_v1	216	90.3%	91.7%	98.1%
unseen_v2	432	88.2%	97.9%	94.0%
Order rotations	1,008	93.9%	97.0%	98.1%
HellaSwag	256	92.6%	82.8%	81.2%
WinoGrande	256	92.6%	73.8%	70.7%
MMLU	256	88.7%	41.0%	53.1%
Latency p50 / p95 ms, policy		118 / 190	44 / 71	82 / 100

Order and repeat winner flips are 0% for both local models. The larger backbone buys 12 MMLU points and better unseen_v1 transfer, at twice the latency, and is slightly behind the marker encoder on policy panels and unseen_v2 after a fifth of the marker's training updates. Its fine-tuned MMLU (53.1%) is below its own zero-shot letter-token score (56.6%): scoring each option as a separate "does this candidate answer" path is a worse fit for a causal model than reading the letter directly, and LoRA on about 10,000 short MMLU-style rows adds no knowledge.

Routing letter-choice questions to a zero-shot 7B model

decisions/router.py sends a request to a knowledge model only when it is one unconstrained choice over option letters; everything else stays on the marker encoder. The knowledge model is untrained Qwen2.5-7B-Instruct in bf16 reading the letter from its language-model head (decisions.benchmark --knowledge-model). Both fit in 24 GB. Paired with the reference API on the 768 public validation rows:

	Reference API	Marker G alone	Routed, Qwen 1.5B	Routed, Qwen 7B
MMLU	88.7%	41.0%	56.6%	68.4%
HellaSwag	92.6%	82.8%	59.4%	77.3%
WinoGrande	92.6%	73.8%	53.5%	68.4%
NLL, all 768	0.530	1.316	2.021	2.833
Latency p50 / p95 ms	102 / 160	20 / 35	42 / 56	87 / 136

The blind panel routes nothing to the knowledge model and is unchanged (98.2%, 40 / 63 ms). On policy-length prompts (563 input tokens per request) the 7B model alone takes 162 / 250 ms against the reference API's 118 / 190 ms over HTTP, and is untrained there (59.2% field accuracy).

Routing by format lifts MMLU by 27 points but loses HellaSwag and WinoGrande, where the fine-tuned 395M encoder beats the untrained 7B model; the three tasks share one request shape, so format cannot separate them. Zero-shot probabilities are badly overconfident (ECE 0.26). The remaining levers are fine-tuning the 7B letter readout with LoRA on the HellaSwag and WinoGrande train splits and fitting a temperature for that route.

Out-of-family tasks and stage I

decisions/families_data.py is a test-only benchmark of four task families outside every training schema (invoice matching, incident triage, refunds, access requests): new field names, numeric thresholds, percent tolerances, rule precedence, no typed constraints. 400 requests, 900 fields, paired with 400 reference API calls.

Stage I continues G for 12,000 updates with decisions/schema_data.py (20,000 train-only cases, each a random task: 12 domains disjoint from the held-out families, random fields, thresholds, percent tolerances, code lists, 2–5 outcomes, rule order, three rule phrasings, four layouts; labels from executing the rules) and the full HellaSwag and WinoGrande train splits (about 39,000 rows each, 500 more per task held out as calibration rows). Temperature 0.584.

Panel	Fields	Reference API	Marker G	Marker I
Out-of-family, all	900	94.1%	50.2%	66.6%
Out-of-family NLL		0.149	1.675	1.472
HellaSwag	256	92.6%	82.8%	86.7%
WinoGrande	256	92.6%	73.8%	80.5%
MMLU (not trained; routed elsewhere)	256	88.7%	41.0%	41.4%
Cached development	2,600	92.5%	99.8%	99.8%
Policy development	1,280	78.3%	99.8%	99.8%
Blind test	1,180	85.8%	98.2%	98.6%
unseen_v2	432	88.2%	97.9%	96.3%
Order rotations	1,008	93.9%	97.0%	97.8%

On 400 fresh tasks from its own generator stage I scores 96.5% on the outcome, 96.9% on condition checks and 90.9% on tallies, including thresholds (91–98%) and percent tolerances (90%). Rule execution is learnable at this size; the out-of-family gap is generator breadth. The weakest held-out fields need primitives the generator lacks: a question derived from the outcome (page_oncall, 50%), exact text equality between two fields plus a tolerance (disposition, 34%), and counting factors named in the rules (risk, 37%). The benchmark was written before the generator, so primitive choice is informed by it; vocabulary and domains are disjoint (tested).

Stage J and the limit of hand-written task templates

decisions/families2_data.py is a second out-of-family holdout (expense claims, service-credit tiers, age ratings, pharmacy refills; 675 fields, 300 reference API calls), written before the generator changed and not used to tune it. schema-v2 then added the primitives stage I lacked (consequences derived from the outcome, same-person text matches, factor lists defined in the rules), two rule phrasings, a sentence layout and six domains; stage J continues I for 6,000 updates on 30,000 such cases. Stage J is not promoted.

Panel (field accuracy)	Reference API	Marker I	Marker J	Qwen 7B zero-shot	Qwen 1.5B zero-shot
families_v2 (clean holdout)	96.1%	63.9%	63.6%	63.8%	36.2%
families_v1 (informed the generator)	94.1%	66.6%	71.9%	52.0%	37.8%
Cached development	92.5%	99.8%	99.1%	—	—
Blind test	85.8%	98.6%	97.5%	—	—
HellaSwag / WinoGrande	92.6 / 92.6%	86.7 / 80.5%	83.6 / 77.3%	77.3 / 68.4%	—

Widening hand-written templates lifts only the holdout that informed them: the model learns the generator, not rule-reading in general. An untrained 7B model is no better on unseen families than the trained 395M encoder, so the reference API's 94–96% is not explained by size alone at the scale Atlas can run; it implies training on a far broader task distribution. The scalable lever is task diversity from a generator that does not share an author with the holdouts, for example LLM-written task families with executable oracles through the existing decisions.llm_synth admission pipeline.

Task-family synthesis with Bend oracles (synth/)

synth/ is a Rust pipeline that asks a writer model for whole task families and labels every case by running model-written Bend 2 code. A family is a JSON spec: fields (flags, categories, numbers with sampling boundaries; numbers may be rules-text parameters so thresholds vary per case), outcomes, questions (one outcome choice plus noul and score questions), prose rules with placeholders, 2-4 state layouts, and bend_defs. The harness wraps the defs in a State record, argument parsing and printing, plus a helper library (if_str, if_u32, abs_diff, within_pct, count3) so the model seldom needs Bend's restricted match. Bend's checker gives purity, termination and no IO for free.

Per family: type-check and native compile (0.7 s), 600 sampled states for admission (every outcome reachable at >=3%, no constant question, every field changes some answer, deterministic), up to two repair calls that feed the checker's message back, a reviewer call that applies each case's own filled rules by hand to 12 cases (agreement is recorded; rendering skips families under 0.75), and a held-out-phrase filter (holdout_vocabulary.txt) so no synthesized family reuses a benchmark family's domain. Every provider call is journaled before it is sent with a worst-case cost reservation; nothing retries.

Pilots with GLM 5.3 Flash: the helper library lifted acceptance from 6/16 to 14/19; with per-case reviewer rules, 8/12 accepted and agreement 0.83-1.0. Cost is about $0.002 per accepted family. render draws balanced cases per family and holds out 5% of families by hash as a rotating out-of-family test set.

Stage K: training on synthesized families

The first synthesis run produced 1,870 families (69% of attempts, $5.13); 1,705 passed the 0.75 reviewer-agreement gate and rendered to 68,194 rows, with 96 families (3,840 rows) held out by hash. Stage K continues I for 12,000 updates with those rows at about 40% of samples. Paired with the reference API as before. Evidence.

Panel (field accuracy)	Reference API	Marker I	Marker K
families_v2 (clean holdout)	96.1%	63.9%	80.4%
families_v1	94.1%	66.6%	89.3%
Synthesized held-out families (256 requests)	—	—	93.4%
unseen_v2	88.2%	96.3%	98.1%
Cached development	92.5%	99.8%	99.6%
Blind test	85.8%	98.6%	97.0%
Order rotations	93.9%	97.8%	82.9%
HellaSwag / WinoGrande	92.6 / 92.6%	86.7 / 80.5%	84.4 / 77.7%

Model-designed families move the clean out-of-family holdout by 16 points where hand-written templates moved it by nothing. The remaining families_v2 errors are concentrated in two fields whose rules derive a yes/no answer from the outcome (money_paid_now 53%, concerns 55%). The order-rotation drop is not instability (winner flips stay at 3%) but low-evidence judgment cases falling to 48%: the "unresolved means review" rule was crowded out by the synthetic weighting. Stage L continues K with judgment data reweighted up and synthetic rows down.

Stage L (6,000 updates from K, judgment files at twice the weight, wording_v3 at 4, synthetic rows at about half K's share) recovers the order panel (82.9% to 90.0%, low-evidence action 48% to 79%) and the blind test (97.0% to 97.6%), but gives back out-of-family accuracy: families_v2 80.4% to 71.6%, families_v1 89.3% to 88.2%, synthesized holdout unchanged at 93.5%. Reweighting trades one capability for the other at this data volume; K stays the reference checkpoint, and the lever for both is more synthesized families with judgment replay kept.

Throughput

The pipeline is bound by provider latency, not local work: a call takes about 22 s (800-1,600 completion tokens from GLM Flash) and a family needs about 2.4 sequential calls, against under 2 s of Bend checking, compiling and admission. synth run is therefore one process with --in-flight worker threads (default 64, at most 256) sharing one budget ledger that locks only for bookkeeping, never across an HTTP call. Throughput scaled linearly up to 64 concurrent requests with no rate limiting from the provider: 8 in flight gave 8 attempts per minute, 64 gave about 80. Reviewer replies were cut from 810 to 590 completion tokens by asking for compact one-line JSON and 8 cases instead of 12; the writer's 1,500-token reply is the family itself and cannot shrink much. The admission repair loop, boundary-biased resampling, literal-derived boundaries and the narrower vocabulary filter raised acceptance from 69% to 92% of attempts.

Second seed and second synthesis run

The K recipe with seed 1997 reproduces seed 97 on every panel within noise except the small clean holdout: families_v2 73.8% against 80.4% (675 fields), families_v1 89.9% against 89.3%, synthesized holdout 94.0% against 93.4%, order 83.6% against 82.9%, blind 97.1% against 97.0%, policy development 100% against 99.5%. Treat families_v2 as accurate to about plus or minus 4 points per seed.

The second synthesis run (64 in flight, yield fixes) accepted 3,950 families from about 4,300 attempts (92%) for $11.17, mean reviewer agreement 0.94, 86 families under the 0.75 gate.

Stage M: both synthesized datasets

Stage M continues K for 8,000 updates over 176,000 synthesized rows (1,705 + 3,864 families) with judgment files at twice K's weight. Paired with the reference API as before.

Panel (field accuracy)	Reference API	Marker K	Marker M
families_v2 (clean holdout)	96.1%	80.4%	79.7%
families_v1	94.1%	89.3%	90.8%
Synthesized holdouts (run 1 / run 3)	—	93.4% / —	94.0% / 93.2%
unseen_v2	88.2%	98.1%	98.4%
Blind test	85.8%	97.0%	96.4%
Order panel accuracy	93.9%	82.9%	80.1%
HellaSwag / WinoGrande	92.6 / 92.6%	84.4 / 77.7%	84.4 / 80.5%

Doubling the synthesized corpus did not move the clean holdout: the gain is in specific fields (money_paid_now 53% to 89%, access:risk 73% to 80%) while three primitives stay weak wherever they appear: counting factors that the rules define (rating:concerns 53%), differences between two numbers in the state compared against a step (uptime:credit 63%, invoice:amount_within_tolerance 63%), and a category label that changes which threshold applies (rating:label 69%). Breadth is no longer the lever; those primitives are underrepresented in what the writer model produces unprompted, so the next batch requests them.

experiments/run_marker_encoder.sh