the white-box lab

Three days inside a research campaign: language models derived from objectives, held to preregistered probes, decomposed by factorial ablation, and corrected in public when the data said so. The through-line is a discipline — formalize before training, lock predictions before results, name every confound — and the results below include two genuine findings, one honest reversal, and a measured trade-frontier where a single winner was hoped for.

← who owned what: the M4 story ← the counter that replaced attention ← why attention for silicon ← to silicon whitebox-lm repo

the arc in one table

model (5.2M unless noted)	ppl @5,500	note
causal CRATE, faithful wiring (M0)	15.37	the honest baseline
+ reordered wiring (M2-control)	14.22	the reorder is an improvement
+ counters instead of attention (CRSA)	13.78	beats its parent — see the reversal below
spike-consuming variants (M2 ladder)	36.8–2,472	calibration solves extinction, not meaning — closed by preregistered rule

matched 20k-step pair (one seed)	ppl @20k
CRSA + overcomplete dictionary (14.0M)	8.40
plain CRSA d=672 (13.8M)	9.72

A reduced-LR extension of the dictionary model reached 8.08; the M4 binding program then rebuilt the memory system and produced the campaign's final model: slots-lm-v2 at 6.48 (17.1M — counters + dictionary + owner-routed slots with a learned convolutional write path), versus 6.04 for a true untied-QKV transformer at matched parameters — the constant-state cost measured at last against the real ceiling, ~194 KB of recurrent state versus a KV cache that grows forever. The twist the counterfactual suite caught: the best models' gains owe nothing to binding — TinyStories rewards fluency and position priors, never ownership. The binding capability exists (measured at 0.94 on preregistered probes, capacity proportional to slot count), sits latent, and awaits a task that pays for it.

finding one — the derivation–execution gap

Deriving a layer from an objective does not guarantee training leaves it executing that objective — and whether it does depends on the operator.

Under aligned measurement, every softmax-attention variant was repurposed by training: attention learned to expand the coding-rate term it was derived to compress (12/12 layers, both wirings, confirmed by directional derivatives and α-sweeps — genuine ascent, not overshoot). The counter operator is the only one whose trained dynamics match its derived sign, monotonically, across all of training. White-box derivation supplies a falsifiable mechanistic hypothesis, not a guarantee.

finding two — the factorial

fixed d=448, 3,000 steps	identity prox	active prox
dictionary q = d (6.8M)	22.32	20.60
dictionary q = 4d (14.0M)	22.40	19.75

Overcompleteness has no value as a linear factorization; it becomes useful only when the proximal threshold partitions the input space into selectively activated feature regions.

In cross-entropy: the prox alone is worth −0.080 nats at zero added parameters; doubling the dictionary with the prox disabled is worth +0.004 (nothing, as the rank argument demands); the interaction is −0.046 nats — width pays only through the nonlinearity. At the 20k horizon this compounds: the dictionary model's early disadvantage inverted into a 1.32 ppl win — slower to organize, lower floor.

the reversal — published, not patched

The counter operator's headline ("beats its softmax parent, 13.78 vs 14.22") was true and is now reinterpreted: that comparison ran in blocks with no real elementwise nonlinearity — a feature-starved regime where neither operator could exploit retrieval. Give both a transformer-grade MLP and softmax wins decisively (12.54 vs 16.79 at 23.7M): constant-statistics attention trades retrieval capacity for state efficiency. The preregistered probes had already located the cost — selective retention under distractors (0.56 vs 0.91) — before the perplexity result surfaced it. The methodology predicted its own headline's demotion, which is the strongest thing we can say for the methodology.

what the counters still own

claim	evidence
perfect induction, all distances, constant state	1.00 at every delay incl. 89 tok — tied with full attention, from 384 counters/layer
the dyadic ladder is load-bearing	−0.381 nats/token over uniform prefix statistics, −0.415 over a numerically verified literal-TSSA reproduction (which lands at 25.43, the same performance class as the uniform arm despite +2.4M params) — the isolated variable is multiscale exponential temporal weighting, not routing or parameter count
derivation fidelity under training	the only operator whose ΔR^c sign survives (finding one)
hardware	60 gates + 14 registers per counter coordinate; a 16-wide bank placed & routed at 690 LCs, 103.7 MHz; no exponential, no divider, no softmax anywhere

And the failure mode is principled: the operator forgets by collision, not by time — which names its repair (a small representative memory between counters and a full cache) without undermining what works.

The close, stated as a trade frontier: on a one-seed, 3,000-step TinyStories screen with identical MLP feature blocks, softmax led at 12.54 PPL, dyadic-leaky CRSA reached 16.79, and both all-history statistics operators — the uniform ablation and a numerically verified reproduction of causal TSSA — trailed at 24.57–25.43. CRSA therefore recovers substantial capability over its direct cumulative-statistics ancestor while preserving bounded recurrent state, but does not eliminate the retrieval advantage of full attention. The staleness mechanism behind the forgetting effect remains a hypothesis (testable by distribution-change adaptation or a context-length sweep); the controlled result — exponential temporal weighting as the load-bearing modification — is measured.

methods, because they carried the campaign

Probes locked with seeds and predictions before the operator existed (six predictions: four confirmed, one refuted in the strong direction, one untestable due to an invalid baseline — declared, not hidden). Screens scoped as trajectory reads, never verdicts — vindicated when both 20k runs finished 57% below their screen numbers. Factorial decomposition in nats. Claim freezes with named qualifications: one seed, matched budgets, parameter caveats. Corrections published as findings — the misaligned metric that once said "attention compresses as derived," the naming collision with prior art, the feature-starved reinterpretation above. The direct ancestor (Token Statistics Transformer) is cited as ancestor, with CRSA's claim narrowed to what it adds: multiscale exponential event statistics, bounded shift-update state, spike compatibility, and a gate-counted implementation.

what 8.08 sounds like

Perplexity is only a number; here is the best constant-state model (CRSA + overcomplete dictionary, 14.0M, one seed, 8.08 censored) actually writing — unedited samples from fixed seeds, generated by the recurrent sampler.

Tom and his mom went to the park. They played in the park, and had a picnic. They liked to feed the ducks. The sun went down and had a lot of fun.
After a while, it was time to go home. The sun went down and the little boy said goodbye to his mom. He sat down and went home. He felt happy and content. <|endoftext|>

A complete story arc with a proper ending and no name drift — the first drift-free full story the model has produced. Grammar, pacing, and closure are all in place.

Once upon a time, there was a little girl named Lily. She loved to play outside. One day, she went outside to play in the park with her friends. She saw something unusual. It was a big, red ball. Lily was very excited to play with it.
When she got to the park, she saw her friend Billy and asked, "Do you want to play too?" Sarah replied, "Yes, let's play golf."

And the surviving failure, marked in red: Billy is asked, Sarah answers. This is the probe suite's selective-retention deficit rendered as prose — the counters forget by collision, so entities blur into one another. Binding which entity owns which words is exactly what the M4 experiments target.

honest edges

Every number above is one controlled seed on one corpus (TinyStories) at small scale. The 20k pair had not plateaued when schedules ended — true floors are lower. Replication seeds, a two-size scaling law, and the spike/forward-only/hardware rungs of the ladder remain open work. The repo carries the full preregistrations, outcome maps, and negative results.

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.