the model that couldn't remember who owned what

Three days of preregistered experiments on a question most language-model benchmarks never ask: can a network whose memory never grows remember that Tom has the red ball and Tim has the blue one? The answer rebuilt our architecture — and then the data delivered a twist about why nobody had noticed the deficit.

← the white-box lab ← attention for silicon ← the counter that replaced attention

the symptom

Our best constant-state language model wrote fluent children's stories with one recurring flaw: characters blurred. Billy is asked a question; Sarah answers it. Tom starts a story; Ben ends it. We built an instrument to measure the failure precisely: pairs of prompts that are identical except for which color belongs to which name — "Tom has the red ball, Tim has the blue ball" versus its swap — then ask what Tom picks up. A model that binds tracks the swap. A model that doesn't sees two nearly identical token bags.

the deficit, measured at two scales

Every tied-projection mechanism scored at chance on arbitrary binding — while holding perfect scores on sequence memory.

Counters, uniform statistics, windowed and full attention with shared projections: all at stored-set chance on the probe grid, all at 1.00 on induction. And the trained language models were worse than at chance — they were faking it: our best model showed strong swap sensitivity that reversed sign with the queried entity's position. A "prefer the first-mentioned color" rule imitating binding on half the cells, caught only because the instrument counterbalanced query position.

finding one — binding lives in the query

At equal parameter count, we split attention's shared projection one role at a time. Separating the value path: binding 0.44. Separating the query path: 1.00. The bottleneck was never what to return — it's matching a retrieval context ("Later, Tom picks up his…") to a storage context ("Tom has the…"). One matrix can't speak both dialects. Better: the query correction is low-rank — a rank-16 adapter keeps 97% of binding, so the repair costs ~1% of the model, not a rebuild.

finding two — eight drawers

A constant-state memory of M content-addressable slots binds — once writes are routed by the owner's key.

Our first slot design filed "red" under red's own address, so ownership was structurally unrepresentable — it scored at chance with every basis we tried, and a measured gradient tug-of-war (cos −0.46) ruled out the easy fixes. Routing each write by the preceding token's key — so the attribute lands in the drawer its owner opened — took binding from chance to 0.94 in one change: induction 1.00, selective recall 1.00, flat across delays, robust to distractors. And the capacity law is legible: binding capacity scales with slot count — eight slots are near-ceiling through eight active facts and degrade under sixteen; sixteen slots solve all sixteen (0.97). Soft slots can superpose beyond their nominal capacity (a grouped eight-slot variant reached 0.75 at sixteen facts), so the limit is gradual, not literally one fact per drawer.

The oracle rule "owner = previous token" then fell to a learned replacement: a four-tap causal convolution (1,792 parameters, a shift register in hardware) learns to blend the owner's identity into the attribute's representation before the write — binding 0.74 with nothing hard-wired. On the adjacent-owner grammar, pair formation could be learned by a local convolution while pair retention stayed global; generalization to displaced owners and unseen syntax remains an open gate.

finding three — separation mattered, connectivity didn't

Statistical and associative memory could not share a representation basis — but the independent basis did not need to be dense.

Matched owner-routed binding scored 0.201 with a basis shared with the counters versus 0.941 with an independent one, and both memory tasks degraded under sharing (a measured gradient conflict, cos −0.46 at layer 0). Yet a four-group block-diagonal basis reached or exceeded dense-basis binding at a quarter of its parameters (602k vs 2.41M at model scale). The two memories need separate spaces; within its own space, associative memory tolerates — even prefers — restricted connectivity.

the language model

model	params	steps	ppl	inference state
untied QKV + MLP transformer	19.4M	10k	6.04	KV cache — grows linearly with retained context
counters + dictionary + slots	17.1M	20k	6.48	~194 KB FP32, constant
dictionary CRSA (previous best)	14.0M	~28.5k	8.08	constant

The rebuilt memory system cut the family's best perplexity by 1.6 (similar — not matched — parameters and schedules; a matched-schedule reference and a parameter-matched no-slot control are both owed, so the 1.6 is not yet causally attributable to slots alone). The state accounting is one equation: state = L·[d + M(d+1)] — at L=12, d=448, M=8 that is 48,480 values, ~194 KB FP32, versus a K/V cache of ~10.5 MiB at context 256 and ~42 MiB at 1,024. Bounded recurrent state does not itself establish unbounded useful context: the model trained at finite context, and slot retention was probed through delay 96. On hardware: every sequence operator has a bounded realization — the counter datapath is synthesized (60 gates/coordinate); the fixed-size slot softmaxes await calibrated winner-take-all hardening and synthesis. The slot pathway also trains ~3× slower per step today — an engineering deficit (unfused scan), not an architectural one.

the twist

The counterfactual suite found no behavioral binding in the 6.48 checkpoint: candidate-restricted accuracy at stored-set chance, swap sensitivity reversing with mention order — the signature of a primacy heuristic, not ownership. This does not mean the slot pathway was unused (it may serve selective context or other functions), or that binding is latent in these particular weights; it means the architecture possesses binding capacity under explicit supervision, while TinyStories optimization did not express it. And the same suite run on the 6.04 transformer found the mirror image: accuracy at chance with the sign pattern of a recency heuristic (swap sensitivity up to ±0.6, following the latest mention). Neither architecture family binds on this corpus — each substitutes its own position prior, and aggregate perplexity improves to at least 6.04 while counterfactually verified ownership remains absent. That is why this deficit hides under perplexity benchmarks, and why the next test belongs to a task that pays for memory — long-form consistency, question answering, or music, where a voice that forgets which hand owns it is not a rounding error but the whole failure.

what carried the work

Every mechanism entered through a preregistered probe with the verdict rules written before the run: gates, outcome maps, controls that caught two masquerades (a "ceiling" that couldn't bind, a grammar that leaked to a shortcut). Failures were kept and diagnosed — three mechanisms died the same cold-start death and taught a design rule (circuits must be reachable from initialization by gradient); a composition of two individually-proven parts — the same-layer convolution-plus-grouped-basis stack, the very combination we had shipped — failed and was caught by a thirty-minute probe rather than a deployed model. Twenty-nine probe arms across thirty-plus runs and six language models, one seed each except where replicated: the strongest result (query-map localization) replicated exactly at 1.00, while the noisiest (slot-circuit formation) is reported as a range, not a headline. Counterfactual cells use 100 counterbalanced prompts each (binomial 95% CI ≈ ±0.10 at chance); "chance" is stored-set chance — 1/N in the number of stored candidates — so every cell is judged against its own N.

Numbers: one controlled seed unless stated (slot binding replicated at 0.94/0.71/0.44 across three seeds — formation is stochastic, the open engineering problem alongside kernel fusion and WTA hardening for silicon). Full preregistrations, negative results, and the paper draft live in the whitebox-lm repo.

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.