the counter that replaced attention

A language model whose attention is a bank of decaying event counters — no token pairs, no softmax, no growing cache — derived from the same objective that derives the transformer it replaced. It slightly outperforms its softmax parent, solves induction perfectly at every tested distance, and its state machinery is 60 gates per coordinate, placed and routed. Working name: CRSA — Causal Rate-Statistics Attention.

the operator, in four verbs

project   h = U z                      the CRATE subspace projection, unchanged
count     c ← ρc + h²                  decaying activity per coordinate; per-head
                                       half-lives 5 / 11 / 22 / 44 tokens
price     d = f′(c)                    marginal coding cost: 1/(1+c), or two
                                       comparator bits in the hardware staircase
gate      out = −η·Uᵀ(d ⊙ h)           suppress what recent context made cheap;
                                       pass what is still informative

Where softmax attention asks "which past tokens resemble this one?" — requiring every pairwise comparison and a cache of the entire past — CRSA asks "which directions has recent context already made cheap to encode?", a question a fixed bank of counters answers. The derivation follows the white-box program (CRATE → ToST): attention as a step on a coding-rate objective, with the honesty note carried in full in the repo — the step is a prox-linear frozen-code surrogate, verified empirically, not a proven descent.

what it did as a language model

The counter model beats the softmax model it warm-started from by 3.1%, holding 384 counters per layer where attention keeps a cache that grows with every token (~98k values per layer at this context — a ~256× state reduction, unbounded with length). A 13.8M scratch-trained scale-up against the RWKV reference is running as this page is written; its expectations are logged before its results, as this project does.

the mechanism finding under it

Every softmax variant we trained was repurposed by training: under aligned measurement, attention learned to expand the coding-rate term it was derived to compress (while helping the broader objective — the "derivation–execution gap"). The counter operator is the first whose trained dynamics match its derived sign, descent-form monotonically across all of training. Three measured points: softmax — good quality, repurposed; reordered softmax — better quality, more strongly repurposed; counters — best quality, faithful. The gap is an operator-dependent variable, not a law.

ruthless probes, preregistered

Locked before the operator existed: copy, associative recall, induction, selective retention, at delays 0.25–4× the counters' half-lives, against matched-parameter baselines (two seeds; full table and prediction scorecard in the repo):

A constant-state operator with no token pairs solves induction perfectly at every tested distance — including 89 tokens, twice the slowest counter's half-life. Retention does not decay with the horizon ladder: the multiscale population retains what no single counter could. And the operator's cost concentrates exactly where its mechanism predicts: interference-prone retrieval under distractors — it forgets by collision, not by time. (One baseline — a decayed-value recurrence — floored at chance both seeds and is declared invalid pending an input-rescaling fix; no comparative claims against it.)

and as a circuit

The deployable form (event-bit counters, comparator staircase) is a compiled Morpho circuit — examples/crsa/crsa_unit.py — verified bit-exact on 100,000+ event streams at every horizon:

Compare the wkv cell (7,201 gates per channel): its area went to exponentials and a restoring divider — softmax-lineage machinery. CRSA's attention has no exponential, no divider, no softmax; decay is a shift, prices are comparators. A full 672-counter layer extrapolates to ~40k gates of state machinery at over 100 MHz. One design decision made visible: integer dyadic decay is sticky below 2^m, so each counter carries six fraction bits — a bug you meet only when the mathematics becomes registers.

a reversal, published rather than patched

Subsequent controls reframed this page's headline. The 13.78-vs-14.22 comparison ran in blocks with no real elementwise nonlinearity — a feature-starved regime. With a transformer-grade MLP in both models, softmax attention wins decisively (12.54 vs 16.79): constant-statistics attention trades retrieval capacity for state efficiency, exactly where the probes located the cost (selective retention: 0.56 vs 0.91). What survives strengthened: perfect induction at constant state, the dyadic ladder worth +7.8 ppl over uniform statistics, derivation fidelity, and the 60-gate hardware. And the best constant-state model so far is CRSA plus an overcomplete sparse dictionary: 8.40 at 14M over a matched 20k horizon, beating plain CRSA's 9.72. Full campaign: the white-box lab.

honest edges

One corpus (TinyStories), one scale (5.2M), one seed for the LM result so far — the robust claim until replicated is matches softmax, not beats it. The trained float model is not a circuit for free: fixed-point conversion of the full LM is the semantic-alignment axis our M2 experiments left unresolved (activity calibration is solved; meaning preservation is not). The name is provisional — "TSSA" was taken — and the prior-art boundary is drawn precisely: RetNet shares the dyadic-decay implementation family; what appears distinct is the combination of derived subspace statistics, the rate-derived price, no attention matrix, K·p state, and the measured derivation-fidelity result. The full qualifications, prediction scorecards, and negative results live in the repo — this page is the view from above, not the evidence.

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.