attention for silicon

Why this program is trying to derive a new attention operator with a chip in mind — the rationale, the measured costs on both sides of the trade, and what would count as failure.

the counter that replaced attention → the white-box lab → morpho → silicon →

Transformer attention is shaped by the machine it grew up on. Softmax attention is what you invent when matmuls are nearly free, memory is vast, and everything runs in large parallel batches — a GPU. We are asking the inverted question: what does attention look like when it is designed under silicon's constraints first — fixed memory, no transcendental functions, energy per operation that you can count in gates — and how much language-modeling quality does that inversion actually cost? Both halves are measured, not assumed.

what softmax attention costs a chip

Three separate bills come due when you put standard attention on an FPGA or ASIC, and they compound:

1. The operators are expensive. Softmax needs an exponential and a division per score. When we compiled RWKV's wkv kernel — already a simplification of attention — into gates, the exponentials and dividers dominated: 7,201 gates for one channel's kernel. These functions don't decompose into cheap logic; they become lookup tables, iterative units, or precision compromises.

2. The state grows without bound. Attention re-reads every past token's key and value at every step. The KV cache grows linearly with context, so a chip must either budget worst-case memory (and the bandwidth to stream it every token) or truncate context. On edge silicon, memory traffic — not arithmetic — is where the energy goes.

3. The work per token grows too. Each new token attends over all previous ones: cost per token scales with context length. A fixed-clock device cannot promise a fixed token rate.

what a counter costs

CRSA replaces pairwise attention with per-channel event statistics: each coordinate keeps a leaky counter of recent squared activity, decayed on a dyadic ladder (half-lives ~5, 11, 22, 44 tokens), and prices its own output by 1/(1+c) — rare features speak louder. Every piece was chosen to be countable in gates, and then counted:

quantity	measured
one counter coordinate (verified bit-exact)	58–61 gates + 14 registers
decay	a bit-shift — dyadic ρ needs no multiplier
the price 1/(1+c)	2-comparator staircase — no divider
16-wide bank, placed & routed (iCE40)	690 LCs @ 103.7 MHz
updating a full layer's 448 counters	28 cycles ≈ 270 ns
state per layer, forever	448 numbers — independent of context length

No exponential, no divider, no softmax anywhere in the datapath. The counters are effectively free next to the matmuls — which is precisely the situation you want: the remaining cost is the part every architecture shares.

the inference asymmetry

The same design choice shows up on ordinary hardware as an asymptotic difference. A softmax model generating token 10,000 must consult 10,000 cached keys; CRSA generating token 10,000 does exactly what it did for token 10: update its counters (shift, add) and apply fixed matmuls. Constant time per token, constant memory, no cache eviction policy, no memory wall growing with the conversation. Measured, not claimed: the naive sampler (recompute the prefix every token) manages 9 tok/s on CPU; the recurrent stepper — carrying only the counters — runs the same model at 95 tok/s on the same CPU, verified equivalent to the parallel forward to 2.4×10⁻⁵ over 256 tokens. A 10× speedup from an afternoon's transcription, and the rate never degrades with context length. An in-browser demo on this path is queued work; on silicon the story is already measured above.

why pay for this at all

Because the places language models are not are the places with these constraints. A $10-class FPGA, a sensor node, a hearing aid, an always-on assistant that cannot ship audio to a datacenter — all need bounded memory, bounded energy, and a fixed token rate. A model whose attention is 60 gates per coordinate and whose state never grows can live there; a model that needs a KV cache and an exponential unit cannot. And the counter operator is spike-compatible by construction — events, thresholds, and integer state — which keeps the door open to event-driven silicon where energy scales with activity rather than with clock.

There is also a scientific reason, independent of deployment: CRSA is derived — from a compression objective over exponentially weighted token statistics — rather than found by search. Derived operators are auditable: we can state what each part is for, test whether training preserves that role, and count what it costs. In our measurements it is the only operator whose trained behavior still matches its derivation (softmax attention gets repurposed by training — the derivation–execution gap). For hardware, auditability is not aesthetics: it is how you justify precision choices and prove the fixed-point circuit computes the same thing the model trained.

the price, measured honestly

This is a trade, not a free lunch, and we have measured both sides. With identical feature blocks, softmax attention beats CRSA by 4.25 ppl (12.54 vs 16.79) — full attention retains a substantial retrieval advantage, and it shows up in generated text as character-name drift over long stories. But the ablations localize what the counters' forgetting is worth: dyadic decay improves 0.381 nats/token over uniform all-history statistics and 0.415 over a numerically verified reproduction of published causal TSSA — the multiscale forgetting, not routing machinery, is the load-bearing idea. Our best constant-state model (CRSA + overcomplete dictionary, 14M) reaches 8.40 ppl on TinyStories and writes coherent short stories. The open question this program exists to answer: how close can bounded-state, gate-countable attention get to the retrieval class — and whether a small representative memory (far short of a full cache) closes the gap the probes located.

what would count as failure

The rationale is falsifiable, and parts of it already bit back once: an early "counters beat softmax" headline dissolved when both operators got a real feature block, and we published the reversal rather than patching it. The path stops being worth chasing if the retrieval gap proves unclosable at any acceptable state budget, or if the hardware advantage evaporates once full models (not just the attention operator) are costed on silicon. Both are measurable, both are on the roadmap, and the numbers will be reported either way — that is the program.

Every quality number is one controlled seed on TinyStories at 5–26M params; gate counts are from verified Morpho circuits synthesized and placed via yosys/nextpnr. Sources and preregistrations live in the whitebox-lm repo.

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.