rwkv atlas

The whole model, not just the cell: a 6-block RWKV-v4 as one navigable machine — where the state lives, how far back each layer can remember, what the full circuit would cost, and which numbers on this page are measured versus derived. Companion to the wkv cell.

← the wkv cell ← found machines run the whole model live → to silicon → code

the stack — click a block

Six residual blocks, each a time-mix (the wkv attention) followed by a channel-mix (the feed-forward). All recurrent state in the entire model is per-channel and local: three wkv registers plus one token-shift register per sublayer. Click any block to unfold its circuit.

memory horizons — how far back each block sees

Each channel's decay w₂ sets a memory half-life of ≈ 1/|w₂| tokens. The raster shows all 384 channels per block, sorted, colored by horizon; the curves compare each block's median channel forgetting a token planted Δt steps ago. Trained facts: w₂ spans −26.5 … −0.0098; medians run from ≈ −1.2 (block 0) to ≈ −0.09 (block 5) — deeper blocks hold memory longer; exactly one channel in 2,304 remembers past 100 tokens.

Exact — every pixel is a real trained channel: all 2,304 w₂ values from the model export (spikelm/export/wkv-atlas.json) are embedded in this page, sorted per block.

token planted at Δt =

the whole model as hardware

subsystem	per block	whole model	notes
wkv state	3 × 384 registers	6,912 registers	aa, bb, pp — the model's entire attention memory
token-shift state	2 × 384 registers	4,608 registers	one previous-token vector per sublayer
wkv arithmetic	384 cells × ~12k gates, or time-multiplexed	~5% of area	one divider serves all channels — only the output path divides
matmuls	R/K/V/O projections + FFN ≈ 1.9M weights	≈ 11M weights in the block stack*	the real area: multiply-accumulate arrays, not the recurrence
nonlinearities	sigmoid, squared-ReLU	—	same ROM-LUT treatment as the verified 2^(−x) unit

*Architecture-derived estimate (d = 384, 4d FFN, 6 blocks, excluding embeddings) — not a measured count. Everything else in this table follows from the architecture; the measured facts on this page are the decay/bonus statistics, the per-block error growth, and the perplexity verdict.

The striking inversion: the part of the model that is recurrent — the part this whole research line exists to build — is tiny. All of RWKV's memory is 6,912 registers, smaller than a single row-pair of its embedding matrix. The expensive part is ordinary linear algebra, which is exactly the part hardware already knows how to build. A full-model Morpho circuit is therefore possible and mostly boring: the interesting, novel silicon — the wkv bank — is already designed, format-validated (perplexity 6.244 in every format tested), and componentwise verified; the matmuls are commodity MAC arrays around it.

the ladder of timescales

block	median half-life	slowest channel
0	0.8 tokens	102 tokens
1	1.6	99
2	2.8	94
3	4.5	89
4	7.7	93
5	10.8	86

The export revealed structure the summary statistics had hidden: memory horizon grows monotonically with depth — early blocks handle roughly the current token, deep blocks integrate over ten — while every block keeps a handful of ~90–100-token channels. Not a flat cloud of decays: a learned, depth-ordered hierarchy of timescales, with a thin long-memory reserve at every level. It also closes the loop on the error-propagation measurement: block 5's quantization error grows over a sequence precisely because its median channel remembers 13× longer than block 0's.

could the whole model run on this page?

Step 1 is done — this atlas is now exact. Step 3 is unblocked too: the full model has been exported and validated at int8 with held-out perplexity 6.4224 → 6.4215 (−0.014%), per-row symmetric quantization with the dequant recipe and tokenizer bundled — everything a browser page needs. The 14.9 MB binary stays out of git clones; the plan is a GitHub release asset on the fork for distribution, with a copy staged into site/ on the deploy branch when the in-browser runner page ships (same-origin fetch keeps the deploy self-contained). That runner — the whole model generating text through the same circuit numerics the sweep validated — is the next page in this series.

Measured numbers from the trained 6-block model and its quantization sweep: w₂/u ranges and medians, the 1-in-2,304 slow channel, per-block error growth, perplexity 6.244 across all formats. Derived numbers are labeled. The wkv cell's circuit design and its exhaustively-verified 2^(−x) unit: the wkv cell · examples/arithmetic/.

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.