the wkv cell

RWKV's attention, per channel, is three registers and a leak. This page is the circuit plan: the numerically-stable recurrence, the unit inventory, its mapping onto MorphoHDL's verified components — and a live simulator running the recurrence in floating point and in the proposed fixed-point circuit numerics side by side, so the precision question is a picture instead of a guess.

← found machines ← circuits that move run the whole model live → to silicon → code

the recurrence

RWKV-v4's per-channel state is aa (signed numerator), bb (positive denominator) and pp (running max-exponent). Per token, with inputs k, v and trained constants u (bonus) and w (decay, negative):

ww  = u + k                     p  = max(pp, ww)
out = (2^(pp−p)·aa + 2^(ww−p)·v) / (2^(pp−p)·bb + 2^(ww−p) + ε)
ww2 = pp + w                    p2 = max(ww2, k)
aa′ = 2^(ww2−p2)·aa + 2^(k−p2)·v
bb′ = 2^(ww2−p2)·bb + 2^(k−p2)
pp′ = p2

Two facts do most of the engineering work. Every exponent is ≤ 0 — each is x − max(…, x) by construction — so only the decaying half of the exponential exists, the result lives in (0, 1], and underflow to zero is correct, not an error. And base 2 costs nothing: absorb log₂e into the trained constants and 2^(−n.f) becomes a barrel shift by n plus a small lookup for the fraction.

            k ───┬──────────────┐        v ──────────┐
                 │              │                    │
   u ──▶(+)── ww │       ┌──▶ max ── p              │
                 │       │      │                    │
  pp ────────────┼───────┤   ┌──┴───────────┐        │
   ▲             │       │  2^(pp−p)   2^(ww−p)     │
   │             │       │   │  │        │  │        │
   │  w ──▶(+)── ww2     │   ▼  ▼        ▼  ▼        │
   │             │      aa ×    bb ×     ×  ◀────────┘
   │             ▼       │       │       │
   │     max(ww2, k)=p2 └──(+)──┼──(+)──┘
   │             │           num │  den(+ε)
   └── pp′ ◀─────┘               ▼   ▼
        aa′,bb′ ◀── same trick ──▶ (÷) ──▶ out

one channel, live

float vs circuit numerics

The same recurrence runs twice on the same token stream: once in double precision (green), once in the proposed hardware numerics (amber) — fixed-point registers, the 2^(−x) unit as barrel shift + 32-entry interpolated LUT, and division by a 16-bit restoring divider (the one added to examples/arithmetic/). Red is the error, magnified. Drag the decay: slow channels (w near 0) are the precision stress test.

v (input) out — float out — circuit |error| ×32

error accumulation over the recurrence (log scale, 10⁻⁶ … 1) — the stateful question

decay w bonus u format 2^x LUT

trained-model presets:

evidence, not illustration

The whole cell is now a compiled circuit. examples/rwkv/wkv_cell.py builds this page's entire Q.step — saturating Q8.8 registers, base-2 exponentials, floor-semantics multiplies, restoring division, the lot — as one MorphoHDL netlist (7,201 gates + 48 flip-flops), verified bit-exact against this page's own JavaScript on shared token streams over trained-model presets (0 mismatches in 2,000 steps, checked in a real browser), and synthesized to a real FPGA through the yosys → nextpnr flow (examples/hardware/).

The simulator's 2^(−x) unit is bit-identical to a compiled MorphoHDL circuit: examples/arithmetic/exp2.py implements it as ROM LUTs + the article's logarithmic shifter + an 11×3 Wallace interpolator (388 gates) and verifies it exhaustively over all 65,536 Q8.8 inputs against exactly the semantics running on this page — worst deviation from true 2^(−x) is 6.5×10⁻⁵. The division uses the same restoring-division semantics as the verified Morpho divider (examples/arithmetic/divider.py, 527 gates at 16/8). Registers, maxes, multiplies and adds model the verified components bit-for-bit at these widths. Full-cell bit-exactness — the compiled wkv_cell against this page on shared token streams — is the first deliverable after the format is chosen by measurement.

Independent validation (trained-model side): a separately-written implementation of the same circuit numerics, run on synthetic keys/values across the full decay range, measured RMS error 2.6×10⁻³ with exact division and 3.5×10⁻³ with the restoring divider — independently reproducing this page's ≈3×10⁻³. More importantly, the stateful question has a measured answer: error is bounded over a 256-step recurrence, not merely small at one step — 1.8×10⁻³ in the first quarter, 2.5×10⁻³ in the last, rising slightly then settling, with zero saturation events at Q8.8. The max-subtraction normalization is doing its job: quantization error does not compound through aa/bb.

the sweep reports

Held-out perplexity: 6.244 in every format tested. Q8.8, Q6.10, Q10.6; LUT 32 or 64; interpolation on or off; exact division or the restoring divider — all identical to the float64 reference to three decimal places, and the Q8.8 circuit's generated story is verbatim the float model's:

Once upon a time, there was a little girl named Lily. She loved
to play outside in the sunshine. One day, she went to the park
with her mommy and saw a very big slide. She wanted to go, but
she knew she had to be careful.

Not because the arms are equivalent — at the wkv output they span a 12× range in RMS error (Q6.10 best at 8.4×10⁻⁴, Q8.8 with interpolation 2.5×10⁻³, without 4.3×10⁻³, Q10.6 worst at 9.8×10⁻³) — but because the model has enormous headroom: ~0.2% relative error on a signal spanning ±4.9 does not measurably move the next-token distribution. The circuit numerics are not the binding constraint. Pick the format on area and timing grounds.

A prediction falsified, and why. This page predicted, on record, that Q6.10 would fail first on the decay side because pp + w₂ must represent steps to −26.5. Measured: Q6.10 racked up 98,746 saturation events — and they cost nothing. The reason is structural: pp clips at the bottom of the format exactly in the regime where pp − p is already below ≈ −20, i.e. where the exponential has already underflowed to zero. The clamp lands only where the value's magnitude no longer matters. Revised sizing rule: what must be representable is w₂ and k₂; pp's lower bound is cheap.

Error accumulation is depth-dependent (measured on real activations, per block and timestep):

block	t = 0–32	t = 32–128	behaviour
block 0 (median w₂ ≈ −1.2)	1.6×10⁻³	2.2×10⁻³	flat — decay flushes error as fast as it enters
block 5 (median w₂ ≈ −0.09)	6.0×10⁻³	1.7×10⁻²	grows ~6× — slow decay integrates error longer

The honest caveat to pair with "indistinguishable": deeper blocks hold memory longer and therefore compound quantization error. It does not hurt this model at this context length — but it is exactly the mechanism that would bite a longer-context one. The simulator's stress case is precisely this: drag the decay slider toward zero and watch the error trace climb.

unit inventory · per channel per token

unit	count	in Morpho today
comparator + mux (max)	2	have — ≥ is the ripple adder's carry-out (a + ~b + 1); Mux2 selects; signed = flip sign bits first
2^(−x) unit	4 (or 1, time-multiplexed)	mostly have — barrel shift is the article's `right_shifter`; the fractional table is an arity-5 LUT per output bit (native — tables are just ints) + one small multiply for interpolation
divide / reciprocal	1	have — restoring divider added in `examples/arithmetic/` (527 gates at 16/8); Newton–Raphson reciprocal (seed LUT + 1–2 iterations of r·(2−d·r)) is the smaller-area alternative and the denominator is always > 0, so no sign handling
fixed-point multiply	~6	have — Wallace tree; Qm.n product is a bit-slice of the full 2N-bit output, pure wiring
add	~4	have — ripple / Brent–Kung; two's complement works unmodified
registers	3 banks (aa, bb, pp)	have — REG/DRIVE

The structural gift: wkv is completely independent per channel. One wkv_cell with its three REGs, and Morpho's ordinary SPLIT recursion generates the bank at any width — 384 channels is a parameter, not a rewrite. Rough area at 16-bit: ~10–15k gates per fully-parallel channel; the streaming divider/exponentiator pattern gives the channel-serial rotation if area wins over latency.

precision plan

The max-subtraction already bounds every exponent, so dynamic range is solved by construction; the open question is fraction width. The presets above come from a trained 6-block model: w₂ = w·log₂e spans −26.5 to −0.0098 (median ≈ −1.2 in early blocks, ≈ −0.09 by block 5 — deeper blocks hold memory longer), u spans −0.07 to 0.72, and slow-decay channels are a genuine but rare tail: exactly 1 channel in 2,304 holds memory beyond 100 tokens. That tail is the stress case — it integrates quantization error the longest — but it is a tail risk, not the operating point, which changes how much headroom is worth buying.

pp width, concretely: pp is a running max of k₂ and pp + w₂, so it tracks recent key magnitudes rather than accumulating without bound. The measured sizing rule (see the sweep results below, which falsified this page's first guess): the format must represent w₂ and k₂ faithfully; pp's lower excursion is cheap to clamp, because by the time pp hits a format floor the exponentials it feeds have already underflowed to zero, so the clip lands where magnitude no longer matters.

Two architecture notes. Only the output path divides — the state update doesn't — so one divider (or Newton–Raphson reciprocal) can be time-multiplexed across all 384 channels rather than instantiated per channel. And the base-2 export transform is exactly: w₂ = w·log₂e, u₂ = u·log₂e, with k₂ free by scaling the key projection matrix by log₂e at export — mathematically identical, hardware-native.

The committing test remains measured task quality: quantize the trained model's wkv path to the exact candidate numerics (this LUT exponential, this divider, candidate Q-formats) and read perplexity and rollout quality per format. This page picks candidates; that measurement commits one.

Recurrence: RWKV-v4 numerically-stable form, base-2 semantics (log₂e absorbed into trained constants — mathematically identical). The circuit path in the simulator: fixed-point registers with saturation, 2^(−x) = barrel shift + interpolated fractional LUT, division = 16-bit restoring divider. Building blocks verified in the repository: examples/arithmetic/ (divider, exponentiation), the article's adders, multiplier and shifter, and the sequential extension's REG/DRIVE.

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.