rwkv, live
The trained six-block RWKV model — the same one the wkv circuit was validated against — running token by token in this page, from the 14.9 MB int8 export. No server, no framework: a manifest, a tokenizer, a flat binary of per-row-quantized weights, and the recurrence written out longhand in JavaScript.
the runner
Load fetches the weights (14.9 MB, once) and dequantizes
per the manifest recipe w = int8_row × scale_row. Generation runs
the exact training-time semantics: layernorm → token-shift time-mix with the
numerically-stable wkv recurrence → squared-ReLU channel-mix, six blocks, tied
head. Switch the wkv state to Q8.8 circuit numerics to route every
time-mix through the fixed-point path the hardware sweep validated
(base-2, 32-entry LUT + interpolation, saturating registers).
what exactly is running
Everything the training code does, transcribed operation for operation from
spikelm/model.py:
x = LN_in(emb[token])
for each of 6 blocks:
xx = LN1(x) # time-mix
k,v,r = W·(xx·μ + xx_prev·(1−μ)) # token-shift mixing
wkv = aa/bb/pp recurrence (w = −e^decay, u = time_first)
x += W_out · (σ(r) · wkv)
yy = LN2(x) # channel-mix
x += σ(W_r·mix) · W_v · relu(W_k·mix)² # 384 → 1536 → 384
logits = head · LN_out(x) # head tied to the embedding
The wkv recurrence keeps the same three registers per channel that the
circuit page simulates — a value accumulator
aa, a weight accumulator bb, and the running maximum
exponent pp that keeps every exponential argument ≤ 0. In float
mode those are IEEE doubles; in Q8.8 mode they are saturating fixed-point
registers and e^x becomes a barrel shift plus a 32-entry lookup
with 3-bit interpolation — the same unit that exists as
compiled Morpho, exhaustively bit-exact over
all 65,536 inputs.
the numbers behind the toggle
Neither mode is a leap of faith — both halves of the quantization story were measured on held-out data before this page existed:
| what | perplexity | change |
|---|---|---|
| float32 weights (training checkpoint) | 6.4224 | — |
| int8 weights (this page's binary, per-row scales) | 6.4215 | −0.01% |
| wkv state in Q8.8, LUT32 + interp (weights float) | 6.244* | parity with float64 wkv |
*The wkv-format sweep used its own evaluation batch, hence the different absolute number; every format tested — Q8.8, Q6.10, Q10.6, no-interp, restoring division — landed on the same perplexity to three decimals. Details on the wkv cell page.
What this page adds is the two cuts combined: int8 weights
and fixed-point wkv state at once, generating text you can read. The
JavaScript recurrence here mirrors the validated emulation exactly — base-2
reparameterization, saturating Q8.8 quantization after every add and multiply,
pp initialized to −60, the same ε in the denominator.
where the weights live
The three small files (manifest, tokenizer, validation record) travel with
the page. The 14.9 MB binary is deliberately not in the repository — it is
staged next to the page at deploy time, and the natural durable home is a
GitHub release asset on the fork: release downloads are served with
Access-Control-Allow-Origin: *, so a deployed page can fetch the
binary straight from the release URL with no copy at all.
If the binary isn't reachable the loader says so rather than pretending — this page makes no claims it can't back with a running model.
AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.