five models, one page
Every language model this project trained, running in your browser from the same flat int8 format: a plain RWKV, three spiking variants of it, and a Mamba-style selective SSM. Same tokenizer, same data, same 13.1M parameters. Load any of them, give them the same prompt, and read the difference.
what you are switching between
| model | what changed | held-out ppl |
|---|---|---|
| RWKV float | the baseline — squared-ReLU activations | 6.84 |
| RWKV spiking | channel-mix nonlinearity → integer spikes, 0–4, ~9% firing | 7.18 |
| RWKV binary | annealed to single-bit events, ~11% firing | 7.35* |
| RWKV fully-spiking | signed input spikes too — both channel-mix matmuls multiply-free — plus channel-axis leak | 7.43 |
| Mamba S6 | a different recurrence: selective decay, 16-wide state, no division | 7.83† |
* binary was annealed from the spiking model with extra fine-tuning steps, so its number is not budget-matched to the others. † Mamba trained at batch 8 against everything else's 16, so it saw half the tokens — the caveat in full. All five int8 exports were validated against their float originals (worst case +0.04%), and the browser code is verified against PyTorch to four decimal places.
what to look for
The models are close enough that the interesting comparison is qualitative. Give them the same prompt and seed and they write recognisably different stories — but all of them write stories, with dialogue, causality and endings. That a model whose activations are single-bit events does this at all is the point of the spiking work; that a model with a completely different recurrence does it equally well is the point of the Mamba work.
For the spiking variants the page shows live firing rates per block — the fraction of spike units that fire on each token. They run around 4–5% in early blocks and 20% in the last, a depth gradient that emerged from the firing-rate regularizer rather than being imposed. On event-driven hardware that fraction is the share of the channel-mix arithmetic that would need to happen at all.
Weights load from rwkv-export/ and mamba-export/ beside this
page — per-row symmetric int8, w = int8_row × scale_row. Each model is
~15 MB and is fetched once when selected. Everything here is one HTML file and no
libraries; the arithmetic is the same arithmetic in
spikelm/model.py and spikelm/mamba.py, transcribed.
AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.