two recurrences

We trained a Mamba-style selective state-space model against our RWKV at matched parameters, in the same harness, to find out what selectivity costs and buys — in quality, in speed, and in gates. What works, what it trades away, and the one number we are not entitled to quote. August 2026.

Why bother

The hardware track needs to know which recurrence is worth committing to silicon, and the literature's answer — selective SSMs beat fixed-decay recurrences — was established at scales and budgets nothing like ours. So we built Mamba-mini: 13.14M parameters against RWKV-mini's 13.10M (0.3% apart), same tokenizer, same TinyStories data, same 6 blocks at width 384, same optimizer and schedule, same evaluation harness. One architecture swapped, everything else held.

The recurrences differ in exactly three ways. RWKV keeps three scalar registers per channel; Mamba keeps a 16-vector. RWKV's decay is a fixed constant per channel; Mamba computes its decay from the input at every step — that is the "selectivity". And RWKV normalizes (it divides); Mamba does not divide at all.

What works

It trains without drama. No stability fights, no exploding scans, no architecture-specific hyperparameter search — the same learning rate and schedule that trained RWKV trained this. Given how much of the spiking work was spent on stability, that is worth stating.

It writes well. At the end of training, from the same prompt and seed the other models use:

Dialogue with correct attribution, two speakers, a coherent domestic scene. On the degeneration metrics that actually predict usable generation it is the equal of anything we have trained: repeated 4-grams 0.8%, distinct bigrams 0.91.

It quantizes for free. Held-out perplexity 7.274 in float32 → 7.275 in per-row int8, a +0.02% change. The same result we got for RWKV (−0.01%) and the spiking models, now for a third architecture: weight precision is not where these models are fragile.

Its state is constant. 1,536 inner channels × 16 states × 6 blocks = 147,456 numbers, and that never grows no matter how long the text gets — the property this architecture family exists for, shared with RWKV and absent from attention. You can watch it hold in the browser page, which runs the model in true streaming form: one token in, one token out, constant work.

The tradeoffs

Speed. Four times slower to train at the same parameter count, and the reason is structural rather than incidental: the scan carries a 16-wide vector per channel instead of three scalars, so every timestep touches an order of magnitude more state.

Gates. It is tempting to conclude that dropping the divider makes S6 the cheaper circuit. It does not. The divider is roughly 5% of a wkv cell; the N-wide state costs 11×, and Mamba-mini runs four times as many inner channels. The circuit page works this through, and the difference that survives all of it is data dependence: RWKV's decay is a constant that can be baked into the circuit — potentially a hard-wired shift — while Mamba's must be computed live, every token, forever. That is what selectivity costs in silicon.

What it buys is the thing our own reservoir campaign found independently, from the other end: input-dependent state writes carry information that fixed decay cannot. Back in the backprop-free work, a Mamba-style selective state beat a plain slow state at matched dimensions (30.0% vs 28.5%) even in a substrate with no gradients at all. The mechanism is real. The question is only ever what it costs.

The number we are not entitled to quote

The honest statement is narrower and still useful: a selective SSM at half the data budget lands 14% behind a fixed-decay recurrence, while costing 4× the training time and an order of magnitude more state. The controlled rematch — RWKV re-run at batch 8, about forty minutes of machine time — is the outstanding experiment, and until it runs the quality column stays provisional. The circuit accounting in the table above is unaffected, because it follows from the architectures rather than from either run.

A second caveat we owe: we configured Mamba at expand=4 to hit parameter parity at 6 blocks, where canonical Mamba uses expand=2 with more, thinner layers. Parameter parity was the right call for this comparison, but it is not necessarily Mamba's good operating point.

And a prediction on the record, wrong: before running this we predicted Mamba would win at matched parameters, on the strength of the selectivity literature. It did not, on the evidence we have. We are keeping the prediction visible rather than quietly reframing it — partly because the batch confound means the prediction has not really been tested yet.

The result that outlives either model

That is also why both models now run in the browser from the same export format, and why both live at the root of the Morpho repository: two architectures, one flat binary format, one page of arithmetic each, verified against their PyTorch originals to four decimal places. Portability is the point — the same argument the hardware pages make in gates, made here in bytes.

Reproduce: cd spikelm && python -m spikelm.train --arch mamba --batch 8 (RWKV is the default --arch rwkv). Model, training logs and rollout samples are committed under spikelm/; the export and its int8 validation come from export_weights.py. Run them: mamba, live · rwkv, live. The circuit treatment: the s6 cell · the wkv cell.

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.