mamba, live

A 13.1M-parameter selective state-space model (Mamba/S6) generating text in this page, from a 15.5 MB int8 export — no server, no framework, no libraries. The companion to rwkv, live: same tokenizer, same data, same parameter count, a different recurrence.

what is actually running

Every operation transcribed from spikelm/mamba.py, in the streaming form — one token in, one token out, constant work per token regardless of how much text came before. That is the property this architecture exists for, and it is why the page needs no growing cache:

x = emb[token]
for each of 6 blocks:
  h        = LayerNorm(x)
  [xin, z] = in_proj(h)                    # 384 → 2 × 1536
  xc       = SiLU( depthwise_conv4(xin) )  # 4-tap causal, per channel
  [dt,B,C] = x_proj(xc)                    # 1536 → 24 + 16 + 16
  dt       = softplus( dt_proj(dt) )       # > 0, per inner channel
  A        = −exp(A_log)                   # < 0, shape 1536 × 16
  h_ssm    = exp(dt·A) · h_ssm + (dt·B)·xc # THE recurrence
  y        = ⟨h_ssm, C⟩ + D·xc
  x       += out_proj( y · SiLU(z) )
logits = head( LayerNorm_out(x) )           # head tied to the embedding

The whole model's memory is h_ssm — 1,536 channels × 16 states × 6 blocks = 147,456 numbers, plus a 3-token convolution history per channel. It never grows. Compare a transformer, which must keep every past key and value.

the numbers

The caveat that matters: Mamba trained at batch 8 against RWKV's 16, so at equal step counts it saw half the tokens — 2.8M against 5.6M. The 14% gap is therefore not a fair architecture verdict, and we say so rather than quoting it as one. The int8 comparison is exact: same weights, quantized.

why both pages exist

The point is portability, not a contest. Two architectures with different recurrences, trained in the same harness, exported by the same script into the same manifest format, and run by two small pieces of plain JavaScript that share a loader. If a model's weights are a flat binary and its forward pass is a page of arithmetic, it runs anywhere — which is the same argument the hardware track makes with gate counts instead of browsers. The S6 cell page carries that across: both recurrences reduce to h ← a·h + b·x with a = 2^(negative).

Weights: mamba-export/ beside this page — per-row symmetric int8 (w = int8_row × scale_row), A_log kept in float32 because exp() of it is precision-sensitive. Trained by spikelm/ on TinyStories, exported and validated by export_weights.py. If the binary is unreachable the loader says so plainly rather than pretending.

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.