a spiking model that speaks

Chapter two of the language research. After the reservoir campaign closed, we rebuilt from the other end: a small spiking language model trained directly with surrogate gradients. Milestone 0 — it writes stories. Milestone 1 — at matched budget, replacing its activations with spikes firing 9% of the time costs about 6% perplexity and nothing readable. August 2026.

The setup

One model, one flag. A 13.1M-parameter RWKV-style recurrent language model — 6 blocks, width 384, a 4,000-token BPE vocabulary, 256-token context — trained on TinyStories with ordinary next-token cross-entropy and AdamW. The spiking arm is byte-identical except that each block's channel-mix nonlinearity is replaced by integer spike units: the pre-activation is quantized into 0–4 spikes against a learnable threshold in the forward pass, and a rectangular surrogate carries the gradient backward. A regularizer penalizes firing above 10%. Thresholds are stored per channel, per block (9,216 of them) — but see the correction below: they never trained.

Everything else — residual pathways, embeddings, the recurrent state update, the output head — stays floating point, deliberately. Three independent sources say so: BICLab's v3 lesson (train soft, infer sparse), SpikeDecoder's ablation where fully-spiking residuals collapsed the model from 98% to 18–42%, and our own reservoir campaign, which learned the same thing the hard way. This is spiking as an increment, measured one site at a time — not a purity contest.

Milestone 0 — does it speak at all?

The conventional baseline is the yardstick everything else is measured against, and it cleared the bar quickly. Perplexity fell 22.0 → 12.6 → 10.6 → 9.9 over the first 2,000 steps and reached 6.42 by step 5,500 (~4.5 hours on an M3 laptop, 2,000 tokens/second). But the number is not the point — the rollouts are. Five fixed prompts, generated fresh at every checkpoint with fixed sampling seeds:

Between those two checkpoints the model stops inventing morphology ("walkinging"), starts closing narrative arcs, and — the detail we enjoyed most — learns that stories end: it emits the document separator and begins a fresh, unrelated story. It learned the shape of the corpus, not just its words.

Milestone 1 — what do spikes cost?

The controlled question. Same data, same 13.1M parameters, same learning-rate schedule, same evaluation harness, same sampling seeds — the only difference is the activation function. Both arms run to step 5,500.

The two curves cross three times. The spiking arm leads early, wobbles while its thresholds settle, leads again around step 3,000, and finishes 0.38 perplexity (≈6%) behind. On degeneration — the metric that actually predicts whether generation is usable — the spiking model is better: repeated 4-grams 1.0% vs 1.6%, distinct bigrams 0.882 vs 0.872.

And read side by side at step 5,500, from the same prompt and seed:

Two models, one with continuous activations and one communicating in integer spikes, writing the same story about the same girl and the same slide.

The sparsity actually achieved

Measured on held-out text at step 5,500, the fraction of spike units firing per block:

block   1     2     3     4     5     6      mean
rate    4%    5%    6%    8%   13%   20%     9.2%

Nine percent. On event-driven hardware, that is the fraction of channel-mix activations that would need to be touched at all — and the gradient of sparsity across depth (dense at the top, sparse at the bottom) emerged on its own from the rate regularizer, not from a schedule we imposed.

Milestone 2 — how little can a spike say?

Our spike units emit an integer count against a learnable threshold — 0 to 4, roughly 2.3 bits per unit per token. Milestone 2 squeezes that alphabet down a rung at a time, fine-tuning 500 steps at each level: 4 → 2 → 1, a true binary event. Fired, or didn't. One bit. This matters because the energy argument that motivates spiking is cleanest when a spike is an event rather than a number, and because the recipe we borrowed — train with soft multi-level spikes, anneal toward binary — is an assumption we adopted on the literature's authority rather than our own evidence.

Quality survived the squeeze. A model whose channel-mix units communicate in single-bit events sits 3% behind the float baseline in perplexity and writes the same kind of story:

The second number matters as much as the first. A binarized model can hide lost amplitude by simply firing more often — same information, worse sparsity, and the deployment story quietly evaporates. It didn't happen: mean firing rate moved only from 9.2% to 10.0% across the whole ladder. The sparsity is real and it held.

Correction: the thresholds never trained

This page originally described the spike thresholds as learnable per channel and per block, citing SpikeDecoder's finding that per-unit spike parameters were their single largest quality lever (+11pp). Exporting the checkpoint for the browser demo revealed that all 9,216 of them are exactly their initial value, 0.5: our surrogate-gradient function returns no gradient for the threshold argument, so the optimizer skipped the parameter entirely. Every result on this page was produced with fixed thresholds.

That makes the correction more interesting than a typo. The lever we believed we had adopted was never engaged — so the 6.80-versus-6.42 gap has an untested explanation sitting in it, and wiring the gradient up is now a specific experiment with a prediction attached rather than a vague "tune it more". It also simplifies the hardware story in the meantime: with a single constant threshold, the spike vector takes only five values (0, 0.5, 1, 1.5, 2), so a spike-consuming matmul is an integer-weighted accumulation — acc += n · w with n ∈ 0..4 — rather than a general multiply.

What that would mean on the right hardware

Nothing yet on ours, and it is worth being exact about why. Our spikes live in ordinary float tensors that happen to contain mostly zeros, and the matmuls are still dense: a GPU does not skip a multiply because an operand is 0.0, and unstructured 10%-density sparsity is slower than dense arithmetic on these kernels, not faster. Both arms measure ~2,000 tokens/second and 2.9GB. Binarizing activations also does not shrink the 52MB of weights.

What it earns in principle: the projection that consumes the spike vector stops being multiply-accumulate and becomes pure accumulation over the ~10% of inputs that fired. For this architecture that projection is about a third of each block's matrix work, so the honest operation-count estimate is roughly 30% off a block's arithmetic energy — not the 87–93% this literature usually quotes, because we have spiked one projection, not the whole model. On event-driven silicon the silent 90% would cost nothing at all. Every number in this paragraph is an operation-count proxy; none of it is measured watts.

Correction: the numbers above are single-batch

Every perplexity on this page came from one 8×256 validation batch per checkpoint, which we later found was too noisy to resolve differences of this size — one arm swung 17% between adjacent checkpoints. Re-evaluating the saved checkpoints on 16k tokens instead of 2k:

The ordering survives but the fully-spiking arm's apparent catch-up does not: on the noisy single-batch numbers it looked level with, and once ahead of, the plain spiking model. On the proper evaluation it is behind both — and it had 500 extra steps. So making both channel-mix matmuls multiply-free costs about 3.4 points on top of spiking's 4.9, not nothing. Still arguably a good trade for ~54% of the model's arithmetic becoming accumulate-only, but a trade rather than a free lunch.

What this is not

It is not an energy result. The commonly quoted 87–93% reductions in this literature are operation counts — spikes turn multiply-accumulates into accumulates, and event-driven silicon skips silent units — and none of that is realized on a laptop GPU, where both of our arms burn identical watts and run at identical speed. Our 9.2% firing rate is the honest input to such an estimate, not the estimate itself. It is also not a parameter-efficiency result: both arms are the same size, and the literature that spikes attention has needed more parameters, not fewer.

And the spiking machinery here is deliberately borrowed rather than invented: the RWKV backbone from SpikeGPT, integer-spike training from BICLab's spike-firing approximation, per-unit learnable spike parameters from SpikeDecoder's ablations, surrogate gradients from standard practice. The novelty budget is being saved for what comes next.

What comes next

Milestones 3 and 4 are where this stops being a reproduction and starts being research again — and both rest on findings the first chapter paid for.

Reproduce: cd spikelm && python bench.py (throughput on your machine), python -m spikelm.data, then python -m spikelm.train and --spiking. Checkpoints, JSONL logs and every rollout sample land in runs/. All figures above come from those files. Chapter one — the developmental reservoir, its positive attribution result and its ceiling — is here, and the reasoning behind the change of direction is here.

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.