spikes in the browser

Two spiking RWKV models — trained with hard integer spikes in place of the channel-mix nonlinearity — running from their int8 exports, next to the float baseline. The model difference is one line of code. The point of this page is the measurement the other pages can't make: on a scalar substrate, does skipping the silent channels actually pay?

the one-line difference

// float baseline (channel-mix nonlinearity):
t = max(0, v);  k[i] = t * t;                    // squared ReLU

// spiking (levels = 4, threshold = 0.5, uniform):
n = min(max(floor(v / 0.5), 0), 4);  k[i] = n * 0.5;   // integer spike count

Everything else — layernorms, token-shift, the wkv recurrence, receptance, the head, sampling — is byte-identical to the float runner. The spike unit is its own rectifier (negatives floor to zero spikes), and after it, the 1536-wide hidden vector is ~90% exact zeros. The matmul that consumes it, W_v (1536 → 384), is where sparsity either pays or doesn't — so that is the matmul this page implements twice.

honest framing, before the numbers

These are three separate trainings, not one model quantized. The float baseline measures 6.42 perplexity, the 4-level spiking model 6.77, the binary model 6.39 — all at their own training budgets. The binary model beating the 4-level one is not binarization helping: its annealing ladder (4 → 2 → 1 levels) gave it 1,500 extra fine-tuning steps. Unlike rwkv-live's int8 toggle — genuinely the same weights — quality gaps here are training outcomes.

The thresholds never trained. All 9,216 per-channel thresholds sit at exactly their 0.5 init: the training code never routed a gradient to them. The page asserts this from the shipped tensors rather than assuming it. It's a useful negative result — per-unit spike parameters were the biggest quality lever in prior spiking-decoder work, so the 6.77-vs-6.42 gap now has a specific untested explanation with an experiment queued.

why the browser is an honest venue

On a GPU, dense SIMD matmul beats unstructured sparsity — lanes execute in lockstep and skipping a channel saves nothing. Plain JavaScript is the opposite: one scalar pipeline, where every skipped channel is 384 array operations that never run. So the neuromorphic argument — computation should cost in proportion to events, not to capacity — gets a fair scalar trial here.

Locality is part of the design, not a trick: the event-driven kernel walks a transposed int8 copy of W_v, so each fired channel reads one contiguous 384-byte row. Iterate the original row-major layout column-wise and the same "sparse" kernel loses to dense on cache misses — misattributing a memory-layout failure to sparsity itself. Silicon has the same requirement; event-driven hardware earns its efficiency only with event-major weight layout.

With the uniform 0.5 threshold the spike vector takes five values (0, ½, 1, 1½, 2), so the event kernel accumulates in pure integers — acc[j] += n · w8[j], int8 weight × spike count ≤ 4 — with the per-row scale and the ×½ folded in once per output. Exact, not approximate: that is why both kernels emit identical text. In silicon that inner op is a shift-add; at binary it is a pure add.

what the numbers mean

The firing ladder echoes the timescale ladder: deeper blocks fire more, just as they remember longer. Per token, the six W_v matmuls are 3.54M of the model's 13.1M MACs; at ~10% firing the event-driven kernel replaces them with ~0.35M integer accumulates. The energy line the counters show uses the standard 45 nm figures (int8 multiply-accumulate ≈ 4.6 pJ, add ≈ 0.9 pJ) — an accounting exercise on counted operations, not a wattmeter, and a saving that exists on event-capable substrates, not on GPUs.

One expected artifact of int8 under a discontinuous nonlinearity: quantization error occasionally moves a channel across a spike threshold, so int8-vs-float logits differ more here (rms ≈ 0.11) than in the float baseline (rms ≈ 0.02). The perplexity columns above show it costs nothing — but the mechanism deserves naming.

does spiking reduce logic gates? — measured

The question that decides whether any of this matters for silicon. Spiking changes hardware in exactly two places, and both are now real compiled Morpho circuits, verified bit-exact on 200,000 random cases each and synthesized through the same yosys flow as the wkv cell:

So: yes, and by a lot — where spikes flow. The consumer lane shrinks 2.1× at 4 levels and 5.1× at binary, because the multiplier's area goes as the operand width and a spike count is 3 bits, then 1. The producer collapses almost entirely: squared-ReLU is a 16×16 multiplier (566 LUTs), the uniform threshold is bit inspection (5 LUTs).

And the whole-model number is counted by construction, not extrapolated: examples/rwkv/count_gates.py builds the actual matmul engines at full width — the 384-lane value engine, the 1536-lane key engine (497,664 gates, dense), all variants, lane-linearity asserted rather than assumed — and assembles the datapath from the emitted netlists (plus the measured 7,201-gate wkv cell):

The signed-ternary input lane the fully-spiking variant needs (add or subtract the weight, never scale it) measures 96 gates — 3.4× smaller than dense. Two honest boundaries hold throughout: this model routes spikes only through W_v — 27% of its MACs — which is why the current saving is single-digit; and event-driven skipping saves energy and time, never area — gates are laid down for the worst case.

A correction, published rather than patched: an earlier version of this paragraph claimed weight bandwidth is untouched by spiking, so bandwidth-bound builds see no tokens/s gain. That's only true of a build that fetches densely. An event-driven build never fetches the W_v rows of silent channels — with weights stored event-major (the same transposed layout the kernel race above depends on; flash and silicon share the locality requirement), per-token traffic drops from 13.1 MB to ~9.9 MB at the measured ~10% firing, which is a real ~1.2–1.3× on the bandwidth-bound builds from the silicon page (the discount from the clean 1.32×: fired rows are known only after the layer's spikes are computed, so those fetches can't be fully prefetched, and row-granular skipping costs some random-access efficiency). The fully-spiking variant would roughly double the skippable share. What spiking raises unconditionally remains tokens per second per unit area and per joule. Reproduce: python3 examples/rwkv/spike_mac.py (lanes) and python3 examples/rwkv/count_gates.py (engines and totals).

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.