changing tack

Why the language research is moving from a reservoir-plus-readout to a directly trained spiking language model — what five experiment generations taught us, what survives the pivot, and what we're building next. August 2026.

← the language campaign (Results 1–9) → chapter two: a spiking model that speaks ← research notes code & experiments

Where we got to

Over experiment generations v8–v18 the system climbed from bigram-level (28.8%) to 47.2% teacher-forced next-character accuracy on tiny Shakespeare — backprop-free throughout, with a genome of eleven numbers growing a 120,000-neuron spiking reservoir and a closed-form readout. Every protocol was pre-registered; every null published with its controls. And then we stopped, for a reason the score can't show: the output was still unusable. Free-running generation remained incoherent at every rung of the accuracy ladder. Chasing decimal improvements to a sub-50% teacher-forced score had stopped being research into language and become research into readouts.

What we learned (the validated ledger)

finding	evidence
Developmental laws select structure well. An 11-gene wiring law evolved at 2–8k neurons transfers to 120k; evolution chose depth 2 over the designed 4, unanimously, and won at up to 6.8× fewer synapses	v13–v14, three seeds each, held-out scales
The spiking dynamics contribute real sequence information. Aligned spike traces add +5.4pp over explicit context; the identical topology run as a non-spiking rate reservoir adds +0.8; generic temporal features +0.2; misaligned traces are harmful	v18a ablation, 5 build seeds, ±0.5pp
Information lives in spike timing, in flight. Memory-selected organisms store the past as spikes circulating around delayed inter-layer loops — not recurrent attractors, not weights. Destroying timing alignment destroys the value	v17 genome census; v18a timeshift arm
The substrate's memory is real but short and noisy — recall-4 at ~2.8× chance — and noisy memory loses to perfect explicit context, so it never converted to language gains	v17 language transfer null
Teacher-forced accuracy diverges from usable generation. Accuracy climbed 8 points while rollouts stayed garbled; greedy decoding even gamed the generation metric by looping	v15 generation benchmark
Local-only learning did not produce capability. Forward-Forward lost to a linear solve by 26pp with every recommended ingredient present; reward-modulated plasticity reshaped half the synapses with zero effect. The capability always came from global optimization — the ridge solve and evolution	v16c/d, controlled

Why we're changing tactics

Three of those findings, taken together, decide it. The system's learning was already global — "backprop-free" never meant "forward-only," and we published that taxonomy. The metric that guided the climb doesn't measure the behavior we want — a model that writes. And the ingredients that demonstrably work (temporal spiking dynamics, developmental structure selection, global optimization) are exactly the ingredients of the directly trained spiking language models that now exist at scale: SpikeGPT (autoregressive spiking RWKV, trained with surrogate gradients), BICLab's spike-driven transformers (whose v3 lesson — relax binary spiking during training, keep spike-driven inference — our substrate taught us three times in miniature), and Sakana's Continuous Thought Machines (Llion Jones et al.), whose thesis — neural synchronization as representation — is precisely what our ablation just measured: the value of our reservoir is carried by spike timing alignment, and our one synchronization-flavored readout (pairwise trace products) beat rate features at matched size.

Morpho's spiking substrate was not empty: controlled ablations (5 build seeds) found genuine timing-borne sequence information — aligned spiking traces 42.5 ± 0.5% vs 37.9 ± 0.2% for the matched non-spiking reservoir, 37.3% for neuron-free delay-line features, 36.0% with temporal alignment destroyed — that disappeared when timing was broken and was not reproduced by matched non-spiking dynamics. But the information was short-range, the supervised readout became the dominant capability, and rising teacher-forced accuracy failed to produce usable text. We therefore retain temporal spikes, developmental structure and rigorous evaluation, and replace the learning architecture with directly trained spiking sequence models. Build on what works; don't reinvent the wheel.

What survives the pivot

The evaluation culture — pre-registration, fixed-prompt rollouts, repetition and degeneration metrics, calibrated evaluator bits-per-char, nulls published with controls. Rollout coherence, not teacher-forced accuracy, is the headline metric from day one. The developmental thesis — Morpho's role sharpens rather than disappears: grow and prune the sparse structure of a directly trained spiking model between training phases, exactly the phase-wise regime our v13/v14 evidence supports. The timing results — delay distributions and synchronization readouts become design features, not curiosities. And the browser organism stays what it always was: the musical, interactive, artistic track at the lab.

What we're building

step	what	gate
0	Tiny conventional RWKV baseline on TinyStories (~10M params, 4k BPE tokenizer) — calibrates everything	coherent baseline rollouts
1	Same model, spiking activations + surrogate gradients + firing-rate regularization (SpikeGPT-style)	rollouts within sight of step 0
2	Integer-spike annealing (BICLab v3 style): train soft, infer sparse	sparsity without rollout collapse
3	Morpho development of the sparse structure between training phases; CTM-style synchronization features as a research arm	beats fixed architecture at matched params

The first milestone is unglamorous and non-negotiable: a small spiking model whose TinyStories rollouts are recognizably language. Everything else is gated behind it.

Forward-only learning was a hypothesis, not an identity. The experiments identified which biological mechanisms carried real signal; the next system keeps those mechanisms and gives them an optimization method capable of producing language.

The full chronological record — including every null and correction — is in snn/EXPERIMENT.md (v8–v18) and the language campaign page. References: SpikeGPT; BICLab spike-driven transformer series; Continuous Thought Machines; SpikingBrain conversion report. The reservoir campaign's claims are stated precisely in the claims taxonomy on the language page: the full system was a backprop-free hybrid, and the demonstrated result is "a developmental spiking reservoir provides useful, spike-timing-borne features under a supervised readout."

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.