soundlark language notes

A sideline to the music research: can a developmental spiking organism predict text — and can prediction error drive its growth toward the capacity the task needs?

→ changing tack: where this campaign led → chapter two: a spiking model that speaks ← music research notes code & experiments

Setup: the organism as a liquid state machine

No gradients enter the network — ever. Characters of tiny Shakespeare arrive as spike bursts through per-character input neurons; the recurrent developmental organism acts as a temporal feature map (a reservoir); and only simple readouts are trained on its spike traces: closed-form ridge regression, an online delta-rule softmax, and a Forward-Forward head (layer-local, goodness-based — Hinton's backprop-free algorithm, recently ported to SNNs).

We are deliberately not chasing transformers (~1.5 bits/char, ~58% next-char accuracy on this dataset with millions of trained parameters). The questions are about the substrate:

Result 1 — developmental exposure improves the reservoir

model	next-char accuracy	bits/char
uniform	—	5.95
bigram (exact)	28.8%	3.54
char-only readout (sanity ≈ bigram)	28.6%	—
fresh organism reservoir	29.2%	—
reservoir after developmental exposure	31.7%	—
char transformer (reference)	~58%	~1.5

Letting the organism develop while listening (growth + pruning + STDP over 15k characters, then frozen) produced a better feature map than the identical fresh genotype: +2.5pp, replicated across seeds, organisms growing ~150 → ~260 neurons. Development is acting as backprop-free representation learning — the same mechanism the music experiments found as "attention-as-morphogen", confirmed on a different modality.

Result 2 — error-driven growth

The developmental hypothesis, sharpened: an organism should grow when it lacks the capacity to answer correctly, and stop when capacity suffices. Mechanism: an online readout predicts each next character as text streams; the rolling error rate sets development's growth budget, and correct predictions deposit survival energy on the neurons that were just active — what helps, lives; what doesn't, starves. Controls: a frozen arm (no development) and an always-grow arm (maximum growth pressure regardless of error — so "more neurons" alone can't masquerade as the result).

arm	final size	held-out accuracy (fresh ridge)
frozen (no development)	143 neurons	31.9%
always-grow control	600 (budget cap)	32.3%
error-driven	600 (cap)	33.3%
error-driven, ceiling raised to 1400	834 — self-limited	33.3%

Error-driven growth beats the frozen organism (+1.4pp) and beats blind growth at identical final size (+1.0pp) — the error gating and correctness-survival direct capacity usefully; it is not just "more neurons". And with the ceiling out of reach, growth decelerates as error falls (+100 neurons per 10k chars early, +30–40 late, settling at 834 of a possible 1400) while accuracy saturates near 600 neurons — the progressive-growth-toward-adequate-capacity trajectory, with the residual limit now in the linear readout rather than the organism.

Honest negative: our minimal Forward-Forward head (one layer, 64 hidden units, goodness classification) underperformed closed-form ridge badly (12.1% on the frozen organism, worse on grown ones). FF likely needs feature normalization, depth and more training to be competitive here — a real follow-up, not a refutation of FF.

Result 3 — going big: 120,000 neurons, then prune down

The growth experiments run at 10² scale. The opposite regime — closer to how brains actually develop — is to start from a known structure, massively overprovisioned, and prune: a new engine (typed arrays, CSR synapses, event-driven, lazy decay) builds a 4-layer spiking network of 120,000 neurons / 2.4M synapses with feedforward, recurrence, skips, feedback and 15% inhibition, streams the full tiny Shakespeare corpus, and prunes by error-credit: neurons active during correct predictions live.

Getting a deep spiking network to compute at all is the classic stability fight — our first attempts were near-silent, then seizing at 200 Hz. The working recipe: sparse target rates (~2 Hz), continuous per-layer homeostatic thresholds, damped recurrent gain, membrane clamps. Then:

stage	size	held-out accuracy
full structured brain	120,000	33.0% — matches the 834-neuron grown organism
naive credit pruning	84k → 20k	collapses (27–29%) — inhibition pruned first → seizure → death
role-aware pruning (E and I pruned separately)	120k → 50k	32.7% — 58% smaller, accuracy intact
role-aware, further	38k	26.2% — activity floor reached

Three findings. (1) The overprovision-then-prune regime works: more than half the network is removable at zero accuracy cost, automatically, by error credit — but only if pruning respects roles: naive pruning strips the inhibitory scaffolding first and the network seizes, then dies (we measured the whole arc: 2030 → 5233 → 197 spikes/char). (2) A 120k-neuron brain and an 834-neuron grown organism score the same — the current ceiling (~33%) belongs to the linear readout, not to network capacity. The next lever is the readout: deeper Forward-Forward stacks, more feature taps, longer context. (3) 10⁵-scale spiking networks are perfectly tractable in a browser-class engine: ~75 chars/s while streaming the full corpus, sparse activity ~1.7%.

Result 4 — readout v2 and the Mamba mechanism

With the readout identified as the ceiling, one upgrade round on the 120k brain: more feature taps, traces at three timescales (τ = 20/80/320 ms — a multi-τ exponential trace bank is exactly a diagonal linear state-space memory, S4-family), previous-character context, a normalized 2-layer Forward-Forward rerank head, and a gradient-free analog of Mamba-Spike / SpikingMamba's core mechanism: selective state — the slow trace's write gain is the character's bigram surprise, so rare inputs write strongly into memory and predictable ones barely perturb it.

readout	held-out accuracy
v10 baseline (1024 fast taps)	33.0%
+ previous-char context	34.2% — new best
384 taps × 3τ state-space bank	28.5%
same + selective (surprise-gated) state	30.0% (+1.5pp over non-selective)
3072-dim bank at fixed 16k fit samples	14–18% — ridge collapse (sample-starved)
FF-v2 rerank (2-layer, standardized)	10.1% — still negative

Mixed, with two clean signals. Selectivity transfers: at matched dimensionality the Mamba-style surprise gate beats the plain slow state (+1.5pp) — input-dependent state retention helps even computed gradient-free from corpus statistics. Context is cheap accuracy: one extra symbol of context (+1.2pp) outperformed the entire temporal trace bank at this sample budget — big trace banks need proportionally more fit data or stronger regularization (the 3072-dim collapse is a sample-budget artifact, ratio 5:1). And Forward-Forward failed a third, properly-normalized attempt in our regime — recorded as a robust negative for shallow FF heads on reservoir features.

Result 5 — autoregression: conditioning pays, belief feedback doesn't, generation is humbling

The readout already trains on the autoregressive factorization P(next | history) with teacher forcing — GPT's objective. Three questions remained: does richer explicit conditioning keep paying? does feeding the model's own posterior back as features (RNN-style belief recurrence, via two-stage stacking) help? and what happens when it actually writes — sampling a character, feeding it back into the spiking brain?

arm (all at 40k fit samples)	accuracy
v11 features, 16k fit (prior best)	34.2%
same features, 40k fit	38.0%
+ second previous char	39.1%
+ belief feedback	39.2% (≈ null)

New best 39.1% — closing on halfway between bigram (28.8%) and a transformer (~58%), still fully backprop-free. The jump came from data and conditioning, not machinery: +3.8pp from more fit samples alone, +1.1pp from one more character of context. Belief feedback is an honest null under teacher forcing (the posterior adds nothing the features didn't already contain). And free-running generation produces gibberish — the classic exposure-bias lesson made vivid: 39% teacher-forced accuracy is mostly confident predictions of spaces and common letters, and a model never trained on its own outputs drifts off-manifold within characters. Generation quality, not held-out accuracy, is the honest yardstick still far away.

Result 6 — evolving the law, not the network

The experiment this whole fork was circling: don't evolve a network, evolve the rule that produces networks. An 11-gene genome parameterizes the per-neuron developmental wiring law behind Results 3–5 (E/I ratio, fan-outs, weight bases, gains, delay spread) — genome length constant in N, so the same genome instantiates a brain at any size. The readout stayed frozen at Result 5's; a deliberately boring (μ+λ) evolution strategy ran at 2k–8k neurons with fitness = mean accuracy across scales − a variance penalty (selecting scale robustness) − a connectivity cost. The pre-registered test: freeze the winners, instantiate at held-out 16k–120k — 15–60× beyond anything evolution saw — with 120k unlocked only after everything else was recorded.

arm (mean of 3 independent seeds)	acc @ 120k	synapses @ 120k
hand-designed law (Results 3–5)	37.6% (36.1–39.0)	2.38M
best random genome (gen 0)	39.6% (38.9–40.7)	1.34–2.03M
best evolved genome	38.6% (38.1–38.9)	0.71–1.44M

The developmental representation transfers across scale. All six small-scale-selected genomes hold 38.1–40.7% at 120k — a size they never saw — beating the hand law's mean while using up to 3.3× fewer synapses. And three independent lineages converged on the same signature: inhibition-rich (E/I gene pinned at its upper bound), feedforward cut, long-range skips kept, intra-layer recurrence selected out. Eleven numbers now specify a million-synapse brain better than we designed by hand. (Accuracies here use a smaller frozen readout budget than Result 5's 39.1% — compare within the table only.)

New best 42.5% — the ladder falls. Giving the best small-scale-selected genome the full Result-5 readout budget at 120k: 42.5% vs the prior 39.1%, at 44% fewer synapses — and all three selected genomes tested beat 39.1% (39.8%, 40.4%, 42.5%). The hand-designed law's control run on the same build seed seized (~100 Hz, 36.1%): its published best rode a favorable build, while the selected laws calibrate stably — and, being sparse and quiet, simulate ~40× faster. Still backprop-free end to end. Caveat: single build seed per arm.

Honest null: accuracy-wise, 12 generations of evolution ≈ random search — the gen-0 winner transfers as well as the evolved one. What evolution demonstrably bought is sparsity (19.9 → 5.9 synapses/neuron at matched accuracy). Selection screened the space of developmental laws; it didn't yet climb it. Bigger populations, wider gene bounds (the E/I pin), and structural genes — evolving the program, not its parameters — are the next rungs.

Result 7 — structural evolution, measured generation, and the ceiling that moved

Three campaigns ran on top of Result 6, every protocol pre-registered before results (full logs in EXPERIMENT.md §v14–v16):

Structural evolution (v14). Layer count entered the genome (2–6) with widened bounds. All three independent lineages chose depth 2 over the hand-designed 4 — and with depth free, Result 6's "inhibition-rich" signature reversed (E/I fell to 0.05–0.22): it had been compensation for imposed depth, not a universal law. A caution for reading any evolved genome: signatures are relative to the constraint set. The depth-2 laws win at held-out 120k in all seeds; the sparsest computes at 2.9 synapses/neuron (~350k synapses, 6.8× fewer than the hand law) and still beats it.

Generation, measured (v15). Generated text scored in bits/char under a 1–5 gram corpus model (real text 2.52, uniform noise 8.25). First finding: Result 5's "gibberish" verdict conflated two things — a decoder artifact (ridge scores softmaxed at T≈1 sample near-uniformly) with genuine incoherence. Decoded properly, generation shows non-random local English structure at 4.38 bpc — but it remains incoherent and operationally unusable. Rising teacher-forced accuracy has not produced usable generation. Closed-form scheduled sampling (exposure-bias repair without backprop) was null: the teacher-forced ↔ generation gap is a memory gap, as pre-registered. Caveat logged: greedy decoding games the metric (1.93 bpc by looping "the in the come") — bpc needs a diversity companion.

The ceiling moved (v15/v16a). The scaling sweep showed accuracy saturating in fit-data at fixed readout size but re-opening when taps × data grow jointly — the binding constraint was samples-per-parameter, not the linear form. And nonlinear features beat size at matched parameters: char-gated bilinear taps (reservoir traces read differently per current character) are the best mechanism tested.

readout (all on the Result-6 genome @120k)	trained params	accuracy
1024 rate taps, 40k fit (Result 6 ladder best)	79k	42.5%
4096 rate taps, 80k fit	274k	45.6%
2048 taps + char-gated features, 80k fit	277k	45.8%
4096 taps + char-gated features, 80k fit	407k	47.2%
multi-τ trace banks, fair sample budget	211k	42.2% (null)
char transformer reference (~10.7M params, backprop)	10,700k	≈58%

New best 47.2% — the session arc is 39.1 → 42.5 → 45.8 → 47.2, still backprop-free, still ~39× fewer trained parameters than the transformer reference. The multi-τ null is the honest signpost: slow reservoir traces add nothing even with ample data, so the remaining gap is long-context memory — which neither more taps nor more data will close.

Result 8 — the deep levers, honestly closed

The two pre-registered deep levers both resolved negative, each with a proper control:

Forward-Forward, done properly (v16c). Standardized inputs, three 512-unit layers with inter-layer L2 normalization, label embedding, hard negatives — every ingredient the three earlier casual failures lacked. Result: 11.5% (depth 3), 16.9% (depth 1) vs 42.5% for closed-form ridge on identical features. Depth made FF worse. Four attempts, one rigorous: goodness-based FF classification is mismatched to dense 65-way reservoir readout in our hands.

Three-factor plasticity (v16d). Reward-modulated STDP (local eligibility × global correctness, no gradients) reshaped 669k of the substrate's 1.34M synapses by ~10% over 60k characters — perfectly stably — and the frozen readout moved 41.5% vs 41.1% control: a null inside noise. The substrate is remarkably robust to reward-gated perturbation, and entirely unsculpted by it.

The v16 ledger: richer features +4.0pp (the 47.2% best) · FF depth negative · substrate plasticity null. What remains between 47.2% and the transformer's ≈58% is long-context memory — and no readout-side lever touches it. The substrate evolution keeps deleting recurrence because next-char prediction never rewards memory; the mechanism-backed next step is selecting substrates on memory-rewarding task families first, then reading language out of what evolution builds.

Result 9 — memory-selected substrates: stored in flight, and not enough

v17 made memory the fitness itself: delayed recall and temporal parity, with a taps-only readout so the reservoir is the sole channel — the tasks are unsolvable without it. Three seeds, pre-registered predictions, language transfer as the payoff test.

Memory is stored in flight. Even when memory is the fitness, evolution still rejects intra-layer recurrence (rec fan 1–2) — all three lineages instead pinned inter-layer feedback loops high with stretched delays: information survives as spikes circulating between layers, not as attractor state. The dynamic-systems track's evolved parity law found the same trick in exact circuits (registers holding sliding partial results). Two substrates, one physical idea: delay-line memory. The winners hold recall-4 at ~2.8× chance and 95% parity at held-out scales — and, for the first time in this whole program, iterated evolution beat gen-0 screening in all three seeds: memory landscapes have gradient where next-char never did.

The payoff null: memory-selected substrates are worse at language (38.4–40.8% vs 41.5–42.8% for the prediction-selected winner). Noisy 4-char recall cannot beat the readout's perfect explicit 3-char context; useful gains live at ranges one-hots don't reach, which demands memory far more reliable than these substrates offer. So the 47.2% → 58% gap is not "add memory" — it is "add reliable long-range memory, or longer explicit context." The dynamic-systems track's FSM-exact registers mark exactly how reliable.

What is actually claimed — and the reframing

Precision about learning types, so "backprop-free" is not over-read: development/STDP is local forward-time learning; the ridge readout is global supervised closed-form optimization; genome evolution is global black-box optimization; the full system is a backprop-free hybrid — not end-to-end forward-only learning. The demonstrated claim is "a developmental spiking reservoir provides useful features for next-character prediction under a supervised readout," not "an SNN learned language." And the campaign's honest summary: local prediction rewarded ever-stronger readouts while memory tasks shaped the substrate — neither produced reliable free-running language. Sequence stability under self-driven rollouts, not teacher-forced accuracy, is the unresolved problem, and it is now the primary benchmark (controlled generative grammars, validity horizons, error recovery — see EXPERIMENT.md §v18). A matched feature-source ablation (aligned traces vs shuffled vs neuron-free delay-line bank vs identical-topology non-spiking reservoir) is running to establish what the spiking substrate actually contributes.

Where this sideline goes

The two mechanism-backed levers left: longer explicit context in the readout (wider one-hot windows, closed-form as ever), and hybrid substrates — the spiking medium carrying an exact-state backbone, importing the dynamic-systems track's registers into the organism. Plus stacked closed-form fits, word-level targets, and the same error-driven mechanisms pointed back at music: grow when dialogue relatedness drops.

Reproduce: cd snn && npm run experiment:language · npm run experiment:growth · npm run experiment:evolve (fetches the dataset on first run). Part of the morpho·snn research track. References: Hinton, The Forward-Forward Algorithm; Ghader et al., Backpropagation-free SNNs with FF; Maass et al., liquid state machines; Spyx and Training Deep SNNs mark the surrogate-gradient road we deliberately did not take here.

AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.