Whitebox program · structural search & reuse

Overparameterization is a phase, not a property

Why do networks start big? We tested whether the value of a large network is in executing the computation or in discovering it — and found a lifecycle where a model is overconnected only briefly, only when it needs to be.

2026 · whitebox program oscillator + MLP toy studies structural search · reuse

A recurring intuition: brains overproduce connections early and prune them, and lottery tickets show a small trainable subnetwork hides inside a big one — so maybe the bigness is developmental scaffolding, not the mature machine. We took that literally and asked, on controlled toy tasks, a sharper question: is overparameterization useful for execution, or for discovery? The answer that emerged is a whole lifecycle.

01Discovery, not execution

On a task with a genuinely sparse solution buried among distractors, we compared four ways to find a small circuit at a fixed sparse budget: a fixed random subgraph, local prune-and-regrow (SET / RigL), and training a dense net then extracting a ticket. Dense discovery won — and the reason is mechanistic, not incidental. A local search can only evaluate the connections it currently holds; the candidates it lacks are untrained, so their structural credit is uninformative. It is blind to the very wiring it needs to find. A dense pass trains all candidates at once, so structural credit is clean over the whole space.

Overparameterization’s value here was as a structural search engine — simultaneous credit over all candidate connections — not as the substrate that runs the mature computation.

02But permanent density is unnecessary

If density is for discovery, you should only need it briefly. We ran a duty-cycle sweep: stay sparse, but spend a fraction of training in a short dense “scout” phase, then consolidate the useful wiring by necessity and go sparse again.

03Only where structure exists to find

Scouting cannot help if there is no privileged sparse circuit. We built a matched control: take the same features but pass them through a fixed dense random mixing, so information is smeared across all coordinates and no special subset exists. In the tested regimes, scouting helped where a privileged sparse solution existed, and provided no advantage in the distributed control — where a random sparse graph was already near-optimal (any coordinates carry signal), necessity scores went diffuse, and the consolidated masks varied wildly between seeds. That degeneracy is the fingerprint of computation that genuinely wants to stay distributed.

So scouting must be conditional: expand only when the current circuit is insufficient, and consolidate only what verifies.

04The lifecycle

Assembled into one online system: keep a bounded sparse core; for each task, test whether the core copes; if not, scout densely, verify on held-out data, and consolidate only verified structure into the fixed-capacity core (evicting the least-used).

Run over a task sequence with a distributed control inserted, one system produced:

Dense for only 17% of lifetime training, yet matching the dense-discovery accuracy (~0.951); a bounded 72-edge core (no bloat); and roughly half the tasks solved with no new scouting at all.

05A digital-developmental analogue

This does not model real synaptic pruning or myelination. But it is a clean digital analogue of the shape:

The twist that makes it more than “bigger networks search better”: the system does not stay overconnected. It becomes overconnected only when its existing structure is insufficient, and returns to sparse once it has consolidated what it found.

06What is and isn’t shown

The full lifecycle is demonstrated in the tested toy setting: deliberately constructed task families on small MLP and oscillator models. The architecture is demonstrated; its scalability and usefulness on realistic workloads are not yet. Every claim above holds in the tested regimes, single-run in places, and one honest edge remains: the bounded core still churns — it re-scouts and evicts on some tasks rather than settling into a stable shared circuit, because the capacity sat just below the union of what the tasks needed.

07The problem has changed

Earlier the hard question was how do we discover good circuitry? In this setting, that is largely answered. The new hard question is:

How should a finite structural memory decide what to retain, share, protect, and evict over a long lifetime? The interesting failure is no longer discovery — it is the churn of a bounded core under a stream of tasks.

Overparameterization need not be a permanent property of a model. In our experiments it was useful as a temporary structural-search state: briefly expand the candidate graph, causally verify useful structure, consolidate it into a bounded sparse core, and later reuse that circuitry. The remaining challenge is not discovery but lifelong structural memory management.

Part of the whitebox / MorphoHDL program. Toy studies on oscillator coupling graphs and small MLPs with distractor / mixed-input regimes. Methods: dense-vs-local structural discovery; duty-cycle dense scouting; necessity (held-out ablation) vs gradient-magnitude consolidation; a fixed-capacity online lifecycle with conditional scouting and usage-based eviction. Results are in-setting and, where noted, single-run; scaling to realistic workloads is future work.