the control that killed the result
We tested whether a model still works when its weights are crushed down to −1, 0 and +1, using recordings of real industrial pumps — some healthy, some faulty. The first answer was that crushing the weights made the model better. It reproduced across every seed. It was also completely worthless, and the thing that exposed it took four lines of code.
Why pumps
Everything else on this site has been measured on text. Text is a bad place to ask a hardware question, because nobody deploys a tiny language model into a factory. So we went to MIMII — a public dataset of industrial machine sound, recorded by Hitachi. We used the pump: 1,006 recordings of a healthy pump and 143 of a faulty one, ten seconds each, with factory noise mixed in at a realistic level.
The job is anomaly detection, and the setup matters: the model is only ever shown healthy sound. It learns to predict what the pump does next. When a machine starts to fail it stops sounding like what the model learned, the predictions get worse, and that error is the alarm. You never have to collect examples of the failure — which is the entire appeal, since failures are rare and you cannot ask a factory to break its pump for your training set.
Scores are AUC: the chance that a randomly chosen faulty clip is rated more anomalous than a randomly chosen healthy one. 0.5 is a coin flip. 1.0 is perfect. The paper that published the dataset reports around 0.85 for this pump using a neural autoencoder, which is the rough order to have in mind — though not a like-for-like target, since we use one machine unit and our own split rather than their protocol.
The result that looked good
Our model is the same recurrence the rest of this site keeps coming back to —
h ← a·h + b·u, a running state that decays a little and takes in a little,
8,384 parameters, no growing memory. Trained on healthy sound only. Then we took the
trained weights and forced every one of them to −1, 0 or +1, and scored it again.
| weights | AUC | gates per multiply |
|---|---|---|
| float32 | 0.806 | 305 |
| ternary {−1,0,+1} | 0.912 | 74 |
Ternary was better, by a lot, at a quarter of the hardware cost. Run over five different random seeds it won five times out of five. That is the kind of result that writes its own headline.
The four lines that killed it
Before scoring a trained model, we also scored it untrained — random weights, straight from initialisation, no learning at all. If a random model does just as well, then whatever is happening has nothing to do with training.
And separately, the dumbest detector we could think of. Average each recording down to a single 64-number spectrum — how much energy sits at each pitch, averaged over the whole ten seconds. Take the average of all the healthy ones. Score a new clip by how far its spectrum sits from that average. No model. No training. No time. Just a stored template and a distance.
| detector | AUC (5 seeds) | |
|---|---|---|
| no model: spectrum template | 0.983 ±0.001 | 64 stored numbers |
| no model: frame-to-frame change | 0.810 ±0.008 | |
| no model: loudness | 0.682 ±0.001 | |
| no model: texture | 0.513 ±0.008 | |
| untrained recurrence | 0.923 ±0.047 | random weights |
| trained recurrence, float32 | 0.796 ±0.007 | ← worse than random |
| trained recurrence, ternary | 0.964 ±0.011 |
Two things are wrong here at once. The untrained model (0.923) beats the trained one (0.796) — so training was actively destroying the score, and ternarising it was simply undoing some of the damage. "Ternary wins 5/5" was a win over a baseline we had broken ourselves. And nothing we built came near a stored template of 64 numbers, which scores 0.983 and beats the published autoencoder too.
So the task never tested what we thought. A faulty pump has a different spectral shape, permanently, throughout the recording — and that shows up in a ten-second average. Nothing has to model time at all. Worth noting the shortcut is not simply volume: average log-mel level reaches only 0.682, and raw waveform loudness 0.699.
Taking the shortcut away
If the giveaway is each clip's average spectrum, then subtract it. Every recording gets its own average removed, so all that survives is how the sound changes over time — which is exactly what a recurrence is supposed to be good at, and the only thing left to be good at.
| detector, shortcut removed | AUC |
|---|---|
| trained recurrence, float32 | 0.483 ±0.016 |
| trained recurrence, best ternary | 0.568 ±0.009 |
| untrained recurrence | 0.645 ±0.027 |
| trivial frame-to-frame change | 0.810 ±0.008 |
The trained model lands at 0.483 — a coin flip. Take away the static giveaway and this recurrence extracts essentially nothing usable from the timing of the sound, while a one-line statistic about how fast the spectrum wobbles gets 0.810. That is a clean negative result, and it is the honest ceiling on what our sequence model did here.
A trap worth recording. When we first removed the shortcut, we kept computing the template baseline anyway — and it still scored 0.892. It should have been exactly 0.5, because after subtracting each clip's own average, that average is zero. It was zero: about 5×10⁻⁷, the rounding error left behind by doing the subtraction in 32-bit floating point. A quantity that is mathematically zero was separating healthy pumps from broken ones, and not merely by leaking volume — its correlation with loudness was only 0.17. Benchmarks leak through cracks this small.
Asking the hardware question properly
The original question was never really about our model. It was: does arithmetic this cheap still detect a broken pump? We had been asking it of a model that did not work. So ask it of the detector that does.
That turns out to be the same shape as the hardware anyway. A stored template of 64 numbers, with a live measurement compared against it, is a crossbar — the template is the array of weights, the incoming sound is the signal passing through, and scoring a clip is one multiply-accumulate per pitch band. So we quantised both sides: the stored template, and the live measurement streaming in.
| stored template | float32 in | int8 in | ternary in | binary in | gates/MAC | template size |
|---|---|---|---|---|---|---|
| float32 | 0.983 | 0.983 | 0.989 | 0.939 | 305 | 2048 bits |
| int8 | 0.983 | 0.983 | 0.989 | 0.939 | 305 | 512 bits |
| ternary | 0.978 | 0.978 | 0.987 | 0.958 | 74 | 101 bits |
| binary | 0.960 | 0.960 | 0.964 | 0.944 | 9.5 | 64 bits |
Ternary costs nothing. Ternary weights against ternary inputs score 0.987 — a shade above the float32 reference of 0.983 — for 4.1× fewer gates, with the entire detector stored in 101 bits. Going all the way to binary costs 0.039 AUC (0.944) for 32× fewer gates and 64 bits: the whole model fits in one machine word. Both still beat the published autoencoder baseline of ~0.85, and both beat our trained recurrence's 0.796.
The gate counts are not estimates — they are measured by compiling the arithmetic in MorphoHDL and counting: 305 gates for an exact signed multiply-accumulate, 74 when the weight is ternary and the multiply collapses into an add or a subtract or nothing at all, 9.5 when both sides are one bit and the whole operation is an XNOR and a popcount.
What we actually learned
Two claims, kept carefully apart, because the difference between them is the whole point of this page.
Not supported: "ternary weights survive in our trained sequence model on a real deployment task." The model never worked. It lost to random initialisation and then to 64 stored numbers, and a comparison between two versions of a broken thing measures nothing.
Supported: on this task, ternary arithmetic costs essentially nothing — +0.004 AUC at 4.1× fewer gates — and binary costs 0.039 at 32× fewer gates, measured on the detector that genuinely wins, against a published baseline it genuinely beats.
The second claim is the one we wanted, and we only got to make it honestly because the first one collapsed. Every run in this experiment now prints the no-model baselines in the same table as the model, so a trained network that cannot beat 64 stored numbers can never again be written up as one that can.
MIMII: Purohit et al., MIMII Dataset: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection (arXiv:1909.09347), CC-BY-SA 4.0. Pump, machine id_00, +6 dB signal-to-noise. All figures are means over 5 seeds that move both the held-out split and the initialisation; the template is always built from training-set healthy clips only.
AI-generated research. The experiments, code and write-ups on this site were produced by AI agents (Claude and Codex) working semi-autonomously, directed at a high level by a human. Nothing here is peer reviewed and none of it is affiliated with an academic institution. Measurements are reported as taken — including the nulls, the caveats and the corrections — but the work has had no external review.