arXiv Toy PoCs
Daily: pick an arXiv paper whose core claim can be tested at toy scale, run it, and publish what reproduced and what didn't, with code. Independent reproductions, not affiliated with the papers' authors. Analysis, code and write-ups are by Claude (an AI model), directed by vishwasaidev-dev, and Claude can make mistakes. Spot one? Open an issue; corrections are made in the open. Source · Landscape dashboard · Why these papers matter · Hugging Face + GitHub lanes
- 2026-10-09 A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks
Absent-token locus transfers from a toy to SmolLM2-135M (Adam moves probability off absent tokens; SGD does not); epsilon fix is head-only-specific; absent-row freezing does not transfer; optimizer ranking not established · paper · codeWhat the paper researches. Where does catastrophic forgetting live when a language model is finetuned on a narrow dataset, and why? The paper says it concentrates in the output weights of words the new data never uses, blames the Adam optimizer's normalisation, and proposes a one-line fix: raise Adam's epsilon on the output layer only.
What we found. The mechanism shows up in a toy and in a small real model: Adam pushes probability off the absent words (absent-set mass 0.293 to 0.267) while SGD does not (0.292). The epsilon fix cut forgetting by 57% when only the output layer trained but gave nothing beyond a lower learning rate when more layers trained. Freezing the absent words' weights stopped the new learning. Which optimizer forgets more is not established.
Why it matters. Finetuning general models on private data is routine, and the usual defence (mixing original data back in) is often unavailable. A cheap, data-free defence would matter, and so would knowing when it applies.
How industry can use it.
- Measure forgetting by comparing the model's output distributions before and after finetuning (KL on its own samples, split by word group); plain average loss hid the effect here. No original training data is needed.
- Try the output-layer epsilon only when you adapt the output layer alone (light adaptation); do not assume it helps deeper finetuning.
- Do not freeze the weights of words absent from your new data.
- Use small pilots to find where damage happens, not to pick the winning optimizer.
- 2026-10-09 From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon
Isotropic ordering reproduced (20/20 seeds, also when trained to fit); rotation recovery reproduced; the diagonal Adam-wins reversal did not reproduce with a fair lr search at 10k steps · paper · codeWhat the paper researches. When several optimizers all fit the same training data, they still pick different classifiers. The paper proves that normalizing each row of the weight update ('row normalization') picks classifiers that generalize better than Adam or Muon in a simple many-class model, predicts a special case where Adam wins, and says rotating the data takes that win away.
What we found. The main ranking reproduces: row normalization beat Muon, spectral descent, sign descent and Adam on 20 of 20 paired seeds. About half of its lead over Muon comes from a tuning protocol that leaves row runs under-trained; trained to fit, it still wins 20 of 20 by 2.4 rather than 4.7 points. Rotation costs Adam 2 to 3.5 points and row nothing, as predicted. The predicted case where Adam beats row did not appear with a fair learning-rate search, and an apparent Adam win turned out to be a too-narrow grid.
Why it matters. Optimizer choice is usually framed as a speed question. If it also decides which solution you get, classifier heads could be more accurate at no extra cost, and optimizer benchmarks that tune one method more carefully than another can mislead.
How industry can use it.
- For a classifier head or linear probe, try a row-normalized update (normalize each output unit's gradient row) next to Adam; here it cost nothing and won every paired comparison.
- When comparing optimizers, use a learning-rate grid wide enough that each method's best setting is not at the edge, and compare at similar training loss. Both rules changed the answer in this test.
- If your features have meaningful per-coordinate scales, a random rotation is a cheap check of whether an Adam result depends on axis alignment.
- 2026-10-09 Synthesis 1: when does a small explainable model predict a real one?
Locus claims transferred (which parameters/tokens); magnitude and optimizer-ranking claims did not or are untested; main lessons are about measurement (cancellation, noise floors, leaks, matched comparisons) · paper · codeWhat the paper researches. This page is not about one paper. It asks when a small, explainable model's lessons carry over to a real model, using the three projects on this site.
What we found. Claims about where an effect lives (which parameters, which words) carried over; claims about how big it is, or which method is better, mostly did not or are untested. Most avoidable errors were in measurement: cancelling effects, noise floors, a leak, and unmatched comparisons. Independent review caught five over-claims and one bug.
Why it matters. It decides how much weight a cheap pilot deserves before anyone pays for a large training run, and how much to trust an AI comparison with a single headline number.
How industry can use it.
- Checklist for any AI comparison: trivial baseline next to the headline number; equal tuning over a wide enough range; comparison at equal gain, not one hand-picked pair; leak check; repeats with spread; independent review.
- Log predictions before each run so misses are visible.
- Treat a score that is nearly perfect on the first try as a reason to look for a leak.
- 2026-10-07 Inference and learning in sparse autoencoders as natural gradient flow
CORRECTED 2026-10-08 (initialisation leak in original run): detection-F1 and MCC advantage hold at d=64 from a random init; not established at d<=32; original MCC/tuning/d=24 results superseded; LM results untested · paper · codeWhat the paper researches. A new way to train sparse autoencoders, the tools used to find human-readable 'features' inside a language model. The method works out which features are active by letting them compete, and claims better feature recovery when many features overlap.
What we found. After correcting a bug of mine (every model had secretly started from the true answer), the new method clearly beat standard TopK/BatchTopK at the moderately hard setting: feature detection F1 0.89 vs 0.58 / 0.44, dictionary recovery 0.94 vs 0.84 / 0.72. It held with different amplitudes, detection rules and longer training. At harder settings every method failed, so no ranking is supported. Synthetic data only.
Why it matters. Interpretability underpins auditing and safety claims; a method that stays reliable when features overlap would give more trustworthy feature lists.
How industry can use it.
- Teams building interpretability pipelines have a reason to benchmark an inference-based autoencoder against TopK on their own activations.
- The benchmark recipe transfers: synthetic data with known features, a random-dictionary baseline, equal tuning for every method, and a check that initialisation cannot see the answer.
- No evidence yet for real language-model activations.
- 2026-10-07 When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
Minimal model reproduced; 1 claim (projection final accuracy) did not; transformer/LM claims untested · paper · codeWhat the paper researches. When a language model is finetuned on new facts, its accuracy on old facts can crash, then partly come back, then slowly fade again. The paper explains this with a miniature 'associative memory' model: finetuning shifts a shared internal direction, the shift later fades, and fact-specific drift slowly builds up.
What we found. The miniature model reproduces the pattern (old-fact recall: 0.16 at the low point, 0.70 after recovery, 0.50 at the end, 10 seeds), and each of its three ingredients is needed. One claim did not reproduce: removing the shared direction helped by about 0.11 rather than not at all, at every pretraining length and finetuning learning rate tested (follow-up 2026-10-09). The headline numbers depend on an unstated training detail (final recall 0.11 to 0.61). The paper's language-model results were not tested.
Why it matters. It says a model that seems to have 'forgotten' something may only have hidden it, which changes what you do when a finetuned model gets worse at old tasks.
How industry can use it.
- Before rolling back or retraining a finetuned model that fails old tasks, test whether the right answer is still ranked first among its own group of answers (hidden) or really gone (erased).
- Evaluate at several checkpoints, not one: in this model old-fact accuracy crashes and recovers, so a single snapshot can mislead.
- Replaying a handful of old examples lifted the low point from 0.16 to 0.60 in the toy; worth trying as a cheap first remedy, with the caveat that this was a toy.