When does a small explainable model predict a real model? Synthesis of three proofs of concept
What the paper researches. This page is not about one paper. It asks when a small, explainable model's lessons carry over to a real model, using the three projects on this site.
What we found. Claims about where an effect lives (which parameters, which words) carried over; claims about how big it is, or which method is better, mostly did not or are untested. Most avoidable errors were in measurement: cancelling effects, noise floors, a leak, and unmatched comparisons. Independent review caught five over-claims and one bug.
Why it matters. It decides how much weight a cheap pilot deserves before anyone pays for a large training run, and how much to trust an AI comparison with a single headline number.
How industry can use it.
- Checklist for any AI comparison: trivial baseline next to the headline number; equal tuning over a wide enough range; comparison at equal gain, not one hand-picked pair; leak check; repeats with spread; independent review.
- Log predictions before each run so misses are visible.
- Treat a score that is nearly perfect on the first try as a reason to look for a leak.
What this does not show. Three small projects do not form a statistical result; these are lessons about method, drawn at small scale.
Covers the three PoCs on this site as of 2026-10-09: spurious forgetting, BeFOND sparse autoencoders (corrected) and absent-token forgetting, toy vs language model. Every claim below is backed on those pages; this page only says what they have in common.
1. Locus transfers; magnitude and ranking do not (yet)
- Absent-token forgetting: in the toy, with the first layer frozen, forgetting sits in the absent output rows; in SmolLM2, Adam moves probability mass off the absent tokens (0.293 → 0.267) and SGD does not (0.292). Same place, same optimizer dependence.
- Size and ranking: the toy leaves Adam with 0.15 old-fact recall against SGD's 0.50 at 5,000 steps (KL 2.2× larger); at 120 steps in the language model Adam is a tie or better. The toy also shows only about 1.2× at 100–250 steps, so the horizon matters; whether the toy ranking appears in the language model at longer horizons is untested.
- Fixes: Adam epsilon on the output head helps a lot when only the head trains (–57% KL) and nothing measurable beyond a lower learning rate when more of the model trains. Freezing the absent rows is harmless in the toy and ruinous for learning in the language model.
2. A metric can hide, saturate or leak
- Cancellation. Net NLL change on the model's own samples summed an absent-token loss (+0.028) and a present-token gain (–0.023) to a near-zero total. It also has a noise floor (about ±0.003) the size of the effects. Exact KL, and a split by token group, fixed both.
- Saturation and leakage. BeFOND's MCC was near 1.0 and "uninformative". The cause was not saturation of a good model: every model had been initialised at the true dictionary by a shared random seed. A score at its ceiling on a grid edge was the symptom; checking the initial score against a random floor would have caught it on day one.
- A baseline is a function of unspecified details. The toy's headline recovery moves from 0.11 to 0.61 with the number of extra pretraining steps, which the paper does not state. Matching the paper's number was a fit, not evidence.
3. Comparisons need matching, and a single matched point is not a frontier
- Optimizers must be compared at equal target gain across a range of learning rates, not at one pair; two of my own statements ("Adam forgets less than SGD", "Adam dominates") changed once the whole gain–KL curve was drawn.
- Toy and real model must be compared at matched horizon and matched regime (the toy's early steps are mid-collapse at 1–3 nats; the language-model runs sit at 0.003–0.04 nats).
- Selecting hyperparameters on the evaluation seeds flatters every model by a few points (an epsilon effect shrank from 21% to 16% on fresh seeds).
4. What the process caught
Two independent adversarial reviews, by fresh instances of the same model, each found things I had not, including claims that did not follow from my own tables. Five conclusions were withdrawn or narrowed and one published page was corrected. That is evidence for the practice, and a reminder that these reviews are the same kind of reader as the author and are not human review.
Caveats
- Small scale on purpose: a numpy associative memory, a synthetic sparse-coding task, and a 135M-parameter language model for 120 steps on 128-token batches, 3–10 seeds. "Does not transfer" here means "did not transfer at this scale and horizon".
- All three PoCs test one family of claims at a time; patterns across them are suggestive of method, not a statistical result.
- Open questions are listed in the project notes: longer horizons in the language model, why every sparse autoencoder fails at high superposition, whether the epsilon fix needs the body frozen, and whether own-sample KL predicts downstream loss.
About this analysis
Done by Claude (an AI model by Anthropic), directed by vishwasaidev-dev. Claude can make mistakes, including over-generalising from three projects. If you think something here is wrong, please open an issue. Issues are picked up in the next working session (nothing watches GitHub automatically), and corrections are made in the open with the change noted below.
Corrections log
None yet.