Absent-token forgetting: does a small explainable model predict a real language model?
What the paper researches. Where does catastrophic forgetting live when a language model is finetuned on a narrow dataset, and why? The paper says it concentrates in the output weights of words the new data never uses, blames the Adam optimizer's normalisation, and proposes a one-line fix: raise Adam's epsilon on the output layer only.
What we found. The mechanism shows up in a toy and in a small real model: Adam pushes probability off the absent words (absent-set mass 0.293 to 0.267) while SGD does not (0.292). The epsilon fix cut forgetting by 57% when only the output layer trained but gave nothing beyond a lower learning rate when more layers trained. Freezing the absent words' weights stopped the new learning. Which optimizer forgets more is not established.
Why it matters. Finetuning general models on private data is routine, and the usual defence (mixing original data back in) is often unavailable. A cheap, data-free defence would matter, and so would knowing when it applies.
How industry can use it.
- Measure forgetting by comparing the model's output distributions before and after finetuning (KL on its own samples, split by word group); plain average loss hid the effect here. No original training data is needed.
- Try the output-layer epsilon only when you adapt the output layer alone (light adaptation); do not assume it helps deeper finetuning.
- Do not freeze the weights of words absent from your new data.
- Use small pilots to find where damage happens, not to pick the winning optimizer.
What this does not show. It does not show these effects hold for the large models companies deploy: the language-model runs are a 135-million-parameter model, a short run, one new dataset, three seeds. The paper's own large-scale results were not reproduced.
Tests claims from arXiv 2610.09835 (forgetting concentrates in the output rows of tokens absent from the new data; Adam's normalisation is the claimed cause; raising Adam's epsilon on the output head is the claimed fix), inside the minimal model of arXiv 2610.08718 and in SmolLM2-135M. Experiments 2026-10-08/09, CPU only.
- Transfers. Adam moves probability mass away from tokens that are absent from the new data; SGD does not. In SmolLM2 the probability on the absent set falls 0.293 → 0.267 under Adam and 0.293 → 0.292 under SGD (3 seeds). In the toy the same shows up as absent output rows drifting about as far as present rows (2.65 vs 3.09 in norm); in SmolLM2 the absent rows' Adam second-moment scale is ~100× smaller than present rows' yet they move comparably (a near-automatic property of Adam's per-parameter normalisation over 120 steps, so this is weak evidence on its own).
- Regime-specific. Raising Adam's epsilon on the output head cuts forgetting (KL) by 57% when only the head trains (finance gain 0.014 vs 0.013 nats). When the top two blocks also train, it buys nothing beyond what simply lowering the learning rate buys.
- Does not transfer. Freezing the absent rows removes 21% of forgetting in the toy with learning intact, but in the language model it destroys the new learning (gain 0.014 → 0.002 nats).
- Not established. Which optimizer forgets more. In the toy Adam forgets more at long horizons (1.7× at 1,000 steps, 2.2× at 5,000) but not clearly at the earliest step; the language-model runs are only 120 steps and a tiny target gain, where Adam is a tie or better than SGD. Whether the toy ranking appears in the language model at longer horizons was not tested.
What was run
Toy (pure numpy; the 2610.08718 associative memory: d=128, 32 values in two halves, old facts A and B, new facts B′ use only B's half, so A's value tokens are absent from finetuning; hand-derived gradients, finite-difference checked). SGD lr 0.02 vs Adam lr 7e-4, chosen so the new facts reach 99% at about the same step (~190–230); 5,000 steps; 10 seeds, replicated on 5 fresh seeds. Optimizers, per-block freezing, Adam epsilon on the output layer only, and "restore" probes that swap a pretrained block back in.
Language model. SmolLM2-135M-Instruct, finetuned on finance news (300 chunks of 128 tokens; a seed uses half). Forgetting is measured as the exact KL divergence KL(p0‖p1) between the original and finetuned next-token distributions over the full 49,152-token vocabulary, on the model's own samples (64 sequences at temperature 1) and on technical prose, 4,096 tokens each; target gain is the drop in NLL on held-out finance text. 120 steps, batches of 128 tokens, 3 seeds. Two regimes: head-only (output matrix trained, body frozen, features cached) and top-2 blocks + head (last two blocks, final norm and head trained from cached block inputs; recomputing the features reproduced the cached ones exactly). "Absent" = never a target in that seed's finetuning chunks (91% of the vocabulary; 29% of eval tokens).
Toy results
| Quantity | SGD | Adam |
|---|---|---|
| Old-fact recall after 5,000 steps (10 seeds; chance 0.03, untrained net 0.04) | 0.50 ± 0.06 | 0.15 ± 0.04 (fresh seeds: 0.57 vs 0.20) |
| Recall restored by swapping pretrained W1 back / only the absent rows of W2 / all of W2 | 1.00 / 0.60 / 0.90 | 1.00 / 0.46 / 0.78 |
| Adam with the first layer frozen (only output layer trains) | recall 0.03, at the untrained floor, restoring only the absent rows gives 0.91 | |
| Forgetting removed by freezing the absent rows / by Adam epsilon on the output layer (best) | 10% / – | 21% / 21% (16% on fresh seeds) |
| KL on old-fact keys, SGD → Adam, step 100 / 250 / 1,000 / 5,000 (10 seeds) | 2.61→3.18 (1.2×) / 0.90→1.10 (1.2×) / 0.96→1.60 (1.7×) / 1.35→3.01 (2.2×). At step 25 Adam is not worse per unit of learning (SGD 1.55 with new-fact accuracy 0.19; Adam 7e-4 0.90 with 0.14). KL is not monotone in time (collapse, then partial recovery). | |
Reading: with a trainable first layer the forgetting is sufficiently undone by restoring W1 (a 128×128 block, against a 16×128 slice for the absent rows, so the blocks are not equal in size); output-layer-only fixes cannot remove most of it. With the first layer frozen, the forgetting lives in the absent output rows, which is the paper's claim.
Language-model results (3 seeds; KL sd across seeds ≤ 0.0015)
Each point is a configuration (label = learning rate, or SGD's c in units of 1/E|h|²). Up and left is worse, down and right is better.
| Regime | Configuration | Finance gain | KL (own samples) | KL (prose) | Absent mass pre→post |
|---|---|---|---|---|---|
| head-only | SGD c=10 | 0.015 | 0.0090 | 0.0070 | 0.293→0.292 |
| head-only | SGD c=20 | 0.018 | 0.0194 | 0.0156 | 0.293→0.290 |
| head-only | Adam 3e-5 | 0.014 | 0.0075 | 0.0041 | 0.293→0.267 |
| head-only | Adam 5e-5 | 0.013 | 0.0178 | 0.0096 | 0.293→0.252 |
| head-only | Adam 1e-4 | –0.006 | 0.0527 | 0.0276 | 0.293→0.224 |
| head-only | Adam 3e-5 + head eps 1e-4 | 0.013 | 0.0032 | 0.0017 | 0.293→0.287 |
| head-only | Adam 3e-5, absent rows frozen | 0.002 | 0.0041 | 0.0018 | 0.293→0.307 |
| top-2 blocks + head | SGD c=10 / c=20 | 0.031 / 0.036 | 0.0186 / 0.0365 | 0.0135 / 0.0283 | unchanged |
| top-2 blocks + head | Adam 1e-5 / 3e-5 / 5e-5 | 0.017 / 0.035 / 0.042 | 0.0017 / 0.0107 / 0.0236 | 0.0010 / 0.0061 / 0.0138 | 0.292→0.290 / 0.280 / 0.273 |
| top-2 blocks + head | Adam 3e-5 + head eps 1e-4 | 0.031 | 0.0087 | 0.0047 | 0.292→0.292 |
| top-2 blocks + head | Adam 3e-5, body only / head only | 0.023 / 0.013 | 0.0042 / 0.0074 | 0.0022 / 0.0042 | 0.295 / 0.265 |
Head-only: at Adam's gain (0.014) SGD's KL, scaled by gain², would be about 0.0074, a tie with Adam's 0.0075 on own samples, and Adam is better on prose (0.0041 vs 0.0070). No Adam setting reaches SGD c=20's gain. With the top-2 blocks trainable, Adam's curve lies below SGD's for gains of about 0.031 and above; below that the SGD grid has no KL measurement. The head epsilon's effect with a trainable body is within what lowering the learning rate achieves (interpolating Adam 1e-5 and 3e-5 to the epsilon run's gain of 0.031 gives KL ≈ 0.008, against the epsilon run's 0.0087).
Calibration sensitivity of the toy baseline (2610.08718)
| Extra pretraining steps | 0 | 500 | 1,000 | 2,000 | 4,000 | 8,000 |
|---|---|---|---|---|---|---|
| Baseline final recall (10 seeds) | 0.11 | 0.34 | 0.43 | 0.50 | 0.57 | 0.61 |
| Project-out final recall | 0.17 | 0.44 | 0.54 | 0.62 | 0.68 | 0.73 |
| Gap (project − baseline) | +0.06 | +0.10 | +0.11 | +0.12 | +0.11 | +0.12 |
The baseline's headline number is a function of an unspecified detail (0.11–0.61), so matching the paper's 0.51 is a fit. The earlier PoC's "projection does not reproduce" holds at every length ≥ 500, so it is not an artefact of the 2,000-step choice; the projection operator itself (mine removes the input-direction component of the first-layer gradient) was not checked against the paper's.
What changed after review
- Net NLL was the wrong metric. My first language-model comparison used the net increase in NLL on the model's own samples. For on-policy samples its expectation is a KL, so it is non-negative, yet many runs showed negative values: its noise floor (≈ ±0.003) was as large as the effects I was comparing. Claims built on it ("Adam forgets less than SGD", "epsilon removes the damage") were withdrawn and the experiments re-run with exact KL.
- Toy-vs-LM ranking. I had said the optimizer ranking "does not transfer"; the reviewer pointed out the toy's earliest horizon was left out of my scoring and its 100–250-step points are mid-collapse (KL of 1–3 nats against the model's 0.003–0.04). Now: not established.
- "Adam dominates SGD" and "head epsilon caps at about 20% with a trainable body" were each narrowed after the full gain–KL frontier was recomputed (above).
- An earlier version of the Adam second-moment statistic was read without bias correction (3× too small); the 100× ratio between present and absent rows is unaffected.
Caveats
- 120 steps of 128 tokens and a finance gain of only 0.2–0.4% of the NLL: a linear-response regime, not sustained finetuning. One model (135M parameters), one new-data corpus, three seeds. Seeds change the data subset and batch order but not the pretrained model.
- Hyperparameters were chosen on the same evaluation data (the toy used separate calibration seeds). The SGD grid tops out at c=20; Adam's grid is coarse (1e-5, 3e-5, 5e-5, 1e-4). SGD uses one learning rate for body and head; a separate body rate was not tried.
- Own-sample KL is a distributional measure; it was not checked against loss on downstream tasks. Eval text for prose was Python's own documentation topics, a narrow "general" text.
- The toy is trained to interpolation on hard labels and measured by recall; the language model is measured by KL at a tiny gain. Differences between them could come from that mismatch, not only from model realism. The toy's own KL-versus-horizon table was added to narrow this, not to close it.
- The paper's own experiments (up to 12B parameters) were not reproduced; this tests whether its mechanism shows up in a toy and a 135M model.
- Restore probes (swapping a pretrained block back in) are confounded by co-adaptation in the language model, where restoring the body made the net result worse, so they were not used there.
Code: pocs/2610.09835/ (scripts contain local-path placeholders; see the README there). The separate BeFOND finding has its own correction on its page.
About this analysis
Done by Claude (an AI model by Anthropic), directed by vishwasaidev-dev. Claude can make mistakes, including implementation errors, misreading the papers, or over-reading small effects from a few seeds on small models. Two independent adversarial reviews by fresh Claude instances were run and their accepted findings are above; they are not human review. If you think something here is wrong, please open an issue. Issues are picked up in the next working session (nothing watches GitHub automatically), and corrections are made in the open with the change noted below.
Corrections log
None yet.