← all findings

Absent-token forgetting: does a small explainable model predict a real language model?

Why this matters (plain-language summary; technical detail is below)

What the paper researches. Where does catastrophic forgetting live when a language model is finetuned on a narrow dataset, and why? The paper says it concentrates in the output weights of words the new data never uses, blames the Adam optimizer's normalisation, and proposes a one-line fix: raise Adam's epsilon on the output layer only.

What we found. The mechanism shows up in a toy and in a small real model: Adam pushes probability off the absent words (absent-set mass 0.293 to 0.267) while SGD does not (0.292). The epsilon fix cut forgetting by 57% when only the output layer trained but gave nothing beyond a lower learning rate when more layers trained. Freezing the absent words' weights stopped the new learning. Which optimizer forgets more is not established.

Why it matters. Finetuning general models on private data is routine, and the usual defence (mixing original data back in) is often unavailable. A cheap, data-free defence would matter, and so would knowing when it applies.

How industry can use it.

What this does not show. It does not show these effects hold for the large models companies deploy: the language-model runs are a 135-million-parameter model, a short run, one new dataset, three seeds. The paper's own large-scale results were not reproduced.

Tests claims from arXiv 2610.09835 (forgetting concentrates in the output rows of tokens absent from the new data; Adam's normalisation is the claimed cause; raising Adam's epsilon on the output head is the claimed fix), inside the minimal model of arXiv 2610.08718 and in SmolLM2-135M. Experiments 2026-10-08/09, CPU only.

Verdict: one part transfers, three parts do not, and one is not established. Several claims I made partway through were withdrawn or narrowed after two independent adversarial reviews; they are listed below so the changes are visible.

What was run

Toy (pure numpy; the 2610.08718 associative memory: d=128, 32 values in two halves, old facts A and B, new facts B′ use only B's half, so A's value tokens are absent from finetuning; hand-derived gradients, finite-difference checked). SGD lr 0.02 vs Adam lr 7e-4, chosen so the new facts reach 99% at about the same step (~190–230); 5,000 steps; 10 seeds, replicated on 5 fresh seeds. Optimizers, per-block freezing, Adam epsilon on the output layer only, and "restore" probes that swap a pretrained block back in.

Language model. SmolLM2-135M-Instruct, finetuned on finance news (300 chunks of 128 tokens; a seed uses half). Forgetting is measured as the exact KL divergence KL(p0‖p1) between the original and finetuned next-token distributions over the full 49,152-token vocabulary, on the model's own samples (64 sequences at temperature 1) and on technical prose, 4,096 tokens each; target gain is the drop in NLL on held-out finance text. 120 steps, batches of 128 tokens, 3 seeds. Two regimes: head-only (output matrix trained, body frozen, features cached) and top-2 blocks + head (last two blocks, final norm and head trained from cached block inputs; recomputing the features reproduced the cached ones exactly). "Absent" = never a target in that seed's finetuning chunks (91% of the vocabulary; 29% of eval tokens).

Toy results

QuantitySGDAdam
Old-fact recall after 5,000 steps (10 seeds; chance 0.03, untrained net 0.04)0.50 ± 0.060.15 ± 0.04 (fresh seeds: 0.57 vs 0.20)
Recall restored by swapping pretrained W1 back / only the absent rows of W2 / all of W21.00 / 0.60 / 0.901.00 / 0.46 / 0.78
Adam with the first layer frozen (only output layer trains)recall 0.03, at the untrained floor, restoring only the absent rows gives 0.91
Forgetting removed by freezing the absent rows / by Adam epsilon on the output layer (best)10% / –21% / 21% (16% on fresh seeds)
KL on old-fact keys, SGD → Adam, step 100 / 250 / 1,000 / 5,000 (10 seeds)2.61→3.18 (1.2×) / 0.90→1.10 (1.2×) / 0.96→1.60 (1.7×) / 1.35→3.01 (2.2×). At step 25 Adam is not worse per unit of learning (SGD 1.55 with new-fact accuracy 0.19; Adam 7e-4 0.90 with 0.14). KL is not monotone in time (collapse, then partial recovery).

Reading: with a trainable first layer the forgetting is sufficiently undone by restoring W1 (a 128×128 block, against a 16×128 slice for the absent rows, so the blocks are not equal in size); output-layer-only fixes cannot remove most of it. With the first layer frozen, the forgetting lives in the absent output rows, which is the paper's claim.

Language-model results (3 seeds; KL sd across seeds ≤ 0.0015)

Target gain versus forgetting (KL) for head-only and top-2-blocks training

Each point is a configuration (label = learning rate, or SGD's c in units of 1/E|h|²). Up and left is worse, down and right is better.

RegimeConfigurationFinance gainKL (own samples)KL (prose)Absent mass pre→post
head-onlySGD c=100.0150.00900.00700.293→0.292
head-onlySGD c=200.0180.01940.01560.293→0.290
head-onlyAdam 3e-50.0140.00750.00410.293→0.267
head-onlyAdam 5e-50.0130.01780.00960.293→0.252
head-onlyAdam 1e-4–0.0060.05270.02760.293→0.224
head-onlyAdam 3e-5 + head eps 1e-40.0130.00320.00170.293→0.287
head-onlyAdam 3e-5, absent rows frozen0.0020.00410.00180.293→0.307
top-2 blocks + headSGD c=10 / c=200.031 / 0.0360.0186 / 0.03650.0135 / 0.0283unchanged
top-2 blocks + headAdam 1e-5 / 3e-5 / 5e-50.017 / 0.035 / 0.0420.0017 / 0.0107 / 0.02360.0010 / 0.0061 / 0.01380.292→0.290 / 0.280 / 0.273
top-2 blocks + headAdam 3e-5 + head eps 1e-40.0310.00870.00470.292→0.292
top-2 blocks + headAdam 3e-5, body only / head only0.023 / 0.0130.0042 / 0.00740.0022 / 0.00420.295 / 0.265

Head-only: at Adam's gain (0.014) SGD's KL, scaled by gain², would be about 0.0074, a tie with Adam's 0.0075 on own samples, and Adam is better on prose (0.0041 vs 0.0070). No Adam setting reaches SGD c=20's gain. With the top-2 blocks trainable, Adam's curve lies below SGD's for gains of about 0.031 and above; below that the SGD grid has no KL measurement. The head epsilon's effect with a trainable body is within what lowering the learning rate achieves (interpolating Adam 1e-5 and 3e-5 to the epsilon run's gain of 0.031 gives KL ≈ 0.008, against the epsilon run's 0.0087).

Calibration sensitivity of the toy baseline (2610.08718)

Extra pretraining steps05001,0002,0004,0008,000
Baseline final recall (10 seeds)0.110.340.430.500.570.61
Project-out final recall0.170.440.540.620.680.73
Gap (project − baseline)+0.06+0.10+0.11+0.12+0.11+0.12

The baseline's headline number is a function of an unspecified detail (0.11–0.61), so matching the paper's 0.51 is a fit. The earlier PoC's "projection does not reproduce" holds at every length ≥ 500, so it is not an artefact of the 2,000-step choice; the projection operator itself (mine removes the input-direction component of the first-layer gradient) was not checked against the paper's.

What changed after review

Caveats

Code: pocs/2610.09835/ (scripts contain local-path placeholders; see the README there). The separate BeFOND finding has its own correction on its page.

About this analysis

Done by Claude (an AI model by Anthropic), directed by vishwasaidev-dev. Claude can make mistakes, including implementation errors, misreading the papers, or over-reading small effects from a few seeds on small models. Two independent adversarial reviews by fresh Claude instances were run and their accepted findings are above; they are not human review. If you think something here is wrong, please open an issue. Issues are picked up in the next working session (nothing watches GitHub automatically), and corrections are made in the open with the change noted below.

Corrections log

None yet.