PoC findings: spurious forgetting
What the paper researches. When a language model is finetuned on new facts, its accuracy on old facts can crash, then partly come back, then slowly fade again. The paper explains this with a miniature 'associative memory' model: finetuning shifts a shared internal direction, the shift later fades, and fact-specific drift slowly builds up.
What we found. The miniature model reproduces the pattern (old-fact recall: 0.16 at the low point, 0.70 after recovery, 0.50 at the end, 10 seeds), and each of its three ingredients is needed. One claim did not reproduce: removing the shared direction helped by about 0.11 rather than not at all, at every pretraining length tested. The headline numbers depend on an unstated training detail (final recall 0.11 to 0.61). The paper's language-model results were not tested.
Why it matters. It says a model that seems to have 'forgotten' something may only have hidden it, which changes what you do when a finetuned model gets worse at old tasks.
How industry can use it.
- Before rolling back or retraining a finetuned model that fails old tasks, test whether the right answer is still ranked first among its own group of answers (hidden) or really gone (erased).
- Evaluate at several checkpoints, not one: in this model old-fact accuracy crashes and recovers, so a single snapshot can mislead.
- Replaying a handful of old examples lifted the low point from 0.16 to 0.60 in the toy; worth trying as a cheap first remedy, with the caveat that this was a toy.
What this does not show. It does not show that the effect behaves the same way in large real models; that part of the paper was not tested here.
Paper: arXiv 2610.08718, "When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting". Experiment run 2026-10-07. Pure numpy, CPU, about 20 minutes for everything.
What was run
The paper's appendix-B.1 setup: keys √α·μ + √(1−α)·e (unit norm, pairwise overlap ≈ α, μ shared), a two-layer network with RMS-norm, softmax readout, d=128, V=32 tokens split in two halves, 128 facts in each of A, B, B′, batches of 32, α=0.5. Pretrain on A and B, finetune only on B′ (values in B's half), SGD, 5,000 steps, 10 seeds. Gradients were hand-derived and checked against finite differences (error < 1e-10). Code: pocs/2610.08718/.
Results vs the paper
| Claim | Paper | Here (mean ± sd, 10 seeds) | |
|---|---|---|---|
| Baseline: collapse → recovery → erosion | trough 0.17; final 0.51–0.54 | trough 0.16 ± 0.06; recovery peak 0.70; final 0.50 ± 0.06 | reproduced (calibrated, see below) |
| Less key sharing (α) weakens collapse, removes recovery | qualitative | α=0.1: trough 0.44, no recovery. α=0: 0.63, no recovery | reproduced |
| New values from whole vocab → only gradual forgetting | qualitative | no collapse; 0.75 final | reproduced |
| ReLU instead of RMS-norm → collapse, never recovers | qualitative | old-fact recall falls to 0.00 and stays | reproduced |
| Collapse hides facts, doesn't scramble them | 86% rank 1st in own half at trough (transformer) | 98% at trough (minimal model) | reproduced |
| Project out shared direction: no collapse | collapse prevented | trough 0.62 (no collapse) | reproduced |
| Project out shared direction: final accuracy | 0.45 vs 0.51, no better | 0.62 ± 0.05 vs 0.50 ± 0.06, better by 0.12 | not reproduced |
| Replay 16 old facts (non-replayed measured) | trough 0.17 → 0.50; final 0.54 → 0.62 | trough 0.16 → 0.60; final 0.50 → 0.60 | reproduced (trough lift larger here) |
Explainer
Illustrates the paper's claimed mechanism, not our result.
Ablations
Mechanism and interventions
Middle panel: the mean hidden-state shift of old facts peaks at 0.79 around step 75 and decays to 0.20, while the fact-specific part grows from 0 to 0.31. That is the paper's story, a shared shift that is withdrawn while per-fact drift accumulates.
Caveats
- Calibration. My first baseline did not recover (trough 0.09, peak 0.21, final 0.11). Recovery depends strongly on how long pretraining runs past the 0.99-accuracy threshold, which the paper does not specify. I added 2,000 extra pretraining steps because that matched the paper's baseline numbers. So the baseline match is calibrated, not independent evidence. The qualitative ablation results were not tuned. The un-extended result is in
results.jsonasbaseline_no_extra_pretrain. - The projection discrepancy may come from that same unspecified detail, or from the learning rate or step count. I did not investigate it further, so I can't say whether the paper's statement is wrong for this model or my setup differs. (Update 2026-10-09: now investigated; see the follow-up section below.)
- Finetuning learning rate (0.02) was set so the new facts reach 99% around step 200–250, as the paper describes, and reused for all configs.
- Only the minimal model was tested. The paper's claims about a controlled transformer and a pretrained language model, and the tiny-LoRA test I had planned, are untested.
- Replay here adds 16 old-fact samples to each batch of 32 new ones, so the replay configuration trains on larger batches.
Follow-up (2026-10-09): why doesn't the projection result reproduce?
Three candidate explanations, each tested before looking at the outcome:
| Candidate | Test | Result |
|---|---|---|
| Our projection differs from the paper's | Compared with the paper's appendix B.13: ΔW₁ ← ΔW₁(I − μμᵀ) at every step | Identical operation (we remove the μ-component of the W₁ gradient). Ruled out. |
| The unstated extra-pretraining length | Extra pretraining E = 0 to 8,000 steps, 10 seeds | Projection beats baseline by +0.06 at E=0 and by 0.10–0.12 at every E ≥ 500. Ruled out. |
| Learning rate calibrated per condition (the paper picks it "so that the new facts reach the same accuracy around step 200") | Six finetuning rates, 0.01–0.08, both methods, 10 seeds; time for new facts to reach 99% | Projection learns the new facts faster (175 vs 220 steps at 0.02), so a timing-matched rate would be lower and help projection more. Projection beats baseline on 10/10 seeds at every rate (+0.13 at 0.01 down to +0.075 at 0.08). Ruled out. |
Old-fact recall at the end falls with the learning rate for both methods (baseline 0.56 → 0.39, projection 0.69 → 0.46). The paper's pair of numbers (0.45 vs 0.51) appears in our model only if projection runs at 4× the baseline's rate (0.46 at 0.08 vs 0.50 at 0.02). The paper does not say it did that, so this is a coincidence we can name, not an explanation. The non-reproduction stands. In this model, removing the shared direction leaves old facts better off at every setting we tried. What remains untested: the pretraining learning rate, batch composition during finetuning, and anything about the paper's code that its text does not describe. Code: pocs/2610.08718/lr_calibration.py, results_lr_calibration.json.
About this analysis
Done by Claude (an AI model by Anthropic), directed by vishwasaidev-dev. Claude can make mistakes, including implementation errors, misreading the paper, or over-reading the statistics from 10 seeds. If you think something here is wrong, please open an issue. Issues are picked up in the next working session (nothing watches GitHub automatically), and corrections are made in the open with the change noted below.
Corrections log
- 2026-10-09: added the follow-up section on the projection discrepancy (pretraining-length and learning-rate tests). No earlier number was changed; the caveat that it was uninvestigated now points to the follow-up.