← all findings

PoC findings: spurious forgetting

Why this matters (plain-language summary; technical detail is below)

What the paper researches. When a language model is finetuned on new facts, its accuracy on old facts can crash, then partly come back, then slowly fade again. The paper explains this with a miniature 'associative memory' model: finetuning shifts a shared internal direction, the shift later fades, and fact-specific drift slowly builds up.

What we found. The miniature model reproduces the pattern (old-fact recall: 0.16 at the low point, 0.70 after recovery, 0.50 at the end, 10 seeds), and each of its three ingredients is needed. One claim did not reproduce: removing the shared direction helped by about 0.11 rather than not at all, at every pretraining length tested. The headline numbers depend on an unstated training detail (final recall 0.11 to 0.61). The paper's language-model results were not tested.

Why it matters. It says a model that seems to have 'forgotten' something may only have hidden it, which changes what you do when a finetuned model gets worse at old tasks.

How industry can use it.

What this does not show. It does not show that the effect behaves the same way in large real models; that part of the paper was not tested here.

Paper: arXiv 2610.08718, "When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting". Experiment run 2026-10-07. Pure numpy, CPU, about 20 minutes for everything.

Verdict. The paper's minimal associative-memory model reproduces: old facts collapse, partly recover, then erode, and each of the three ingredients is needed. The mechanism evidence (a shared shift that fades while fact-specific drift grows) also reproduces. One claim did not reproduce: the paper says projecting out the shared direction leaves final accuracy no better (0.45 vs 0.51); here it ended clearly better (0.62 vs 0.50). The paper's second half, a real transformer and language model, was not tested here.

What was run

The paper's appendix-B.1 setup: keys √α·μ + √(1−α)·e (unit norm, pairwise overlap ≈ α, μ shared), a two-layer network with RMS-norm, softmax readout, d=128, V=32 tokens split in two halves, 128 facts in each of A, B, B′, batches of 32, α=0.5. Pretrain on A and B, finetune only on B′ (values in B's half), SGD, 5,000 steps, 10 seeds. Gradients were hand-derived and checked against finite differences (error < 1e-10). Code: pocs/2610.08718/.

Results vs the paper

ClaimPaperHere (mean ± sd, 10 seeds)
Baseline: collapse → recovery → erosiontrough 0.17; final 0.51–0.54trough 0.16 ± 0.06; recovery peak 0.70; final 0.50 ± 0.06reproduced (calibrated, see below)
Less key sharing (α) weakens collapse, removes recoveryqualitativeα=0.1: trough 0.44, no recovery. α=0: 0.63, no recoveryreproduced
New values from whole vocab → only gradual forgettingqualitativeno collapse; 0.75 finalreproduced
ReLU instead of RMS-norm → collapse, never recoversqualitativeold-fact recall falls to 0.00 and staysreproduced
Collapse hides facts, doesn't scramble them86% rank 1st in own half at trough (transformer)98% at trough (minimal model)reproduced
Project out shared direction: no collapsecollapse preventedtrough 0.62 (no collapse)reproduced
Project out shared direction: final accuracy0.45 vs 0.51, no better0.62 ± 0.05 vs 0.50 ± 0.06, better by 0.12not reproduced
Replay 16 old facts (non-replayed measured)trough 0.17 → 0.50; final 0.54 → 0.62trough 0.16 → 0.60; final 0.50 → 0.60reproduced (trough lift larger here)

Explainer

Illustrates the paper's claimed mechanism, not our result.

Ablations

Ablations

Mechanism and interventions

Mechanism

Middle panel: the mean hidden-state shift of old facts peaks at 0.79 around step 75 and decays to 0.20, while the fact-specific part grows from 0 to 0.31. That is the paper's story, a shared shift that is withdrawn while per-fact drift accumulates.

Caveats

Follow-up (2026-10-09): why doesn't the projection result reproduce?

Three candidate explanations, each tested before looking at the outcome:

CandidateTestResult
Our projection differs from the paper'sCompared with the paper's appendix B.13: ΔW₁ ← ΔW₁(I − μμᵀ) at every stepIdentical operation (we remove the μ-component of the W₁ gradient). Ruled out.
The unstated extra-pretraining lengthExtra pretraining E = 0 to 8,000 steps, 10 seedsProjection beats baseline by +0.06 at E=0 and by 0.10–0.12 at every E ≥ 500. Ruled out.
Learning rate calibrated per condition (the paper picks it "so that the new facts reach the same accuracy around step 200")Six finetuning rates, 0.01–0.08, both methods, 10 seeds; time for new facts to reach 99%Projection learns the new facts faster (175 vs 220 steps at 0.02), so a timing-matched rate would be lower and help projection more. Projection beats baseline on 10/10 seeds at every rate (+0.13 at 0.01 down to +0.075 at 0.08). Ruled out.

Old-fact recall at the end falls with the learning rate for both methods (baseline 0.56 → 0.39, projection 0.69 → 0.46). The paper's pair of numbers (0.45 vs 0.51) appears in our model only if projection runs at 4× the baseline's rate (0.46 at 0.08 vs 0.50 at 0.02). The paper does not say it did that, so this is a coincidence we can name, not an explanation. The non-reproduction stands. In this model, removing the shared direction leaves old facts better off at every setting we tried. What remains untested: the pretraining learning rate, batch composition during finetuning, and anything about the paper's code that its text does not describe. Code: pocs/2610.08718/lr_calibration.py, results_lr_calibration.json.

About this analysis

Done by Claude (an AI model by Anthropic), directed by vishwasaidev-dev. Claude can make mistakes, including implementation errors, misreading the paper, or over-reading the statistics from 10 seeds. If you think something here is wrong, please open an issue. Issues are picked up in the next working session (nothing watches GitHub automatically), and corrections are made in the open with the change noted below.

Corrections log