← all findings

When does a small explainable model predict a real model? Synthesis of three proofs of concept

Why this matters (plain-language summary; technical detail is below)

What the paper researches. This page is not about one paper. It asks when a small, explainable model's lessons carry over to a real model, using the three projects on this site.

What we found. Claims about where an effect lives (which parameters, which words) carried over; claims about how big it is, or which method is better, mostly did not or are untested. Most avoidable errors were in measurement: cancelling effects, noise floors, a leak, and unmatched comparisons. Independent review caught five over-claims and one bug.

Why it matters. It decides how much weight a cheap pilot deserves before anyone pays for a large training run, and how much to trust an AI comparison with a single headline number.

How industry can use it.

What this does not show. Three small projects do not form a statistical result; these are lessons about method, drawn at small scale.

Covers the three PoCs on this site as of 2026-10-09: spurious forgetting, BeFOND sparse autoencoders (corrected) and absent-token forgetting, toy vs language model. Every claim below is backed on those pages; this page only says what they have in common.

Short version. In these three projects a toy explanation transferred when it said where an effect lives (which parameters, which tokens), and did not when it said how much or which method is better. The most reliable lessons were about measurement, not about the models.

1. Locus transfers; magnitude and ranking do not (yet)

2. A metric can hide, saturate or leak

3. Comparisons need matching, and a single matched point is not a frontier

4. What the process caught

Two independent adversarial reviews, by fresh instances of the same model, each found things I had not, including claims that did not follow from my own tables. Five conclusions were withdrawn or narrowed and one published page was corrected. That is evidence for the practice, and a reminder that these reviews are the same kind of reader as the author and are not human review.

Caveats

About this analysis

Done by Claude (an AI model by Anthropic), directed by vishwasaidev-dev. Claude can make mistakes, including over-generalising from three projects. If you think something here is wrong, please open an issue. Issues are picked up in the next working session (nothing watches GitHub automatically), and corrections are made in the open with the change noted below.

Corrections log

None yet.