← all findings

PoC findings: does row normalization pick better classifiers than Adam and Muon?

Why this matters (plain-language summary; technical detail is below)

What the paper researches. When several optimizers all fit the same training data, they still end up with different classifiers. The paper proves that normalizing each row of the weight update ("row normalization") picks classifiers that generalize better than Adam or Muon in a simple many-class model. It also predicts a special case where Adam wins, and says rotating the data takes that win away.

What we found. The main ranking reproduces cleanly: row normalization beat Muon, spectral descent, sign descent and Adam on 20 of 20 paired seeds. About half of its lead over Muon comes from the paper's tuning protocol, which leaves the row runs under-trained; trained to fit the data, row still wins 20 of 20, by 2.4 rather than 4.7 accuracy points. Rotating the data costs Adam 2.1 to 3.5 points and row nothing, as predicted. The predicted case where Adam beats row did not appear with a fair learning-rate search (two settings: row ahead on 9 of 12 seeds, and a tie on 6 of 12). An apparent 12-of-12 Adam win turned out to be a too-narrow learning-rate grid.

Why it matters. Optimizer choice is usually framed as a speed question. If it also decides which solution you get, then classifier heads and other last layers could be more accurate with no extra cost, and benchmarks that tune one optimizer more carefully than another can mislead.

How industry can use it.

What this does not show. Everything here is a linear classifier on synthetic Gaussian data with 60 classes in 20 dimensions. The paper's language-model experiment was not run, and our runs are 1% to 25% as long as the paper's.

Paper: arXiv 2610.11309, "From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon". Experiments run 2026-10-09. Pure numpy, CPU only (2 cores), about 6 CPU-hours in total.

Verdict. The isotropic ordering reproduces (row > spectral/Muon > sign/Adam, 20/20 seeds), including when the row runs are trained to fit the data. Rotation recovery reproduces: Adam loses accuracy under rotation and row normalization is exactly unchanged. The coordinate-aligned reversal (Adam beating row) did not reproduce in two diagonal spectra with a fair learning-rate search, at our 10,000-step horizon. The paper's reversal uses 1,000,000 steps and an unstated spectrum, so this is a non-reproduction at our scale, not a refutation.

What was run

A. Isotropic Gaussian clouds (the paper's Theorem 3.1 and appendix H.1 setup): k=60 classes, d=20, class means N(0, I/d), 1 to 5 training points per class, training noise 0.2236, test noise 2.2361, class proportions from Dirichlet(1). Bias-free linear classifier, zero initialisation, full-batch cross-entropy in float64, step size η₀/√(t+1) with η₀ = c/‖Δ₀‖. Six optimizers: row-normalized GD with and without momentum, spectral GD, exact-SVD Muon, sign GD, and Adam (β=(0.9, 0.999), no bias correction, ε=0, as in the paper). 20 paired seeds, 5,000 steps.

B. Coordinate-aligned reversal (Proposition 4.5, appendix G): one clean training point per class, class means with a diagonal power-law covariance γₐ ∝ a−s, test noise covariance (1−λ)I + λΓ. The paper's formula puts the switch from "row wins" to "Adam wins" at λ≈0.48 for s=1.25 and λ≈0.38 for s=2, at d=20. Rotated condition: training and test inputs multiplied by the same random rotation. RowGDM vs Adam, 12 paired seeds, 10,000 steps, both optimizers on one 7-point grid c ∈ {0.1 … 100}.

One improvement on the paper's protocol: the paper picks the learning rate on the same evaluation set it reports (it says so itself). We pick on a separate 40,000-example validation set and report on a fresh 40,000-example test set. The paper's way gives the same numbers to within 0.001 here (both are in the results files). Code: pocs/2610.11309/.

Results vs the paper

Left: test accuracy by optimizer over 20 paired seeds. Right: RowGDM minus Adam accuracy per seed for six reversal-test conditions.
ClaimPaperHere
Isotropic: row methods beat spectral GD and Muon30/30 seeds20/20 seeds, +4.7 points (0.328 vs 0.281)reproduced
Isotropic: row methods beat sign GD and Adam30/30 seeds20/20 seeds, +8.8 points (vs 0.238 / 0.240)reproduced
The same ordering for trained-to-fit row runs(the theorem is about the fitted limit)RowGD at training loss 3×10⁻⁴: 0.305; still 20/20 vs all four, +2.4 vs Muon, +6.5 vs Adamreproduced, smaller margin
Diagonal spectrum, λ=0.25: row beats Adam30/3012/12 for both spectra (+6.8 and +4.9 points)reproduced
Diagonal spectrum, λ=1: Adam beats row (reversal)30/30s=1.25: row ahead 9/12 (+0.4). s=2: tie, 6/12 (+0.04)not reproduced
Same, after a random rotation: row beats Adam again30/3012/12 for both spectra (+2.5, +3.5). Adam drops 2.1 / 3.5 points under rotation; row is exactly unchangedreproduced

Two things that changed the answer along the way

Under-training in the tuning protocol. With the paper's protocol (pick the learning rate with the best accuracy), the row runs choose small rates and stop at training loss ≈2.5, nowhere near fitting the data. The other optimizers reach losses between 3×10⁻⁴ and 0.2. Trained to fit (c=1, loss 3×10⁻⁴), RowGD drops from 0.328 to 0.305, and it was still falling with more steps (0.318, 0.311, 0.305 at 1,250 / 2,500 / 5,000 steps). The paper reports that its own row runs sometimes chose the bottom of its grid, so the same effect is probably in its figure. The ranking survives; the size of the lead over Muon halves. For reference, a plain nearest-centroid classifier scores 0.313.

A narrow learning-rate grid manufactured a reversal. With 4-point grids, the steeper spectrum (s=2) showed Adam beating row on 12/12 seeds by 2.5 points, which looked like the paper's prediction. The grid audit flagged the row runs at the edge of their grid. With one 7-point grid for both, row's best rate turned out to be about 10, above the old grid's top value of 3, and the gap closed to a tie. All superseded runs are kept in the results files and in summary.json under B_narrow_grid_superseded.

Audit

Every PoC now runs a floor-and-leak audit before its results are trusted (pocs/audit.py; added after the 2610.07389 initialisation leak). Here: the untrained classifier scores 0.000 (all scores tie) against a random-weights floor of 0.012 to 0.016; training on shuffled labels scores 0.013 to 0.024, below the 0.06 always-guess-the-biggest-class rate; the rotation uses its own random stream; training and test data come from one stream in independent draws (this PoC predates the audit's streams() helper). Grid-edge check on the final runs: no λ=1 selection is at a grid edge except row at s=2 (2 of 12 seeds at 100). At λ=0.25, row picks the grid's lowest rate on 5 to 7 of 12 seeds; lower rates were not tried, but they could only help row, which already wins 12/12. Full output in audit.json.

Explainer

Illustrates the paper's claimed mechanism, not our result.

Caveats

About this analysis

Done by Claude (an AI model by Anthropic), directed by vishwasaidev-dev. Claude can make mistakes, including implementation errors, misreading the paper, or over-reading the statistics from 12 to 20 seeds. If you think something here is wrong, please open an issue. Issues are picked up in the next working session (nothing watches GitHub automatically), and corrections are made in the open with the change noted below.

Corrections log

None yet.