PoC findings: does row normalization pick better classifiers than Adam and Muon?
What the paper researches. When several optimizers all fit the same training data, they still end up with different classifiers. The paper proves that normalizing each row of the weight update ("row normalization") picks classifiers that generalize better than Adam or Muon in a simple many-class model. It also predicts a special case where Adam wins, and says rotating the data takes that win away.
What we found. The main ranking reproduces cleanly: row normalization beat Muon, spectral descent, sign descent and Adam on 20 of 20 paired seeds. About half of its lead over Muon comes from the paper's tuning protocol, which leaves the row runs under-trained; trained to fit the data, row still wins 20 of 20, by 2.4 rather than 4.7 accuracy points. Rotating the data costs Adam 2.1 to 3.5 points and row nothing, as predicted. The predicted case where Adam beats row did not appear with a fair learning-rate search (two settings: row ahead on 9 of 12 seeds, and a tie on 6 of 12). An apparent 12-of-12 Adam win turned out to be a too-narrow learning-rate grid.
Why it matters. Optimizer choice is usually framed as a speed question. If it also decides which solution you get, then classifier heads and other last layers could be more accurate with no extra cost, and benchmarks that tune one optimizer more carefully than another can mislead.
How industry can use it.
- When training a classifier head or linear probe, try a row-normalized update (normalize each output unit's gradient row) alongside Adam. It cost nothing extra here and won every paired comparison.
- When comparing optimizers, give each one a learning-rate grid wide enough that its best setting is not at the edge, and check training loss. In this test both rules changed the answer.
- Adam's behaviour depends on how features line up with coordinate axes. If your features have meaningful per-coordinate scales, a random rotation is a cheap way to check whether Adam's result depends on that alignment.
What this does not show. Everything here is a linear classifier on synthetic Gaussian data with 60 classes in 20 dimensions. The paper's language-model experiment was not run, and our runs are 1% to 25% as long as the paper's.
Paper: arXiv 2610.11309, "From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon". Experiments run 2026-10-09. Pure numpy, CPU only (2 cores), about 6 CPU-hours in total.
What was run
A. Isotropic Gaussian clouds (the paper's Theorem 3.1 and appendix H.1 setup): k=60 classes, d=20, class means N(0, I/d), 1 to 5 training points per class, training noise 0.2236, test noise 2.2361, class proportions from Dirichlet(1). Bias-free linear classifier, zero initialisation, full-batch cross-entropy in float64, step size η₀/√(t+1) with η₀ = c/‖Δ₀‖. Six optimizers: row-normalized GD with and without momentum, spectral GD, exact-SVD Muon, sign GD, and Adam (β=(0.9, 0.999), no bias correction, ε=0, as in the paper). 20 paired seeds, 5,000 steps.
B. Coordinate-aligned reversal (Proposition 4.5, appendix G): one clean training point per class, class means with a diagonal power-law covariance γₐ ∝ a−s, test noise covariance (1−λ)I + λΓ. The paper's formula puts the switch from "row wins" to "Adam wins" at λ≈0.48 for s=1.25 and λ≈0.38 for s=2, at d=20. Rotated condition: training and test inputs multiplied by the same random rotation. RowGDM vs Adam, 12 paired seeds, 10,000 steps, both optimizers on one 7-point grid c ∈ {0.1 … 100}.
One improvement on the paper's protocol: the paper picks the learning rate on the same evaluation set it reports (it says so itself). We pick on a separate 40,000-example validation set and report on a fresh 40,000-example test set. The paper's way gives the same numbers to within 0.001 here (both are in the results files). Code: pocs/2610.11309/.
Results vs the paper
| Claim | Paper | Here | |
|---|---|---|---|
| Isotropic: row methods beat spectral GD and Muon | 30/30 seeds | 20/20 seeds, +4.7 points (0.328 vs 0.281) | reproduced |
| Isotropic: row methods beat sign GD and Adam | 30/30 seeds | 20/20 seeds, +8.8 points (vs 0.238 / 0.240) | reproduced |
| The same ordering for trained-to-fit row runs | (the theorem is about the fitted limit) | RowGD at training loss 3×10⁻⁴: 0.305; still 20/20 vs all four, +2.4 vs Muon, +6.5 vs Adam | reproduced, smaller margin |
| Diagonal spectrum, λ=0.25: row beats Adam | 30/30 | 12/12 for both spectra (+6.8 and +4.9 points) | reproduced |
| Diagonal spectrum, λ=1: Adam beats row (reversal) | 30/30 | s=1.25: row ahead 9/12 (+0.4). s=2: tie, 6/12 (+0.04) | not reproduced |
| Same, after a random rotation: row beats Adam again | 30/30 | 12/12 for both spectra (+2.5, +3.5). Adam drops 2.1 / 3.5 points under rotation; row is exactly unchanged | reproduced |
Two things that changed the answer along the way
Under-training in the tuning protocol. With the paper's protocol (pick the learning rate with the best accuracy), the row runs choose small rates and stop at training loss ≈2.5, nowhere near fitting the data. The other optimizers reach losses between 3×10⁻⁴ and 0.2. Trained to fit (c=1, loss 3×10⁻⁴), RowGD drops from 0.328 to 0.305, and it was still falling with more steps (0.318, 0.311, 0.305 at 1,250 / 2,500 / 5,000 steps). The paper reports that its own row runs sometimes chose the bottom of its grid, so the same effect is probably in its figure. The ranking survives; the size of the lead over Muon halves. For reference, a plain nearest-centroid classifier scores 0.313.
A narrow learning-rate grid manufactured a reversal. With 4-point grids, the steeper spectrum (s=2) showed Adam beating row on 12/12 seeds by 2.5 points, which looked like the paper's prediction. The grid audit flagged the row runs at the edge of their grid. With one 7-point grid for both, row's best rate turned out to be about 10, above the old grid's top value of 3, and the gap closed to a tie. All superseded runs are kept in the results files and in summary.json under B_narrow_grid_superseded.
Audit
Every PoC now runs a floor-and-leak audit before its results are trusted (pocs/audit.py; added after the 2610.07389 initialisation leak). Here: the untrained classifier scores 0.000 (all scores tie) against a random-weights floor of 0.012 to 0.016; training on shuffled labels scores 0.013 to 0.024, below the 0.06 always-guess-the-biggest-class rate; the rotation uses its own random stream; training and test data come from one stream in independent draws (this PoC predates the audit's streams() helper). Grid-edge check on the final runs: no λ=1 selection is at a grid edge except row at s=2 (2 of 12 seeds at 100). At λ=0.25, row picks the grid's lowest rate on 5 to 7 of 12 seeds; lower rates were not tried, but they could only help row, which already wins 12/12. Full output in audit.json.
Explainer
Illustrates the paper's claimed mechanism, not our result.
Caveats
- Shorter runs and fewer seeds than the paper (CPU budget): 5,000 steps vs 20,000 for A; 10,000 to 30,000 vs 1,000,000 for B; 20 and 12 seeds vs 30. At λ=1 the trends over training run against the reversal (row 0.196 → 0.221 from 2,000 to 30,000 steps, Adam 0.218 → 0.213), but whether they cross by 10⁶ steps was not tested.
- The reversal spectrum is not stated in the paper. We tried two power laws chosen so the paper's own formula predicts a reversal at λ=1. A spectrum we did not try may show it. The formula is asymptotic in d, and d=20 is small.
- Learning-rate grids: A used 4-point grids per optimizer, chosen from pilot runs (0.01 to 0.3 for row, 0.3 to 10 for the others), plus a separate 1 to 30 sweep for row. The paper used one 7-point grid for all.
- Not tested: the paper's random-reshuffling (mini-batch) results, the independent-orientation anisotropic theorem, and the LLaMA-60M last-layer experiment.
About this analysis
Done by Claude (an AI model by Anthropic), directed by vishwasaidev-dev. Claude can make mistakes, including implementation errors, misreading the paper, or over-reading the statistics from 12 to 20 seeds. If you think something here is wrong, please open an issue. Issues are picked up in the next working session (nothing watches GitHub automatically), and corrections are made in the open with the change noted below.
Corrections log
None yet.