- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper asks whether a NN can continue learning useful features even after its training loss stops improving. It looks at a simpler version of matrix Muon, that has exact-polar updates and no momentum used for optimization of their model.
The authors use a teacher-student regime. The teacher generates synthetic labels while a student is trained to mimic this. They then track training loss and use AGOP to measure alignment with the teacher’s input directions, finding that alignment improves even while loss keep oscillating.
The authors support this observation with exact results for (simpler) linear models and a mean field theorem proving feature alignment during a plateau. These results provide evidence for the proposed phenomenon, i.e. that the loss and feature alignment can improve on different scale. However, they do not prove the oscillating behavior observed in the nonlinear experiments (which is also the most interesting part in my opinion).
Critical strengths
Across many synthetic teacher models, AGOP alignment continnues to improve during oscillating loss plateaus. I think this is the paper’s main contribution.
The linear models give exact examples where alignment improves althought we have a nonzero loss floor. The mean field result also shows improving alignment during a plateau in a nonlinear setting.
Critical weaknesses
As mentioned above, the linear models show that a loss floor can coexist with alignment improvement, it is unclear how they relate to the nonlinear experiments. The mean-field theorem is a different setting separate mechanism and does not establish the oscillations we see in the experiments. I think the paper's theoretical explanation is quite narrow This makes the paper’s theoretical explanation of the main phenomenon narrow as none explain the observed phenomenon.
The strongest evidence comes from synthetic teacher student tasks with full-batch exact-polar updates and a fixed output head (Figs 2 and 4). The main deep NN results also use a large first-layer step and much smaller steps in later layers. Whether loss stays flat while alignment improves depends on how the later layers are trained (Figs 5 and 14), and this effect is sensitive to how Muon is implemented (Fig 18). To me, this makes it unclear how commonly the phenomenon occurs in ordinary NN training.
Questions for authors
-
What happens if you also train the output weights in the one-hidden-layer experiments? Does alignment still improve while the average loss stays roughly flat?
-
In Fig 5, how can we be sure that training the later layers lets the network use features learned during the plateau and not simply allowing more weights to change which in turn reduce the loss?
-
Which parts of the theory do you expect to carry over to the finite width nonlinear setting, and what evidence supports that connection?
5 comments
Response to Reviewer zn1g, part 2 of 4: W2 (continued), Q1, Q2
Continuation of our response.
W2, continued.
- Where the effect survives. In the one-hidden-layer small-head setting, the plateau with feature learning survives Muon's quintic and population momentum, while fresh minibatches shorten it (Figures 19–21, pp. 47–48). In deep students the outcome depends on how the downstream layers are trained (Figures 5 and 14, pp. 9 and 42).
- Our claim. For polar steps on the first layer, it holds against a fixed readout (one-hidden-layer runs) or slowly trained later layers (deep students), with an accurate orthogonalizer and full-batch gradients, and in the small-head setting also with Muon's quintic and momentum. The paper leaves open how common it is in standard Muon training (p. 9).
Q1: what happens if the output weights are also trained?
-
The loss then usually falls and the alignment gain shrinks. We ran this (E1) on the three Figure 2 configurations with ten seeds each. Heads were trained by Adam (no weight decay), by gradient descent at , and , and by a normalized-gradient rule at 0.001–0.03, each paired with a fixed head. A run counts as flat if its cycle-mean loss changes by at most 5% between steps 4000–4200 and 7800–8000, with a positive alignment gain. Only the faster head rules give flat runs:
Head Rate Flat at 5% Fixed none 28/30 Adam , , 0/30, 0/30, 1/30 Gradient descent , , 0/30, 0/30, 7/30 Normalized rule 0.001, 0.003, 0.01, 0.03 0/30, 3/30, 6/30, 13/30 At 10%, 30/30 fixed-head and 144/300 trained-head runs are flat. A trained head lowers the held-out MSE in 289/300 runs and the late training loss in 270/300, and gives a smaller alignment gain in 279/300. Training the head also changes the feature trajectory, so these runs do not isolate whether earlier features are converted into loss decrease.
-
Our claim. The revised scope paragraph (Section 6) presents the one-hidden-layer decoupling as a fixed-readout phenomenon. In Muon as used in practice the head is trained with AdamW, and with an AdamW readout started small the effect appears in ordinary training on real images (W2, part 4).
Q2: does the Figure 5 drop use features learned during the plateau?
- Short answer: capacity is held fixed. In E2 and in the longer swap test (part 4), the same layers train from the same state with the same step and budget, and only the starting first layer differs, so more trainable weights cannot explain a gap between arms. The plateau's first layer beats a randomly rotated copy with the same singular values by a wide margin (kept 0.481 against rotated 1.602 below; 77% lower held-out loss over a 1,000-step plateau, 12 of 12), so the drop uses the learned directions. The end of that 1,000-step plateau also beats its start by 3.0% (11 of 12), although the training loss rose 3.5% in between; over E2's shorter window, the changes to added nothing measurable.
- Which layers train matters. In Figure 16 (pp. 43 to 44), training one deep layer next to the head, at the first layer's step, keeps the plateau (45 of 48 runs within 5%), while training one next to removes it (1 of 48).
- Design (E2). Depth-8 residual students of Figure 5, 12 problems 4 initializations. At the switch, is kept at step 500, reset to initialization, randomly rotated, or set back to its value at a plateau-entry point with the same downstream state. That point was fixed in advance from past losses only, as the first step between 100 and 400 whose preceding 100 losses are flat to within 5%. is then frozen while the downstream layers train, and for the kept and rotated arms we also ran Figure 5's protocol, where keeps training at (p. 43).
- Results (final held-out MSE). With frozen, entry 0.467, kept 0.481, reset 1.291 and rotated 1.602. With training, kept 1.518 and rotated 1.607.
- With frozen, the later drop uses the first-layer representation, since reset and rotation are far worse.
- But the representation at the entry point already sufficed. The prespecified contrast, entry minus kept, is -0.014 (95% CI [-0.016, -0.013] over problems), negative in 44 of 48 initializations and in all 12 problems.
- The loss from entry to switch rose by 6–11% in all 48 runs, so that window was not flat, and the changes to in it added no measurable utility for the later decrease.
- The revised Section 5.5 says that the switch does not by itself show which first-layer features the later drop uses. Appendix F no longer says that training the later layers turns the plateau's feature progress into loss progress. With your agreement, the appendix will report all of E2.
Response to Reviewer zn1g, part 3 of 4: Q3, W2
Continuation of our response.
Q3: what carries over to finite width.
- Now in the main text. Section 5.4 reports, right after the theorem, that the plateau with feature learning holds at finite width with Muon's quintic, momentum up to 0.95 and minibatches (Figures 19 to 21).
- Exact at every width: the implemented step length, for a full-rank gradient (Proposition 2, p. 7), and the empirical-law identity (Lemma 17, p. 55).
- In the limit only. Theorem 5 is a mean-field statement. Proposition 29 (pp. 64–65) gives convergence of the width-m dynamics on a fixed horizon through t_F+1, under covariance and eigengap conditions, with no rate. Remark 66 (p. 97) applies it to the AGOP subspace at t_F.
- Open. There is no finite-sample guarantee and no joint finite-width statement of the plateau, recovery and later drop through . The hypotheses of Proposition 4 (p. 8) remain unverified for our runs, and exact weight cycles and the toys' closed-form values do not carry over.
- Finite-width evidence. In small-head runs at finite width (m = 256 at d = 64; m = 1024 at d = 402), the AGOP aligns with the teacher subspace () within 120–280 steps (Figure 20, p. 48). For exact polar steps and Muon's quintic this happens before the loss leaves its 1% band. With exact polar steps in the population, alignment comes at step 120 and the plateau ends at step 200 for d = 64, and at steps 160 and 220 for d = 402 (Figure 21, p. 48).
- Measured at finite width (E5 extended). In 33 Table 2 configurations 3 new seeds with a fixed head, the floor is mostly a mean-output offset (at least half the loss in 28 and 29 of 33, in two windows after step 4000). The alternating part of each pair of polar steps has at least 10 times the Gauss-Newton curvature of the persistent drift (31 and 30 of 33). That curvature is below half the top eigenvalue in 28 of 33, so the oscillation is not concentrated at the top of the spectrum. On the Figure 2 configurations (five rates from to , five seeds), the least-squares slope of log window loss on log is 1.44 and 1.50 (ReLU), 1.23 (SiLU) and 1.18 (GELU).
- Gradient rank, measured. At the end of training, the gradient has full numerical rank in all 165 saved one-hidden-layer runs, with smallest singular value at least of the largest. This supports the rank condition of Proposition 2 at that point, not throughout training.
Each result below was confirmed under criteria fixed before its confirmation runs. Every run is reported, and the numbers were recomputed independently from the raw runs.
W2: Muon's implementation, later layers and real data.
- A practical orthogonalizer keeps the effect. In depth-8 residual students with frozen later layers, three quintic steps followed by three convergent steps with coefficients (2, -1.5, 0.5), 18 matrix products instead of 15, give the flat loss with continued alignment in 23 of 24 runs in float32 and 24 of 24 in float64, against 0 of 24 for Muon's five-step quintic (Figure 18, p. 47). In bfloat16 they do so only at depths 4 and 8. On MNIST inputs this iteration matches exact polar steps in 10 of 10 paired runs. The quintic's gain differs across singular values, so the two half-steps of the cycle stop canceling. Momentum is not repaired this way (p. 48).
- Muon in current large-scale training (prior work). The step is held constant for most of the run before a final decay (Kimi Team, 2025; DeepSeek-AI, 2026), and constant-rate Muon gains markedly from weight averaging or a final decay (Kim et al., 2026). Muon's quintic does not converge to the polar factor (Amsel et al., 2026), and the training setup of a recent trillion-parameter model ends it with steps of the same convergent quintic that we use in Appendix F.3 (DeepSeek-AI, 2026). The orthogonalizer sensitivity in Figure 18 therefore concerns the five-step quintic, not polar updates as such.
Response to Reviewer zn1g, part 1 of 4: S1, S2, W1, W2
Code for the new experiments is at the anonymous link (
rebuttal_code/); the submitted study's code stays at the paper's link. Pages refer to the submitted PDF unless marked as revised. The revised PDF is uploaded with new text in violet, and apart from Section 3's subsections and equations every number is unchanged.We thank the reviewer for the clear summary and for crediting both the empirical separation and the supporting theory. We agree that the separation between loss and alignment is the main empirical contribution.
S1 and S2.
W1: theory for the nonlinear oscillation.
W2: generality, and dependence on the Muon implementation.