acceptodds
Under review as a conference paper at ICLR 2027

Can Feature Learning Decouple From Loss Minimization?

Abstract

Does feature learning stop when the training loss stops improving? We study this question for matrix Muon, whose polar-normalised updates keep a fixed step length regardless of the gradient norm. Near the edge of stability, full-batch Muon on teacher–student problems enters approximately period- loss oscillations that persist for thousands of steps: the cycle-mean loss stays flat or rises, yet the weights keep moving and the learned features continue to align with the teacher subspace. For linear teacher–student learning toys, we derive explicit cycle and alignment formulas and conditional plateau and decay bounds. For a population mean-field ReLU model, we prove that, under stated dimension, initialisation and small-head conditions, the leading eigenspace of the average gradient outer product (AGOP) recovers the teacher subspace exactly during a loss plateau, before the loss later drops. Across ReLU, GELU and SiLU teacher configurations, direction-only alignment metrics and projected head refitting show that the directions learned during plateaus are useful for prediction, and further measurements distinguish AGOP alignment from weight-mass concentration. In deep residual ReLU students, freezing the downstream layers while the first layer trains with full-batch exact polar updates recreates a nearly flat cycle-mean loss with improving input-AGOP alignment; freezing and unfreezing switch between this plateau and loss decrease, and the effect is sensitive to momentum and to the choice of orthogonaliser.

Primary area:
optimization
Keywords:
Feature Learning, Neural Networks, Optimization, Polar Updates

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zn1gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper asks whether a NN can continue learning useful features even after its training loss stops improving. It looks at a simpler version of matrix Muon, that has exact-polar updates and no momentum used for optimization of their model.

The authors use a teacher-student regime. The teacher generates synthetic labels while a student is trained to mimic this. They then track training loss and use AGOP to measure alignment with the teacher’s input directions, finding that alignment improves even while loss keep oscillating.

The authors support this observation with exact results for (simpler) linear models and a mean field theorem proving feature alignment during a plateau. These results provide evidence for the proposed phenomenon, i.e. that the loss and feature alignment can improve on different scale. However, they do not prove the oscillating behavior observed in the nonlinear experiments (which is also the most interesting part in my opinion).

Critical strengths

Empirical separation between loss and feature alignment.\textbf{Empirical separation between loss and feature alignment.} Across many synthetic teacher models, AGOP alignment continnues to improve during oscillating loss plateaus. I think this is the paper’s main contribution.

Some theory that supports this separation.\textbf{Some theory that supports this separation.} The linear models give exact examples where alignment improves althought we have a nonzero loss floor. The mean field result also shows improving alignment during a plateau in a nonlinear setting.

Critical weaknesses

The theory does not explain the main nonlinear observation\textbf{The theory does not explain the main nonlinear observation} As mentioned above, the linear models show that a loss floor can coexist with alignment improvement, it is unclear how they relate to the nonlinear experiments. The mean-field theorem is a different setting separate mechanism and does not establish the oscillations we see in the experiments. I think the paper's theoretical explanation is quite narrow This makes the paper’s theoretical explanation of the main phenomenon narrow as none explain the observed phenomenon.

The observed effect has limited demonstrated generality\textbf{The observed effect has limited demonstrated generality} The strongest evidence comes from synthetic teacher student tasks with full-batch exact-polar updates and a fixed output head (Figs 2 and 4). The main deep NN results also use a large first-layer step and much smaller steps in later layers. Whether loss stays flat while alignment improves depends on how the later layers are trained (Figs 5 and 14), and this effect is sensitive to how Muon is implemented (Fig 18). To me, this makes it unclear how commonly the phenomenon occurs in ordinary NN training.

Questions for authors
  1. What happens if you also train the output weights in the one-hidden-layer experiments? Does alignment still improve while the average loss stays roughly flat?

  2. In Fig 5, how can we be sure that training the later layers lets the network use features learned during the plateau and not simply allowing more weights to change which in turn reduce the loss?

  3. Which parts of the theory do you expect to carry over to the finite width nonlinear setting, and what evidence supports that connection?

5 comments

Response to Reviewer zn1g, part 1 of 4: S1, S2, W1, W2

Code for the new experiments is at the anonymous link (rebuttal_code/); the submitted study's code stays at the paper's link. Pages refer to the submitted PDF unless marked as revised. The revised PDF is uploaded with new text in violet, and apart from Section 3's subsections and equations every number is unchanged.

We thank the reviewer for the clear summary and for crediting both the empirical separation and the supporting theory. We agree that the separation between loss and alignment is the main empirical contribution.

S1 and S2.

  • Table 2 (p. 32) now has a column with the change in cycle-mean loss over the second half, next to the alignment gain, for all 33 rows. The estimator is the per-seed mean of adjacent-pair averages near step 4000 against the final 200 steps, with the median over five seeds.
  • The revised abstract and Section 5.1 now say that the regime is found within the sweep, where Table 2 shows a late gain in 26 of the 33 configurations. Section 5.1 names the exceptions: three labeled "aligned before plateau" (aligned by the midpoint), one that declines slightly, and three not recovered.

W1: theory for the nonlinear oscillation.

  • What is proved and what is measured. The revised introduction marks the results in finite networks as measured, and Section 6 lists the exact results, none of which covers the network's oscillation. Theorem 5 concerns the small-head regime and asserts no period-two dynamics.
  • Where the oscillation is proved. For normalized gradient descent on a positive-definite quadratic with a simple top eigenvalue, convergence to a period-two orbit along the top Hessian eigenvector holds at every step size, provided the iterate keeps a nonzero component along it. The even- and odd-step losses then converge to two positive limits (Arora et al., 2022, Theorem 3.1 and Corollary 3.2). Our learning toys work in that setting and its matrix analogue. Under stated hypotheses they give the loss floor, the slow drain of the remaining modes and the resulting alignment (Theorem 7, pp. 14–15), and exact weight two-cycles under matrix polar steps (Proposition 1, p. 6). Beyond quadratics, Chen & Bruna (2023, Theorem 4) recover the teacher direction of a single-neuron ReLU problem while the weights oscillate. Their convergence to the period-two orbit (Proposition 4) is argued by reduction, and Theorem 2 proves it for a scalar model.
  • For our networks. The period-two signature is consistent with this picture, but we measure it rather than prove it (ρ2=1.0000\rho_2=1.0000 on the 15 diagnostic configurations, p. 6). A theorem for it remains open, and Q3 reports which parts carry over.

W2: generality, and dependence on the Muon implementation.

  • Implementation matters in the large-step regime. The revised scope paragraph (Section 6) states those claims for an accurate orthogonalizer and full-batch gradients. The small-output regime holds with Muon's quintic, momentum and minibatches (Figures 19 to 21).
  • How practical Muon differs from our update. It approximates the polar factor by five steps of a Newton–Schulz quintic with coefficients (3.4445, -4.7750, 2.0315), applied to a Nesterov momentum buffer with coefficient 0.95 (Liu et al., 2025, Sections 2.1 and 2.2). It trains embeddings, output heads and gain or bias parameters with AdamW (Jordan et al., 2024, with its reference implementation; Liu et al., 2025, Section 2.2). At scale it adds weight decay and matches the update RMS of AdamW (Liu et al., 2025, Section 2.2, Eqs. (3)–(4)).
  • Our Muon arm uses the same quintic coefficients, run in double precision. In the deep large-step regime (Figure 18, p. 47), a convergent quintic within 2×10−92\times10^{-9} of the polar factor reproduces the exact-polar result. Muon's quintic breaks the clean cycle, and its full update with Nesterov momentum 0.95 stalls the alignment (pp. 47–49). The quintic is tuned for speed rather than convergence: its value at 1 is 3.4445−4.7750+2.0315=0.7013.4445-4.7750+2.0315=0.701, consistent with its design target of singular values within roughly [0.7, 1.3] (Jordan et al., 2024). We do not claim that this alone explains the loss of the plateau.

Response to Reviewer zn1g, part 2 of 4: W2 (continued), Q1, Q2

Continuation of our response.

W2, continued.

  • Where the effect survives. In the one-hidden-layer small-head setting, the plateau with feature learning survives Muon's quintic and population momentum, while fresh minibatches shorten it (Figures 19–21, pp. 47–48). In deep students the outcome depends on how the downstream layers are trained (Figures 5 and 14, pp. 9 and 42).
  • Our claim. For polar steps on the first layer, it holds against a fixed readout (one-hidden-layer runs) or slowly trained later layers (deep students), with an accurate orthogonalizer and full-batch gradients, and in the small-head setting also with Muon's quintic and momentum. The paper leaves open how common it is in standard Muon training (p. 9).

Q1: what happens if the output weights are also trained?

  • The loss then usually falls and the alignment gain shrinks. We ran this (E1) on the three Figure 2 configurations with ten seeds each. Heads were trained by Adam (no weight decay), by gradient descent at 10−510^{-5}, 10−410^{-4} and 10−310^{-3}, and by a normalized-gradient rule at 0.001–0.03, each paired with a fixed head. A run counts as flat if its cycle-mean loss changes by at most 5% between steps 4000–4200 and 7800–8000, with a positive alignment gain. Only the faster head rules give flat runs:

    HeadRateFlat at 5%
    Fixednone28/30
    Adam10−510^{-5}, 10−410^{-4}, 10−310^{-3}0/30, 0/30, 1/30
    Gradient descent10−510^{-5}, 10−410^{-4}, 10−310^{-3}0/30, 0/30, 7/30
    Normalized rule0.001, 0.003, 0.01, 0.030/30, 3/30, 6/30, 13/30

    At 10%, 30/30 fixed-head and 144/300 trained-head runs are flat. A trained head lowers the held-out MSE in 289/300 runs and the late training loss in 270/300, and gives a smaller alignment gain in 279/300. Training the head also changes the feature trajectory, so these runs do not isolate whether earlier features are converted into loss decrease.

  • Our claim. The revised scope paragraph (Section 6) presents the one-hidden-layer decoupling as a fixed-readout phenomenon. In Muon as used in practice the head is trained with AdamW, and with an AdamW readout started small the effect appears in ordinary training on real images (W2, part 4).

Q2: does the Figure 5 drop use features learned during the plateau?

  • Short answer: capacity is held fixed. In E2 and in the longer swap test (part 4), the same layers train from the same state with the same step and budget, and only the starting first layer differs, so more trainable weights cannot explain a gap between arms. The plateau's first layer beats a randomly rotated copy with the same singular values by a wide margin (kept 0.481 against rotated 1.602 below; 77% lower held-out loss over a 1,000-step plateau, 12 of 12), so the drop uses the learned directions. The end of that 1,000-step plateau also beats its start by 3.0% (11 of 12), although the training loss rose 3.5% in between; over E2's shorter window, the changes to W1W_1 added nothing measurable.
  • Which layers train matters. In Figure 16 (pp. 43 to 44), training one deep layer next to the head, at the first layer's step, keeps the plateau (45 of 48 runs within 5%), while training one next to W1W_1 removes it (1 of 48).
  • Design (E2). Depth-8 residual students of Figure 5, 12 problems ×\times 4 initializations. At the switch, W1W_1 is kept at step 500, reset to initialization, randomly rotated, or set back to its value at a plateau-entry point with the same downstream state. That point was fixed in advance from past losses only, as the first step between 100 and 400 whose preceding 100 losses are flat to within 5%. W1W_1 is then frozen while the downstream layers train, and for the kept and rotated arms we also ran Figure 5's protocol, where W1W_1 keeps training at η1=0.66\eta_1=0.66 (p. 43).
  • Results (final held-out MSE). With W1W_1 frozen, entry 0.467, kept 0.481, reset 1.291 and rotated 1.602. With W1W_1 training, kept 1.518 and rotated 1.607.
    • With W1W_1 frozen, the later drop uses the first-layer representation, since reset and rotation are far worse.
    • But the representation at the entry point already sufficed. The prespecified contrast, entry minus kept, is -0.014 (95% CI [-0.016, -0.013] over problems), negative in 44 of 48 initializations and in all 12 problems.
    • The loss from entry to switch rose by 6–11% in all 48 runs, so that window was not flat, and the changes to W1W_1 in it added no measurable utility for the later decrease.
  • The revised Section 5.5 says that the switch does not by itself show which first-layer features the later drop uses. Appendix F no longer says that training the later layers turns the plateau's feature progress into loss progress. With your agreement, the appendix will report all of E2.

Response to Reviewer zn1g, part 3 of 4: Q3, W2

Continuation of our response.

Q3: what carries over to finite width.

  • Now in the main text. Section 5.4 reports, right after the theorem, that the plateau with feature learning holds at finite width with Muon's quintic, momentum up to 0.95 and minibatches (Figures 19 to 21).
  • Exact at every width: the implemented step length, ηrank\eta\sqrt{\mathrm{rank}} for a full-rank gradient (Proposition 2, p. 7), and the empirical-law identity (Lemma 17, p. 55).
  • In the limit only. Theorem 5 is a mean-field statement. Proposition 29 (pp. 64–65) gives convergence of the width-m dynamics on a fixed horizon through t_F+1, under covariance and eigengap conditions, with no rate. Remark 66 (p. 97) applies it to the AGOP subspace at t_F.
  • Open. There is no finite-sample guarantee and no joint finite-width statement of the plateau, recovery and later drop through t2t_2. The hypotheses of Proposition 4 (p. 8) remain unverified for our runs, and exact weight cycles and the toys' closed-form values do not carry over.
  • Finite-width evidence. In small-head runs at finite width (m = 256 at d = 64; m = 1024 at d = 402), the AGOP aligns with the teacher subspace (Ar≥0.9A_r\ge0.9) within 120–280 steps (Figure 20, p. 48). For exact polar steps and Muon's quintic this happens before the loss leaves its 1% band. With exact polar steps in the population, alignment comes at step 120 and the plateau ends at step 200 for d = 64, and at steps 160 and 220 for d = 402 (Figure 21, p. 48).
  • Measured at finite width (E5 extended). In 33 Table 2 configurations ×\times 3 new seeds with a fixed head, the floor is mostly a mean-output offset (at least half the loss in 28 and 29 of 33, in two windows after step 4000). The alternating part of each pair of polar steps has at least 10 times the Gauss-Newton curvature of the persistent drift (31 and 30 of 33). That curvature is below half the top eigenvalue in 28 of 33, so the oscillation is not concentrated at the top of the spectrum. On the Figure 2 configurations (five rates from 0.25η0.25\eta to 1.25η1.25\eta, five seeds), the least-squares slope of log window loss on log η\eta is 1.44 and 1.50 (ReLU), 1.23 (SiLU) and 1.18 (GELU).
  • Gradient rank, measured. At the end of training, the gradient has full numerical rank in all 165 saved one-hidden-layer runs, with smallest singular value at least 1.2×10−31.2\times10^{-3} of the largest. This supports the rank condition of Proposition 2 at that point, not throughout training.

Each result below was confirmed under criteria fixed before its confirmation runs. Every run is reported, and the numbers were recomputed independently from the raw runs.

W2: Muon's implementation, later layers and real data.

  • A practical orthogonalizer keeps the effect. In depth-8 residual students with frozen later layers, three quintic steps followed by three convergent steps with coefficients (2, -1.5, 0.5), 18 matrix products instead of 15, give the flat loss with continued alignment in 23 of 24 runs in float32 and 24 of 24 in float64, against 0 of 24 for Muon's five-step quintic (Figure 18, p. 47). In bfloat16 they do so only at depths 4 and 8. On MNIST inputs this iteration matches exact polar steps in 10 of 10 paired runs. The quintic's gain differs across singular values, so the two half-steps of the cycle stop canceling. Momentum is not repaired this way (p. 48).
  • Muon in current large-scale training (prior work). The step is held constant for most of the run before a final decay (Kimi Team, 2025; DeepSeek-AI, 2026), and constant-rate Muon gains markedly from weight averaging or a final decay (Kim et al., 2026). Muon's quintic does not converge to the polar factor (Amsel et al., 2026), and the training setup of a recent trillion-parameter model ends it with steps of the same convergent quintic that we use in Appendix F.3 (DeepSeek-AI, 2026). The orthogonalizer sensitivity in Figure 18 therefore concerns the five-step quintic, not polar updates as such.
Reviewer_eXR6OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper investigates whether feature learning can continue while training loss remains nearly constant or increases under exact-polar Muon updates. It presents linear toy analyses, theoretical properties of polar updates, and a mean-field theorem establishing teacher-subspace recovery during a near-flat-loss interval and a later loss decrease. Experiments track loss and feature alignment across architectures and training settings.

Critical strengths
  1. I believe the most significant result is Theorem 5, which establishes teacher-subspace recovery while loss remains nearly unchanged and guarantees a larger loss decrease at a later time. This gives a rigorous foundation for the separation between feature learning and loss reduction under explicit assumptions.
Critical weaknesses
  1. Insufficient connection between the theory and the main empirical phenomenon. The purpose and relationship of the theoretical results are not clearly developed: Proposition 2 provides elementary polar identities, Proposition 4 relies on assumptions unverified in the experiments, and Theorem 5 studies a distinct small-head regime. Although these results have individually stated scopes, it remains unclear how these results explain the continued alignment improvement observed in the main oscillatory experiments.

  2. Limited evidence beyond constructed settings. The main positive results rely on specially chosen teacher–student problems, while the real-data experiments in Appendix F.4 do not reproduce continued alignment improvement during a loss plateau. This leaves the relevance of the phenomenon to ordinary neural-network training insufficiently established.

Questions for authors

Clarification questions:

  1. Theorem 5 studies a small-head model, while the main experiments show loss oscillations caused by large steps. What does Theorem 5 help us understand about those main experiments?
  2. What does Theorem 5 add to the silent-alignment results of Atanasov et al. (2022), where alignment improves before loss decreases appreciably?
  3. The real-data experiments in Appendix F.4 do not show continued feature alignment during a loss plateau. Why do you think the result differs from the synthetic experiments? Could useful feature learning still be happening without being detected by your alignment measure?

Suggestion for future work:

  1. It would be useful to compare reducing the learning rate immediately when a plateau begins against keeping it constant for longer before reducing it, using the same total training budget. This would help establish whether the additional alignment gained during the plateau improves subsequent prediction, providing a clearer practical motivation for studying the phenomenon.

  2. The presentation of the paper can be significantly improved. For example, why is the neural-network setup introduced before the linear toys? The transition is unclear, and the purpose of switching between these settings is not sufficiently explained. What is Proposition 1 intended to tell us? It appears without much introduction. What do the assumptions and numerical constants in Theorem 5 mean, and why are they needed? These points need an intuitive explanation in the main text.

7 comments

Response to Reviewer eXR6, part 1 of 4: S1, W1, Q1

Code for the new experiments is at the anonymous link (rebuttal_code/); the submitted study's code stays at the paper's link. Pages refer to the submitted PDF unless marked as revised. The revised PDF is uploaded with new text in violet, and apart from Section 3's subsections and equations every number is unchanged.

We thank the reviewer for the careful reading and for singling out Theorem 5.

W1 and Q1: how the theory connects to the main oscillatory experiments.

W1. The revised Section 1 makes the connection explicit: it lists six labeled contributions, proved first and measured second. Each link is classified below.

  • Proved under stated hypotheses:
    • the polar identities, which fix the step length for a full-rank gradient but imply neither alignment nor periodicity (Proposition 2, p. 7);
    • the exact weight two-cycles, loss floors and alignment formulas of the learning toys (Proposition 1, p. 6, and Theorem 7, pp. 14–15);
    • the three guarantees of Theorem 5 (p. 8).
  • What the toys show: fixed-length steps hold the loss at a floor set by the step size, while misalignment in low-curvature directions drains slowly and carries little loss. This builds on the period-two orbit of normalized gradient descent on a quadratic (Arora et al., 2022, Theorem 3.1 and Corollary 3.2).
  • Open: a theorem for the period-two oscillation of the nonlinear network, and the direction of its drift. Proposition 4 (p. 8) gives sufficient conditions, but its hypotheses remain unverified for our sweep.

New measurement (E5 extended). 33 Table 2 configurations ×\times 3 new seeds, fixed head, per-configuration medians in two windows after step 4000. The mean-output offset carries at least half of the plateau loss in 28 and 29 of 33. The alternating part of each pair of polar steps has at least 10 times the Gauss-Newton curvature of the persistent part (31 and 30 of 33) and mostly rescales neurons (32 of 33). But its curvature is below half the top eigenvalue in 28 of 33, so the oscillation is not concentrated at the top of the spectrum, and the analogy with the toys' leading block is not supported at this level. The drift's turning part is not enriched in the teacher subspace, so its direction stays open.

Q1. Theorem 5 helps with what the toys cannot reach: why the features keep moving while the loss does not. Two properties of the polar step do this. Its length does not depend on the gradient's size, so the hidden weights keep moving even when the loss cannot respond. It also gives every singular direction of the gradient the same step, so weak directions, including one that a linear fit cannot see, are not left behind. Theorem 5 proves, in a model we can solve, that this suffices. At a step t_F that does not depend on the head, the top-r AGOP eigenspace is exactly the teacher subspace, while the loss stays within 1% of its initial value. The regime also holds at finite width with Muon's quintic, momentum and minibatches (Figures 19 to 21), and in ordinary training from a small readout. In Theorem 5's model at equal step length, only polar steps recover the subspace inside the band (E6, part 4 below, now Appendix H.4 of the revised PDF). In the main experiments the loss is held instead by the floor of the period-two oscillation under large steps, with a fixed head of 0.1 per unit. Section 4's toys explain that floor, and Theorem 5 does not model it.

  • The polar factor ignores the head. It is unchanged when the gradient is multiplied by a positive scalar, so the hidden-layer step does not shrink with the head (eqs. (86)–(87), p. 52).
  • The force still depends on the head. It is F=g−αSF=g-\alpha S (p. 54), and the whitening of the polar step depends on α\alpha through F. By Lemma 18 the trajectory depends on the head only through αηˉ\alpha\bar\eta.
  • What the proof certifies: a feature time tF=⌊τFκ⌋t_F=\lfloor\tau_F\kappa\rfloor at which (F) holds for every head with αηˉ≤a0\alpha\bar\eta\le a_0, where a0>0a_0 > 0 exists but is not explicit, while the loss moves on the slower clock αηˉt/σY\alpha\bar\eta t/\sigma_Y.
  • How the main experiments differ. The small-head plateau survives the Newton–Schulz quintic and population momentum, whereas the large-step plateau needs an accurate orthogonalizer (Appendix F.3, pp. 46–49).
  • Scope of Theorem 5. Its proof uses no oscillation, so it does not establish the mechanism of the oscillatory experiments, nor that the recovered directions cause the later drop.

Response to Reviewer eXR6, part 2 of 4: Q2 (Theorem 5 and silent alignment), W2, Q3

Continuation of our response.

Theorem 5 proves three things that silent alignment (Atanasov et al., 2022) does not, and E6 points to where the difference comes from. The small-output idea is shared with silent alignment, and the revised related work presents Theorem 5 as building on it; we claim no priority for that idea.

  1. Nonlinear feature learning, exactly.
    • Theorem 5 concerns a one-hidden-layer ReLU network in the mean-field limit, with Gaussian inputs and a teacher of r≥2r\ge2 ReLU units on orthonormal directions.
    • At t_F, the top-r eigenspace of the input AGOP, isotropic at initialization, equals the teacher subspace, with a certified eigengap.
    • The linear part of the student–teacher covariance involves only the mean teacher direction (Lemma 33, p. 66), so r−1r-1 teacher directions are invisible to any linear fit. They are recovered nonetheless (Theorem 16, p. 53).
    • In the scalar-output linear networks of Atanasov et al., the learned structure is the single direction of the linear target.
  2. Only the output is small. The hidden weights start at an ordinary scale (κ≫1\kappa\gg1), and only the fixed head is small. Silent alignment relies on a small initialization of every layer.
  3. The head sets only the loss clock. Under polar steps, t_F does not depend on the head scale α\alpha once αηˉ≤a0\alpha\bar\eta\le a_0, while the loss moves on the clock αηˉt/σY\alpha\bar\eta t/\sigma_Y. So the plateau can be made arbitrarily long relative to t_F, with explicit bounds. In the two-layer linear analysis of Atanasov et al., the gap between alignment and loss movement is set by the initialization scale σ\sigma (their Section 3.1).
  4. Polar steps made the difference at ordinary initialization (E6, part 4). At the same nominal step length and κ=100\kappa=100, polar steps recovered both teacher directions inside the 1% band in 5 of 5 seeds at two sizes, and normalized and plain gradient descent in 0 of 5. From a small initialization (κ=0.01\kappa=0.01), the regime of silent alignment, all three recovered within three steps.

Where silent alignment goes further, and our limits. Atanasov et al. treat deep and multi-output linear networks, characterize the final predictor as kernel regression with the final tangent kernel, and show the effect weakening for anisotropic inputs. Theorem 5 is a population, mean-field statement for Gaussian data, and its exactness relies on the symmetry of that model. Its finite-width version is a limit without a rate (Proposition 29, pp. 64–65; Remark 66, p. 97), a0a_0 is not explicit, and (F) holds at the single time t_F.

The revised related work also cites Min et al. (2024, Theorem 1) next to Boursier & Flammarion (2025, Theorem 1). In both works both layers start small and neurons align individually.

W2: evidence beyond constructed settings.

  • New real-data results. The reviewed version's positive results are teacher–student, and Appendix F.4 does not reproduce continued alignment during a plateau on real data. New experiments on MNIST, Fashion-MNIST and CIFAR-10 now show the effect in the settings below. Our reply to Reviewer H7Vb (parts 2 to 4) gives each design and count.
  • Every layer trained, as in practice. With minibatch Muon on the hidden layers and AdamW on a readout started small, a held-out probe and the input AGOP reach thresholds fixed in advance before the loss leaves its 1% band, in 19 of 20 runs on MNIST pairs and in 20 of 20 on Fashion-MNIST pairs, on MNIST pairs with six hidden layers and on CIFAR-10 pairs (our reply to Reviewer H7Vb, part 3).
  • Fixed readout, full-batch polar steps. Labels that no single image predicts give the effect on MNIST pairs. A one-hidden-layer student keeps its cycle-mean loss within 5% in 18 of 20 runs while a held-out probe and the input AGOP improve.
  • What the large-step effect needs: a fixed or slowly trained readout and full batches. Training the readout in the full-batch setting usually removes the flat loss (E1, our reply to Reviewer zn1g, Q1), and with minibatches of 1/32 the large step no longer holds the period-two cycle (F1 below).
  • Related evidence in standard training. Large-step SGD and constant-rate schedules also hide progress behind a stable or elevated loss, but through minibatch noise rather than the fixed length of polar steps (Andriushchenko et al., 2023; Hägele et al., 2024; Wen et al., 2025).
  • Our claim. For polar steps on the first layer, it holds against a fixed, frozen or slowly trained downstream, with an accurate orthogonalizer and full-batch gradients, on synthetic teachers and the MNIST tasks above, and in ordinary training on real images from a small readout. Section 6 states the scope; the results are in Section 5.4 and Appendix H.1.

Response to Reviewer eXR6, part 3 of 4: Q3, F1

Continuation of our response.

Q3: why the real-data experiments differ, and whether learning could go undetected.

  • Why F.4 differs.

    • Labels. F.4's digit models learn labels of single images, which carry first-order signal. On MNIST such features are learned within the first few hundred steps, before any plateau, and the F.4 digit models also start from 200 or 500 steps of AdamW pretraining (Table 10, p. 49). A label that no single image predicts restores the effect, while the one-digit label on the same inputs does not (paired contrast 20 of 20).
    • Inputs. With a planted teacher on real images, the input coordinates are dependent. ReLU features of the other coordinates partly predict the teacher's output, so the loss floor shrinks and the fit keeps improving. Shuffling each coordinate independently, which keeps its distribution, restores the Gaussian behavior.
  • What the F.4 probes record. Each F.4 run also refits a ridge-regularized linear probe from the representation that feeds the fixed head, scored on held-out data. For the three models whose loss sits on a clean flat cycle (p. 49), the median over ten runs does not change over the window for the CNN (0.972) or the four-patch vision transformer (0.959). It rises slightly for the transformer (R2R^2 0.459 to 0.467). The probe registers only what a linear readout of that representation can use, so unchanged medians do not show that nothing was learned.

  • Undetected learning is possible in principle, and the revised Appendix F.4 now states this limitation. Our measure asks whether the leading r-dimensional eigenspace of the input AGOP (Radhakrishnan et al., 2022) coincides with a reference subspace (eq. (9), p. 4), and it ignores the eigenvalues. For a planted teacher the reference is the teacher subspace. The digit tasks have no such subspace, so their reference is a learned rank-8 AGOP subspace (p. 49).

  • Learning can escape the measure in four ways.

    • (i) Changes inside an aligned subspace, such as neuron specialization.
    • (ii) Changes in the AGOP spectrum, or in directions beyond the leading r.
    • (iii) Useful directions that differ from a learned reference.
    • (iv) Label-driven changes in the tangent kernel (Atanasov et al., 2022, Section 3.1; Baratin et al., 2021), which need not move a low-rank input subspace.

    We did not apply such measures in F.4, so we draw no conclusion either way about undetected learning there.

F1: learning-rate timing at equal budget.

  • New result (E3). In Figure 2's fixed-head setting (ReLU, SiLU and GELU teachers, 10 new paired seeds each, 8,000 steps), a preset rule found plateau entry in the constant-rate training loss (steps 214 to 298). We compared a tenfold step decay at entry and at step 4000. The later decay raised the final held-out MSE in all three (×10−3\times10^{-3}, 95% bootstrap CI), by 12.7 [11.0, 14.6] for ReLU, 0.62 [0.06, 1.11] for SiLU and 1.54 [1.08, 2.06] for GELU. It was worse in 10/10, 8/10 and 10/10 seeds, a rise of 31%, 0.2% and 0.9% of the entry arm's MSE. Before its decay the later arm was better aligned (mean cos⁡min⁡2\cos^2_{\min} 0.30 to 0.67 against 0.02 to 0.03 at entry), but final cos⁡min⁡2\cos^2_{\min} differed by under 0.003 in every seed. In this setting decaying at plateau entry was better, and the alignment gained by holding the rate did not lower held-out error at equal budget.
  • Other settings (new). At equal budget the answer depends on the setting.
    • Decaying at onset is better in depth-8 students with a Gaussian teacher, where decaying 90% of the way past onset ends with a 27% higher held-out loss (0 of 12 runs favor it). It is also better in a convolutional network with four residual blocks (5.9 points lower accuracy when decaying late, 0 of 12).
    • Decaying late is better in one-block transformers on MNIST and Fashion-MNIST pairs. With the later block frozen it wins in 44 of 48 pairings across four settings (+0.4 to +3.3 points), and in 42 of 48 when the block trains from the decay. On a second server the Fashion-MNIST median gain with the block frozen was +0.02 points rather than +0.44.
    • Minibatches (new, synthetic teachers). With minibatches of 1/8 and 1/32 of the training set, late decay did not beat onset decay by the frozen bars. In depth 8 it ended with a 6% higher held-out loss at 1/8 and a 31% higher one at 1/32 (medians, ahead in 3 and 0 of 12), against 27% with full batches. Minibatch noise removed the constant step's period-two cycle at 1/32 in every setting, and a small step throughout learned the features as well as onset decay. So with stochastic gradients these students do not reward a longer large-step phase either.
Reviewer_H7VbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper concerns the phenomenon of improving feature representations even when the training loss stagnates. The main testbed is teacher-student regression on Gaussian data, using one-hidden-layer networks with fixed output weights, as well as simple linear models. Additional experiments involve deeper residual networks. The central quantity is the Average Gradient Outer Product (AGOP), the second-moment matrix of the student's gradients with respect to its input. Alignment of the leading eigenspace of the student AGOP with the teacher subspace indicates that the student has learned task-relevant feature directions. Student models are trained using normalized GD or matrix Muon. In the regimes considered, these algorithms exhibit persistent, approximately period-two loss oscillations rather than convergence to a minimizer. Nevertheless, feature representations can continue improving during these oscillations, even when the average training loss remains nearly constant or increases. For quadratic models, the authors relate this phenomenon to persistent oscillations in high-curvature directions combined with continued reduction of errors in other directions. For nonlinear networks, the phenomenon is demonstrated experimentally. The paper also provides a separate mean-field theorem establishing feature alignment during a loss plateau under specific assumptions. The submission combines analytical results for simplified models with extensive experiments investigating the phenomenon.

Critical strengths

None

Critical weaknesses

My main concern is presentation. The paper is stuffed with multiple results, both theoretical and empirical, but lacks clear narrative. The results give an impression of being fragmented rather than telling a solid story. Already Introduction exploits much of specialized, unfamiliar terminology, while Sections 3 and 4 introduce a number of entities and technical details that bury the overall message. To my point of view, the manuscripts needs to be considerably refactored.

Questions for authors

It is not actually clear from the main part if the claimed phenomenon generalize well beyond the toy setups presented in the main; could the authors comment on this?

5 comments

Response to Reviewer H7Vb, part 1 of 4: presentation and generality

Code for the new experiments is at the anonymous link (rebuttal_code/); the submitted study's code stays at the paper's link. Pages refer to the submitted PDF unless marked as revised. The revised PDF is uploaded with new text in violet, and apart from Section 3's subsections and equations every number is unchanged.

We thank the reviewer and agree with the main concern. The submission presents many results without a clear thread, uses specialized terms before defining them, and introduces many symbols in Section 3 (pp. 3–5) before the first network result (p. 6). The revised introduction reframes the paper around one question.

The story in three steps. (1) Under fixed-length polar steps, the training loss of a teacher–student network with a fixed output layer can stop improving for thousands of steps. Meanwhile the input directions the network is most sensitive to (top eigenvectors of its average gradient outer product, AGOP) keep turning toward the teacher's. (2) Linear toys show how. A fixed-length step cannot settle at the minimum, so the loss sits on a floor, while error in low-curvature directions, which costs almost no loss, keeps draining. A mean-field theorem proves the same decoupling where a small output layer keeps the loss flat. (3) The effect can be measured without the loss and switched on and off by freezing the later layers of deep networks. In the large-step regime it needs a fixed or slowly trained readout, accurate polar steps and full batches.

Concrete changes.

  1. The introduction states the question, says what is proved (learning toys, and Theorem 5's mean-field model with a small head) and what is measured (finite networks), and lists the contributions.
  2. Section 6 states the scope of every claim, and the new results appear in Sections 5.4 and 5.5 and Appendix H.
  3. Terms are defined in words at or near first use (polar update, AGOP alignment, cycle-mean loss, period-two oscillation).
  4. Section 3 introduces the models in order, from the learning toys (M1 to M3) to the network (M4) and its mean-field version. The deep-residual paragraph leaves the mean-field section (p. 5), and Figure 2 moves next to the network results.
  5. Short paragraphs before Proposition 1 and after Theorem 5 say what each shows and what its assumptions and constants mean, and Figure 3's caption maps each panel to its claim.

The main text stays within the 10-page limit.

Does the phenomenon generalize beyond the toy setups?

Yes. The linear toys isolate the mechanism, but the phenomenon itself is measured in nonlinear networks: one-hidden-layer students of ReLU, GELU and SiLU teachers, deep residual students, and the small-output regime of Theorem 5 at finite width. In the reviewed version these networks are trained on Gaussian inputs labeled by a planted teacher (pp. 3, 41). The revision adds ordinary training on MNIST, Fashion-MNIST and CIFAR-10 pairs, with every layer trained (minibatch Muon on the hidden layers, AdamW on a readout started small). The revised Section 6 states the conditions of each claim, and and parts 2 to 4 below give each setting with its criterion and outcome.

Where it holds.

  • One hidden layer, fixed output layer (Table 2, p. 32). Over the second half, the weakest-direction alignment rises by at least 0.005 in 29 of the 33 configurations (ReLU, GELU and SiLU teachers) and ends between 0.415 and 0.99 in 26. The loss oscillation is period two in all 33.
  • Deep residual students, with slow later layers (depths 6 to 16 in Figure 4, p. 9, and Figure 14, p. 42) or frozen ones (depths 4 to 24 in Figure 5, p. 9, and Appendix F, p. 45).
  • The small-output regime of Theorem 5 at finite width. Alignment reaches 0.9 before the loss leaves its 1% band with exact polar steps, Muon's quintic iteration, momentum up to 0.95 (also Nesterov) and minibatches of 4,096 or 256 samples (Figures 19 to 21, pp. 47–48).
  • On real images, every layer trained (new). With Muon as in practice and a readout started small, MLPs and CNNs on MNIST, Fashion-MNIST and CIFAR-10 pairs learn features before the loss leaves its 1% band. This holds in at least 18 of 20 runs in each of eight experiments (part 3 below).

Response to Reviewer H7Vb, part 2 of 4: generality (continued) and new experiments

Continuation of our response.

Related effects in standard training. Under large-step SGD the loss stabilizes while features become sparser (Andriushchenko et al., 2023). A constant-rate phase keeps the loss elevated until a final decay reveals the progress (Hägele et al., 2024; Wen et al., 2025), also for Muon (Kim et al., 2026). In full-batch gradient descent a kernel measure of the features aligns in jumps (Cohen et al., 2021; Jiang et al., 2025). Closest to our question, linear probes find task features forming during a long flat loss phase in transformers on algorithmic tasks (Naidu et al., 2026). Our work adds the orthogonalized case and its mechanism: the fixed length of polar steps holds the loss on a floor while the AGOP keeps aligning.

What the large-step effect needs. Part 1 names three conditions, and each was tested.

  • A fixed or slowly trained readout (E1, our reply to Reviewer zn1g, Q1). A trained output layer usually lets the loss fall and shrinks the alignment gain. A flat loss (within 5%) with a positive alignment gain holds in 28 of 30 fixed-head runs and 30 of 300 trained-head runs.
  • An accurate orthogonalizer (Figure 18, pp. 47–49). Under Muon's five-step Newton–Schulz iteration or momentum, the flat loss with continued alignment breaks down. An iteration ending with convergent steps restores it (part 4 below), though not with momentum.
  • Full batches (new). With minibatches of 1/32 of the training set at a large step, the period-two cycle is gone in depth-8 students and in Figure 2's setting (our reply to Reviewer eXR6, F1).

The small-output regime is more robust: at finite width it holds with Muon's quintic, momentum and minibatches (part 1).

Real data. Ordinary training from a small readout gives the effect on MNIST, Fashion-MNIST and CIFAR-10 pairs (part 3). With full batches and a fixed readout, labels that no single image predicts give it on MNIST (below). In the digit models of Appendix F.4 (p. 49), which learn labels of single images, alignment against the learned reference does not keep improving during the plateau, and a transformer on synthetic Gaussian tokens learns only weakly.

Our claim. For polar steps on the first layer, the effect holds against a fixed or frozen readout (one hidden layer) or slowly trained later layers (deep students), with an accurate orthogonalizer and full-batch gradients, and more broadly in the small-output regime, including ordinary minibatch training on real images from a small readout.

New experiments

  • New real-data results (MNIST). The first two were confirmed under criteria fixed before their confirmation runs, and E7 reproduces an earlier result on previously used seeds. Every run is counted, and the numbers were recomputed independently from the raw runs. All train the first layer with full-batch exact polar steps against a fixed output layer or frozen later layers.
    • Labels that no single image predicts (two digits labeled by the parity of their digit sum). In a one-hidden-layer student the cycle-mean loss stays within 5% in 18 of 20 runs at the pre-registered step (median -3.9%). Meanwhile a held-out probe gains 1.9 points, and the input AGOP concentrates on the label directions in 20 of 20. The parity of one digit gives no such gain (paired contrast 20 of 20).

    • A convolutional student (one 10×1010\times10 convolution, fixed head; now in Appendix H.3 and Section 5.5) passes all three criteria (flat cycle mean, probe gain, AGOP gain) in 13 of 16 runs, and in 21 of 24 new runs at a step fixed in advance, chosen after confirmations at two other steps and an exploratory block. Paired with the one-digit label, the digit-sum label gains more on the AGOP in 11 of 12 runs. It is one minimal design, with network accuracy near 65%.

    • A one-block transformer without LayerNorm, on the same pairs with its later block frozen, keeps aligning while its loss stays nearly flat (now in Appendix H.3 and Section 5.5). We repeated the design on 24 fresh pairings, each with a branch that continues from step 500 at a tenfold smaller step.

      One-block transformer, later block frozenResult
      AGOP gain over the 1,000-step window20/20 runs, loss -3.8% (median)
      Fresh pairings: AGOP mass rises24/24, loss within 5% in 23/24 (median -3.5%)
      Fresh pairings: gain above the one-digit label's, for the AGOP mass and for the probeat least 10/12 pairings each
      Against the small-step branch: AGOP gain per unit of fit progresslarger in 24/24, loss -3.5% against -41% (medians)

      So the large step gains more alignment per unit of fit progress than a small step, though not more in total, and its held-out accuracy gains 3.6 points less. It is claimed on these measures, not on the three per-run criteria, and for blocks without LayerNorm.

Response to Reviewer H7Vb, part 3 of 4: new experiments (continued)

Continuation of our response.

  • Real inputs with a planted teacher (E7). A one-hidden-layer student learns a rank-4 ReLU teacher on PCA-whitened MNIST inputs (30 previously used seeds). Over a fixed early window (steps 60 to 241) at step size 0.6, alignment rises in every seed (median gains of +0.22 for the mean and +0.44 for the weakest direction), while the cycle-mean loss changes by a median of -4.4% (26 of 30 within 5%). The window is short and nearly flat rather than a sustained plateau, so E7 carries the planted-teacher result to real inputs.

  • Ordinary training from a small readout (new; now in Section 5.4 and Appendix H.1, with Figures 22 to 24). We trained every layer of a network with three hidden layers on MNIST digit pairs labeled by the parity of their digit sum, each image entering as 50 whitened principal components. Training used Muon as in practice (minibatches of 256, Nesterov momentum) and a readout started small, like Theorem 5's fixed readout, but trained by AdamW. We repeated this on Fashion-MNIST pairs, with six hidden layers on MNIST, and on CIFAR-10 pairs, and trained small convolutional networks end to end on the pixels of MNIST, Fashion-MNIST and CIFAR-10 pairs. Thresholds on a held-out probe of the last hidden layer and on the input AGOP's mass on the label directions were fixed before the runs. The band is the training loss within 1% of its initial value.

Data and networkBoth thresholds met inside the band (bar 16)Loss leaves the band (median step)Probe gain at band exit above a standard-scale readout'sHeld-out accuracy at step 400 (median)
MNIST pairs, 3 hidden layers19/204720/20 pairs84%
Fashion-MNIST pairs, 3 hidden layers20/2039.520/2091%
MNIST pairs, 6 hidden layers20/206217/2085%
CIFAR-10 pairs, 3 hidden layers20/202120/2070%
Convolutional network, MNIST pairs20/2055.520/2091%
Convolutional network, Fashion-MNIST pairs20/202920/2091%
Convolutional network, CIFAR-10 pairs, label of both images20/20287.515/2057%
Convolutional network, CIFAR-10 pairs, label of the first image18/203420/2075%
  • When the loss left the band, the probe had gained a median of 7.5 points on MNIST pairs, 16.2 on Fashion-MNIST pairs and 10.7 with six hidden layers. With three hidden layers on MNIST, the AGOP's mass on the label directions rose about sixfold.
  • Both thresholds were met a median of 23.5 steps before the loss left the band with six hidden layers and 7.5 steps before on CIFAR-10, and 41 and 23 steps before in the convolutional network on MNIST and Fashion-MNIST.
  • On CIFAR-10 the label of both images is the parity of the sum of their class indices (40,000 training pairs), and the label of the first image the parity of its class index (10,000 pairs). The thresholds were met a median of 107.5 and 14.5 steps before the loss left the band.
  • With the orthogonalizer of a recent trillion-parameter model (eight quintic steps, then two convergent ones; DeepSeek-AI, 2026) instead of Muon's quintic, both thresholds were again met inside the band in 20 of 20 runs. This held with three and six hidden layers on MNIST.
  • With a one-image label, both thresholds were still met inside the band in 20 of 20 runs with three hidden layers, though the loss left the band sooner in all 20 pairs (three and six hidden layers on MNIST, and the convolutional network on both datasets).
  • At the standard readout scale, the six-layer network still met both thresholds inside the band in 19 of 20 runs. The small readout lengthens the plateau: at the standard scale the loss moved after a median of 21 steps with three hidden layers on MNIST.

Comments on the paper

1 comment

Summary of revisions

We thank the reviewers again. The revised PDF is uploaded with new or reworded text in violet. Section 3 now introduces the models in order (M1 to M4), which changes its subsection and equation numbers, and all other existing numbers are unchanged. Pages refer to the revised PDF.

Presentation (all three reviews)

  • The introduction asks one question, says how we study it in two regimes, and lists our contributions, proved first and measured second (pp. 1–2). The abstract and each section follow this story.
  • Terms are defined in words at first use, including the AGOP and the cycle-mean loss (p. 1).
  • Section 3 introduces the models in order (pp. 3–5), and its conventions paragraph moves to the start of the appendix (p. 15).
  • Figure 2 sits beside the network results it shows (p. 6), and the deep residual students are defined in Section 5.5, where they are used (p. 9).
  • New paragraphs say what Proposition 1 shows (p. 6) and how to read Theorem 5 (p. 9), and the captions of Figures 3 and 5 state their claims (pp. 7 and 10).

New experiments in the paper

  • Ordinary training from a small readout: MLPs and CNNs on MNIST, Fashion-MNIST and CIFAR-10 pairs, with minibatch Muon on every hidden layer and AdamW on a readout started small. Probe and AGOP thresholds fixed in advance are met before the loss moves by 1% (Section 5.4, p. 9, and Appendix H.1 with Table 17 and Figures 22 to 24, pp. 106–110).
  • Section 5.5 (p. 10) adds three tests of freezing: repeated switching of the later layers (Appendix H.2, Figure 25), a transformer with a frozen later block, and a fixed-head CNN on MNIST pairs (Appendix H.3).
  • To make room, Section 6 drops the weight-mass comparison (still in Appendix C.4, p. 37) and Section 5.5 the pretraining remark (still in Appendix F.2, p. 47).
  • The ethics and reproducibility statements cover these datasets and Appendix H, and the datasets are cited (p. 11).

Theory and scope

  • Theorem 5 proves the same decoupling in a second regime, where a small fixed head rather than the oscillation holds the loss (pp. 2 and 9). Section 5.4 names the two properties of the polar step that drive it and reports finite-width runs with Muon's quintic, momentum and minibatches (Figures 19 to 21).
  • New Appendix H.4 (pp. 112–113) runs Theorem 5's model under three update rules at equal step length (E6). Only polar steps recover the teacher subspace inside the 1% band.
  • Section 6 relates the two regimes and states the scope of our claims (p. 10).
  • Related work (pp. 2–3) presents Theorem 5 as building on the small-output idea of silent alignment, and we claim no priority for it. Early-alignment results, Chen & Bruna (2023) and Arora et al. (2022) are cited by result (also pp. 5 and 17), and a new paragraph covers hidden progress in standard training.

Corrections

  • Abstract: the step length is fixed for a full-rank gradient, the regime is found within a sweep of 33 configurations with a late alignment gain in 26, and head refitting used the six cleanest ReLU configurations (p. 1).
  • Section 4 limits the normalized-update cycles to the illustrated runs (p. 6). Section 5.1 names the exceptions in Table 2 and gives ρ2=1.0000\rho_2=1.0000 (p. 7).
  • Table 2 gains a column with the second-half change in cycle-mean loss for all 33 configurations (p. 34).
  • Section 5.5 gives the downstream-step ladder at depth 8, including equal steps for every layer, and says what the freezing switch does not show (pp. 9–10).
  • Section 6 no longer cites Lewkowycz et al. (2020) for the link between catapults and AGOP alignment, and the conclusion adds "under stated conditions" (p. 10).
  • Appendix C replaces a stale sentence by a pointer to Table 2 (p. 33), and Appendix D.3 states the reused seed blocks (p. 42). Appendix F narrows item (ii) of its working principle (p. 47), says "mostly finished" in F.2 (p. 48) and states a limitation of the alignment measure in F.4 (p. 51).

With your agreement, we will add an appendix with the other new experiments of our responses, whose code is in the anonymous folder. The camera-ready version will also recompute Table 3 with the training head and place model-specific setups next to their results.

Anonymous folder: link

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.