acceptodds
Under review as a conference paper at ICLR 2027

Long-Horizon Meta-Objectives Induce Spectral Flattening

Abstract

Muon accelerates matrix optimization by flattening the singular spectrum of gradient updates. But why should spectral flattening be a good optimization strategy, and can it arise without being explicitly prescribed? We approach these questions from a meta-optimization perspective, where update transformations are evaluated by their consequences for future learning rather than immediate descent alone. In controlled matrix-factorization problems, we find that extending the optimization horizon systematically shifts the preferred update toward flatter spectra. This shift reflects a long-term trade-off: spectrum-aware updates may sacrifice short-term progress along dominant directions while improving the optimization of modes that gradient descent leaves behind. We connect this behavior to the evolution of error modes and show how spectral flattening creates more balanced learning dynamics. Our results interpret Muon-like flattening as an emergent strategy for improving the geometry of subsequent optimization. This perspective suggests a path toward learning spectrum-aware optimizers directly from long-horizon objectives.

Primary area:
optimization
Keywords:
meta-optimization, Muon, spectral flattening, deep linear networks, optimizer analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CB6gOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether spectral flattening emerges during optimization for long-horizon terminal objectives. It compares three classes: spectral-power updates, rank-wise gains, and a per-parameter MLP optimizer. Experiments primarily use deep linear factorization. The paper reports that longer horizons generally favor flatter updates at small step sizes.

Critical strengths

None

Critical weaknesses
  1. The theoretical formulation does not match the normalized optimizer in the experiments. The empirical results in Section 4 are based on Equation 7, where rescaling via the Frobenius norm is applied at every optimization step. On the other hand, the quadratic model in Equation 15 applies that rescaling only at initialization; the subsequent steps use unnormalized updates. Applying the normalization with fixed qiq_i yields $ \dot z_i(t) = -\kappa_iq_i z_i(t) \frac{\sqrt{\sum_j\kappa_j^2z_j(t)^2}}{\sqrt{\sum_jq_j^2\kappa_j^2z_j(t)^2}} $. In general, the normalization factor is not one (even in the continuous-time limit). Therefore, the terminal objective at T cannot be identified with that of the normalized optimizer employed in the experiments without further justification. This matters because Theorem 1 is the key finding explaining why longer horizons induce spectral flattening.

  2. How practical is the favoring of flatter updates for longer horizons under tunable step sizes? In the experiments of deep linear factorization, longer horizons favor flatter updates at the three lowest step sizes η\eta. However, according to Section 4.3, every method chooses the largest available step sizes at every depth, width, and horizon. The update families of spectral power and rank-wise gain already favor flat updates at H=1H=1; MLP optimizers no longer exhibit monotone flattening with HH; the model may favor flat updates to keep updates safe yet large. Its practical implications are insufficiently explored: The searched step sizes are at most the stability edge for SGD, yet flatter updates could be stable even at larger step sizes.

Questions for authors
  1. Could the authors derive Equation 15 while applying the Frobenius normalization at every step?

  2. Could the authors extend the deep-linear step size search for each update family, including rates above SGD’s stability edge? How does the selected spectral shape vary with the horizon after jointly selecting the step size and update shape?

6 comments
AuthorsOpenReview ↗

Response to Reviewer CB6g

We thank the reviewer; both points led to substantial changes in the paper.

W1/Q1. Can Eq. 15 be derived with Frobenius normalization at every step?

The revised theory imposes the normalization at every step, as in the experiments: every update map keeps ∥Dϕ(G)∥F=∥G∥F\|D_\phi(G)\|_F=\|G\|_F; differentiating gives ⟨G,B(G)⟩=0\langle G,B(G)\rangle=0, so at GD a deformation only redistributes the update.

  • Theorem 1 (exact, every differentiable loss and horizon): ∂ϕLH∣ϕ=0=−η∑t=0H−1⟨λt+1,B(Gt)⟩\partial_{\phi}\mathcal L_H\vert_{\phi=0}=-\eta\sum_{t=0}^{H-1}\langle\lambda_{t+1},B(G_t)\rangle, with terminal credits λt\lambda_t from a backward recursion. It matches automatic differentiation through the normalized rollout to rounding error (Appendix F.2).
  • Theorem 3: on a quadratic loss, the credit prices update energy at curvature qq by (1−ηq)2(H−t)−1(1-\eta q)^{2(H-t)-1}, with the constraint at every step and at every step size; for the reviewer's ODE this is exactly the first-order term at GD.
  • Theorem 4: for the power family, these prices add up to a covariance criterion.

Relation to the former analysis. Its credit lemma (former Lemma 1, Corollary 1) priced mode ii by ρi2H−1\rho_i^{2H-1}, ρi=1−ηκi\rho_i=1-\eta\kappa_i at GD, so persistent modes gain relative to fast ones as HH grows. This is the t=0t=0 price of Theorem 3 on a quadratic diagonal in the gradient's singular basis. To first order, per-step normalization only removes the part of a deformation that rescales all modes together, so the price of moving energy between modes holds for the normalized optimizer. Only the closed-form optimum far from GD needed the constraint at initialization alone. The mechanism and its predictions (flattening grows with HH; at small rates only ηH\eta H matters) carry over; we drop the closed-form p⋆p^\star.

Being first order, the results fix the side of GD on which the optimum lies, not p⋆p^\star; exact derivatives along the normalized rollouts connect them to the selected p⋆p^\star:

  • The criterion favors flattening in all 90 grid settings and up to H=200H=200 (Appendix G.1).
  • Along the rollout that applies p⋆p^\star at every step, we differentiate the loss with respect to each step's own power: in all 20 settings only steps in the first 22–50% of the run prefer a flatter update and every later step one closer to GD, so the constant p⋆p^\star balances the two (Appendix G.2).

W2/Q2. Go beyond GD's stability edge and select step size and shape jointly.

We agree that flattening can pay by allowing a larger stable step: the large-step regime of the same price (Section 5.2; Corollary 2). Below the edge the one-step price gap is small and the horizon amplifies it; a direction with ηq>1\eta q>1 has a negative price at every horizon, so the gap is of order one already at H=1H=1: the reviewer's "safe yet large" mechanism. The exact criterion at H=1H=1 is 0.27–0.42 at the top rate and below 10−310^{-3} at the smallest.

The top rate of Figure 1. At a fixed top rate the update is flattest at H=1H=1, and p⋆p^\star rises by 0.10–0.15 by H=20H=20. The two regimes of the price explain this: at H=1H=1 overshoot makes the selection flatter than at any small rate, and at longer HH it settles at the plateau every rate reaches at ηH≥0.4\eta H\ge0.4 (Appendix G). The pressure to flatten at GD does not fall: at the top rate the exact criterion is larger at H=20H=20 than at H=1H=1, and every selected update stays flatter than the gradient.

New experiment (Appendix H, Table 6, Figure 15). Rates extended to 16ηtop16\eta_{\rm top}; (η,p)(\eta,p) selected jointly at every depth and H≤100H\le100.

  1. Rate. H=1H=1 keeps ηtop\eta_{\rm top} at every depth; every H≥5H\ge5 picks 2ηtop2\eta_{\rm top} (4ηtop4\eta_{\rm top} and above diverge once H≥2H\ge2): only longer horizons take the larger stable step.
  2. Q2: shape against HH. At every depth, every H≥5H\ge5 is at least as flat as H=1H=1, strictly in 10 of 15 settings, and strictly flatter than the best power at η≤ηtop\eta\le\eta_{\rm top}. It is not monotone: H=2H=2 is sharper than H=1H=1 at L=2,3L=2,3, and at L=2L=2, pp rises from 0.25 at H=5H=5 to 0.45 at H≥50H\ge50.
  3. Practical value. The joint choice beats GD tuned on the same rates on 90–100% of 96 held-out tasks in every setting (divergence counts as a loss; median −14 to −21% at H=1H=1, −37 to −99.9% at H≥5H\ge5).

A real network. In a GPT on FineWeb from random initialization, joint selection picks p=0.25p=0.25, 0.5 at H≤3H\le3 and the Muon direction (p=0p=0) at every H≥10H\ge10; at H=1H=1, p=0p=0 at its own rate loses to SGD on every seed (Appendix D, Table 3).

Learned families (Table 7). Beyond the edge meta-training often diverges; the joint choice mostly keeps ηtop\eta_{\rm top} and beats tuned GD, except at L=4L=4, H≥5H\ge5.

Revised claim. Below the edge, longer horizons flatten the update at a fixed rate; once η\eta is tuned, as the reviewer asks, they choose the larger step with an update at least as flat. The abstract and introduction now state these conditions (title: see the general response).

Thank you for your response.

The revision considers two things: a theoretical setting with the normalization constraint at every step and experiments that jointly select the learning rate and the update shape over a wider range of learning rates. My concern is whether these results properly and directly support the central argument that a longer horizon favors a flatter update in meta-optimization.

W1, Q1. The original analysis characterized the spectrum of the optimal update as a function of the effective horizon T=ηHT = \eta H in a simplified model, with normalization applied only at initialization. For spectral power-type updates, the revised analysis computes the derivative of terminal loss with respect to pp at GD (p=1)(p=1) and empirically examines the selected power (Appendices F.2 and G). The derivative at p=1p=1 is positive on average over validation tasks, indicating that a small decrease in pp from 1 would reduce terminal validation loss. But this can hold at both short and long horizons even if the same pp minimizes loss at both horizons. Its sign alone does not determine how the optimal pp changes with HH.

W2, Q2. For the spectral power-type updates, joint tuning generally selected a learning rate above ηtop\eta_{\text{top}} at longer horizons and an update at least as flat as that for H=1H=1 (Table 6). This outcome is also consistent with flattening making a larger step usable (Section 5.4). But for the MLP optimizer, meta-training or evaluation was often unstable at the higher learning rate, and the effective rank of the MLP update relative to the gradient was lower at the longest horizon than at the shortest (Table 7).

In the experiments on deep matrix factorization, the trend toward flatter updates at longer horizons was most consistent across the three update types at small fixed learning rates. When the learning rate and the shape of the update were selected together, there was limited evidence that this trend extended beyond spectral power-type updates.

AuthorsOpenReview ↗

Follow-up: the claim is a spectral short-horizon bias

Thank you for the careful reading. We think the remaining concerns come from reading the central claim as a monotone trend in HH, so we restate the claim first.

W1/Q1. What the claim is. The title says that long-horizon meta-objectives induce spectral flattening. "Long" is relative to the shortest horizon; the title does not say that the spectrum flattens monotonically as HH grows. An objective that only asks how fast the current loss falls is minimized, to first order, by the gradient, which descends most among updates of its length. This is the update of standard training, and in practice it performs worse than Muon. We read this gap as a short-horizon bias of the spectrum, the counterpart of the short-horizon bias for step sizes (Wu et al., 2018, cited in Section 1). Both rest on one trade-off: their long horizon wins through the step size "by making more progress along the low curvature direction"; our prices show that the trade-off also acts on the shape of the update, which no step size can change. Their claim is also short against long: their theory derives the greedy optimum, and the long-horizon schedule is computed numerically at one horizon.

The positive derivative at p=1p=1 therefore already supports the claim: at every horizon we evaluate, in all 9090 grid settings and up to H=200H=200, the meta-objective prefers an update flatter than the gradient. Its size also depends on the horizon: at the smallest rate it is below 10−310^{-3} at H=1H=1 and grows 4141- to 9090-fold by H=20H=20 (Figure 6).

Our grid already shows that the trend in HH need not be monotone: at the top rate the update is flattest at H=1H=1 (Section 4.4). A monotone fall of p⋆p^\star with HH is what the theory predicts in its persistence regime, ηq<1\eta q\lt 1. There the quadratic model, which fits no parameter, reproduces it at all 1111 grid rates of the three depths at which no direction overshoots (Figure 5a). The characterization in T=ηHT=\eta H is kept: for small ηq\eta q the price depends on η\eta and HH only through ηH\eta H (Section 5.2), and the four smallest rates select the same power along one curve in ηH\eta H (Figures 3 and 5).

W2/Q2, the power family with the step size tuned. The paper fixes the step size and constrains the update length (Section 3) to isolate one variable: how the horizon changes the shape of an update of a given size. When the step size is selected as well, the selection also trades shape against step size and stability, so several factors act at once. That is not the setting in which the claim is made. We added the joint selection at your request as a test beyond it, and we report what it shows rather than resting the claim on it.

The grid of Figure 1 can be read in three ways. Up a column, at a fixed horizon, the step size grows: at H=1H=1 the selected power falls from 0.950.95 at the smallest rate to 0.200.20–0.450.45 at the top rate. A one-step stability account sees only this reading, a larger step favoring flattening at once, which is the overshoot regime of the same price (Section 5.2, Corollary 2). Along a row, at a fixed step size at which no direction overshoots, the horizon grows; this is the reading our claim is about, and a stability account is silent on it, since the state and the step size already fix it. Joint selection moves along both at once, since each horizon picks its own row, so its path mixes the two effects: flattening along it can come from either, and so can its absence. It therefore cannot separate the horizon from the step size; the row can, and that is where the claim is stated.

The controls follow the same work. Wu et al. measure the bias against the greedy choice where it arises (stochastic, ill-conditioned problems), bound the learning rate to exclude unstable solutions, fix the momentum and the decay time constant so that only the learning-rate schedule is selected, and leave out their 20k-step meta-optimization runs, citing optimization difficulties outside their focus. Our fixed step size, length constraint, and treatment of meta-training beyond the edge follow the same logic.

Beyond the power family. At the fixed rates where the claim is made, the rank-wise gains and the MLP optimizer also flatten more at longer horizons, as you note (Figure 1), and the criterion of Theorem 4 covers any family meeting the Assumptions of Section 5, as all three do.

W2/Q2, the MLP optimizer. Making the meta-training of learned optimizers stable is not the topic of the paper. We use the MLP optimizer as a probe of what a long-horizon objective prefers when the update has no spectral prior. It is meta-trained by backpropagating the terminal loss through the full unroll. Beyond the stability edge, meta-gradients through an unstable inner loop are known to explode (Metz et al., 2019, Understanding and Correcting Pathologies in the Training of Learned Optimizers).

Reviewer_mDdeOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper provides a meta-optimization perspective to understand why spectral flattening is preferred in optimizers like Muon. The authors first establish three learning-to-optimize models that can optimize the spectral properties of gradients. The experimental results in deep linear factorization show that in long-horizon optimization, the parameterized learning-to-optimize models would prefer flatten updates. However, in MLP model, the result shows the learned direction is more important than the spectrum. Then the authors extend the experiment to Newton–Schulz and nonlinear network settings. Also, the authors analyze the solvable decoupled model under strict assumptions, and the result leads to the flat spectrum.

Critical strengths

This paper proposes a new meta-optimization perspective to understand why in large model training, spectral flattening updates are preferred. This perspective links spectral flattening updates to algorithm design in long-horizon optimization, showing that longer optimization favors more flattening updates. This new perspective explains part of the success of Muon and may provide more insights.

Critical weaknesses

Most experimental results support that longer optimization favors more flattening updates. However, in MLP model, the results show that the learned direction is more important than the spectrum. The theoretical result only analyzes a solvable decoupled model under strict assumptions.

Questions for authors

Questions

  1. In the learned rank-wise gains model, why do we impose the gauge 1⊤θ=0\mathbf{1}^\top \theta=0? Note that we have done clipping for gig_i, and it impacts the global shift in θ\theta.

  2. In MLP model, the result shows that the learned direction is more important than the spectrum. I think this result contradicts that we need flattening updates in long-horizon optimization.

  3. I think it would be valuable to discuss and explain why in short-horizon optimization, it does not prefer flattening updates.

  4. In the training of CIFAR-10 (Figure 9), the model only chooses 3 NS iterations even with 400 horizons. Could you provide when the 5 NS iterations would be chosen? Also there is difference between flatenning spectrum and NS iteration, for example, for X0=diag(0.8,0.6)X_0=\mathrm{diag} (0.8,0.6), which has unit Frobenius norm, applying f(s)=3.4445s−4.7750s3+2.0315s5f(s)=3.4445s-4.7750s^3+2.0315s^5 yields effective ranks of 1.979631.97963, 1.990031.99003, 1.986361.98636, and 1.976791.97679 after 0–3 iterations, where 0 NS iteration has larger effective ranks than 3 NS iterations.

  5. The analysis in Section 5 requires quite strict assumptions and a function family. I think it would be better to clarify all the assumptions used at the beginning and explain them.

Typos

  1. Line 171 clamp should be clip
  2. Line 427 const should be constant
1 comment
AuthorsOpenReview ↗

Response to Reviewer mDde

We thank the reviewer; each question led to a change in the paper.

Q1. Why the gauge 1⊤θ=0\mathbf 1^\top\theta=0, given that clipping affects the global shift? (Appendix A)

We removed the clip, a safeguard that no trained rule reaches (the largest knot magnitude is 3.82, below the bound of 4). Without it, θ↦θ+c1\theta\mapsto\theta+c\mathbf 1 is an exact symmetry: the shift multiplies every singular value by ece^c, which the length constraint undoes, so the meta-objective is flat along 1\mathbf 1. Adam then takes identical steps with or without the gauge, which only removes this flat direction and changes no trained rule.

Q2. The MLP's gains come mostly from directions. Does that contradict the need for flattening? (Appendix C)

No. The experiment asks whether flattening emerges without a spectral prior, and it does: the update is flatter than the gradient in every configuration, and the learned spectrum alone beats GD at every horizon. Most of the gain is rotational because that is where the terminal credit lies. To first order, a length-preserving change of the update lowers the terminal loss by its alignment with the terminal credit, and a spectral map reaches only the credit's part in span{uivi⊤}\mathrm{span}\{u_iv_i^\top\}. Along GD on 32 tasks, that part holds 51%, 21%, and 8% of the credit energy at H=1,5,20H=1,5,20 (6% for a random direction). The trained optimizer follows both parts (cosine 0.92 with the spectral part at H≤5H\le5 and 0.71 at H=20H=20; 0.85 and 0.65 with the rest at H=5,20H=5,20), and the spectral part supplies 100%, 42%, and 19% of its first-order gain, in line with the 11% the spectrum-only intervention recovers at H=20H=20.

Q3. Why do short horizons not prefer flattening? (Section 5.2, Appendix G)

Theorem 3 prices energy at curvature qq by (1−ηq)2(H−t)−1(1-\eta q)^{2(H-t)-1}. At small η\eta and H=1H=1, the price gap between two directions is only η(qi−qj)\eta(q_i-q_j), so GD is nearly optimal; the horizon amplifies the gap, and in the small-step limit η\eta and HH enter only through ηH\eta H. Both predictions hold. At the smallest rate the exact criterion is below 10−310^{-3} at H=1H=1 and grows 41- to 90-fold by H=20H=20 (Figure 6). Below the edge, (η,H)=(0.005,20)(\eta,H)=(0.005,20), (0.01,10)(0.01,10), and (0.02,5)(0.02,5) all select pp between 0.55 and 0.60 (Figure 3). With large steps, flattening pays already at H=1H=1 through stability (p⋆=0.10p^\star=0.10 at η=0.2\eta=0.2, Figure 13).

Q4. When are 5 Newton–Schulz iterations chosen, and is the iteration count a proxy for flatness? (Appendix D, Tables 1–2)

We select the iteration count on CIFAR because it is what Muon runs, and an exact spectral power would need an SVD of every matrix at every step. As the reviewer notes, the quintic overshoots, so the count is a proxy for flatness, not an exact measure. We checked the proxy with the paper's own measure, the effective rank of NSk(G)\mathrm{NS}_k(G) divided by the rank (1 = flat), on the actual Nesterov matrices of the CIFAR runs. The effect in the reviewer's example does occur on nearly flat matrices, but lowers the effective rank by at most 0.004.

At step 100, for k=0,…,5k=0,\dots,5:

  • ResNet-18: 0.68, 0.72, 0.86, 0.98, 0.99, 0.99. Three iterations already make the update nearly flat, so further iterations add little.
  • TinyViT: 0.28, 0.35, 0.57, 0.83, 0.98, 0.98. Three iterations leave a tail.

We reran CIFAR selecting the learning rate and kk jointly at each horizon, with TinyViT up to H=2000H=2000 (Table 1). TinyViT selects k=5k=5 at every horizon except H=5H=5, where k=3k=3 leads by 0.0005. In held-out loss k=5k=5 beats k=3k=3 at every horizon except H=100H=100, where it loses by 0.006, and on all 12 tasks at H=1000H=1000 and 2000, by 0.023 [0.018,0.029][0.018,0.029] and 0.036 [0.029,0.043][0.029,0.043]. ResNet-18, whose spectra are flat after three iterations, selects k=2k=2–3. Read at each fixed rate (Figure 10, top), the selected count never decreases with the horizon at any rate up to 0.1 on either architecture, rising from k=0k=0 at H=1H=1 to 1–5 at the longest horizon at the three smallest rates; at 0.3 and above, k=5k=5 wins already at H=1H=1. Short horizons select such large rates, which is why the joint choice is flat from H=1H=1 on.

Q5. State the assumptions up front. (Section 5, "Assumptions")

A new paragraph opens Section 5. The update map coincides with gradient descent at its parameters' origin, obeys the length constraint, and is smooth in those parameters; the three families all do. No result needs a decoupled model.

Typos. Both are fixed.

Reviewer_oRNWOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a way of studying Muon and related optimizers from the perspective of meta learning -- they design an outer loop optimizer that operates over longer horizons, and examine what strategy such a meta procedure converges to. Results show that the procedure indeed learns to flatten the singular value spectrum of the matrix updates. This holds first when assuming a power family, and also when rank-wise gains are freely learned. Finally, there are findings that longer horizons desire flatter spectrum. An entirely learned MLP optimizer attributes 10% of the gains to flattening, but other gains come from changing the directions of the update themselves.

Critical strengths

This paper provides an interesting angle to study preconditioning optimizers by defining a meta outer loop and examining what this loop converges to. While the results are not particularly surprising (they agree directionally that Muon's flattening procedure makes sense), these results provide a nice isolated reproduction of the strength of such optimizers. The experiments are conducted carefully and with rigor, focusing more on a scientific conclusion rather than claiming a new method or performance.

The paper is well-written, including motivations and technical details.

Critical weaknesses

A main weakness is that it is unclear what the practical takeaway from the results are, seeing that results indicate that the form used in Muon is indeed close to optimal for long horizon training. The paper could be strengthened by a more fleshed out discussion that does not merely rehash what was described in the paper, but could mention future work and takeaways from the conclusions found.

Questions for authors

Are the authors confident that the MLP outer optimizer is trained for sufficiently long and/or has enough capacity to be a reasonable approximation for an "optimal" inner optimizer? What results indicate this?

Are there takeaways for how one would want to design a Muon-like algorithm near the end of training, given that non-flat spectra seem to be preferred at shorter planning horizons?

2 comments
AuthorsOpenReview ↗

Response to Reviewer oRNW

We thank the reviewer for the supportive review. Section 6 has been rewritten. It no longer restates results; it now gives the practical implication with its evidence, and the limitations with future work.

W. What is the practical takeaway, if Muon is already close to optimal? (Section 6)

The paper does not propose a new optimizer. It explains why Muon's shape is right for long training, and the explanation has a direct consequence: the spectral shape should be chosen at the horizon it will be used for.

At small step sizes, short-horizon selections and one-step arguments understate the value of flattening. This matters most for learned optimizers, which are meta-trained on truncated unrolls:

  • At the three smallest rates, the MLP optimizer's update is flatter the longer the horizon it was trained for (Section 4.3, Figure 1).

A short unroll therefore selects a less flat spectrum. This is the spectral counterpart of the short-horizon bias for step sizes (Wu et al., 2018).

Section 6 also states the limitations: the theory is first order around GD, and the nonlinear experiments are small. As future work it proposes distilling meta-learned optimizers into symbolic update rules, and adding terms such as robustness or generalization to the meta-objective to see which rules emerge.

Q1. Is the MLP optimizer trained long enough, with enough capacity? (new experiment)

We retrained it in the reference setting at hidden width 128, at 1500 outer iterations, and at both, three runs each, at H∈{1,5,20}H\in\{1,5,20\}. Every run beats GD and raises the effective rank. Held-out gains at H≥5H\ge5 change by +0.8% to +6.4%, and the credit shares move by at most 0.03. Rerunning the original setting matches the submitted held-out losses within 2%. The conclusions do not reflect an undersized or undertrained optimizer.

Q2. Takeaways for a Muon-like algorithm near the end of training? (Section 6; Appendices F–G, Tables 4–5; Appendix D, Table 3)

Our main takeaway is that the spectral shape is a schedule, like the learning rate. Theorem 3 prices update energy at curvature qq by (1−ηq)2(H−t)−1(1-\eta q)^{2(H-t)-1}. A step taken with much training ahead values the slow directions that GD neglects, because progress along them persists, and flattening buys that progress. Near the end this future is gone, and the price falls back to the one-step price, under which GD is nearly optimal at small rates. At small rates the rate and the remaining steps enter only through η(H−t)\eta(H-t), so learning-rate decay and the approaching end shrink the same effective remaining horizon. This suggests several designs:

  • Decay the shape together with the learning rate. Let flatness follow ηt(T−t)\eta_t(T-t): fully orthogonalized while it is large, relaxing gradually toward the normalized gradient as it shrinks, through a power ptp_t or a blend of the Muon direction with G/∥G∥FG/\|G\|_F. In deep factorization, optimizing a separate power for every step yields exactly this shape: it starts flat and returns gradually to GD, lowering held-out loss against the best constant power by a median of 21–30% (Table 5).
  • Stay flat while the step is large. Near the stability edge flattening pays already at one step, by avoiding overshoot, so a phase at a large rate should stay flat and only the small-rate tail should relax. Our GPT from an AdamW checkpoint shows this split: the selected power follows the rate, from 1 at η≤0.1\eta\le0.1 to 0 at η≥3\eta\ge3 (Figure 11), and with the rate tuned the Muon direction (p=0p=0) wins at every horizon, beating SGD on every seed by 0.007–0.029 (Table 3).
  • Per layer. The criterion of Theorem 4 is a sum over layers, so it can be read layer by layer: a layer whose weak gradient directions carry more credit per unit should stay flat longer.
  • Horizon-aware learned optimizers. Feeding the remaining horizon to a learned optimizer, instead of meta-training it on one truncated unroll, would let it discover such a schedule itself.

Late starts in deep factorization support the premise. After t0t_0 GD steps, with 20 steps remaining p⋆p^\star rises from 0.35 at t0=0t_0=0 to 1.00–1.05 from t0=50t_0=50 on, while with 200 remaining it stays at 0.30–0.60 up to t0=100t_0=100; the exact criterion has the sign of 1−p⋆1-p^\star in all 22 settings where p⋆≠1p^\star\ne1. Section 6 now discusses these designs alongside the trend toward dynamic optimizers that rewrite their own rules and schedules during training (Bai et al., 2026, Reactive Optimizers). Our theory is first order around GD and our networks are small, so these designs are hypotheses; testing them at scale is the next step.

Thank you for the detailed response. As I have already recommended acceptance, I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for careful and constructive reviews. They raised four concerns: whether the theory holds for the length-constrained optimizer of the experiments (CB6g); step sizes stopped at GD's stability edge (CB6g); the learned optimizer gains mostly through directions, and the practical takeaway was unclear (mDde, oRNW); and several design choices needed justification (gauge, Newton–Schulz count, assumptions; mDde). All four led to changes in the revised PDF. Section, table, and appendix numbers below refer to it.

1. Theory for the optimizer we run (Section 5). Every result now holds for update maps that keep the Frobenius norm of the gradient at every step, as in all experiments:

  • Theorem 1, exact for every differentiable loss and horizon: the first-order change of the terminal loss under any deformation of GD is −η∑t⟨λt+1,B(Gt)⟩-\eta\sum_t\langle\lambda_{t+1},B(G_t)\rangle, with terminal credits λt\lambda_t from a backward recursion.
  • Theorem 3: on a quadratic loss this credit prices update energy in curvature eigenspace kk by (1−ηqk)2(H−t)−1(1-\eta q_k)^{2(H-t)-1}, at every step size.
  • Theorem 4: for the power family the prices add up to a covariance criterion.

The criterion is evaluated exactly and favors flattening in all 90 settings (Appendix F.2). A new paragraph at the start of Section 5 states every assumption.

2. Step sizes beyond the edge (Sections 5.2, 5.4; Appendix H, Table 6, Figure 15). We extended the learning rates to 16× the largest stable one and selected step size and shape jointly. One price covers two regimes:

  • Persistence regime: below the edge, flattening grows with the product of step size and horizon.
  • Overshoot regime: at and beyond the edge, overshooting directions are penalized at every horizon.

Joint selection picks flattened spectra at every horizon and beats tuned GD on 90–100% of tasks. A single step keeps the top rate, whereas every H≥5H\ge5 picks twice the top rate with a spectrum at least as flat as at H=1H=1. At a fixed top rate (Figure 1) the selection becomes slightly less flat with HH, returning to the plateau every rate reaches, and the abstract now restricts the fixed-rate claim to rates below the edge. In a GPT on FineWeb, with the rate selected too, H≤3H\le3 keeps a partly flattened update and every H≥10H\ge10 picks the Muon direction (p=0p=0).

3. What the learned optimizer learns (Appendix C). The learned spectrum alone beats GD at every horizon. At H=20H=20, 8% of the terminal credit lies where a spectral map can reach, and retraining at 4× the hidden width or 2.5× the outer iterations does not change the conclusion (replies to mDde and oRNW).

4. Nonlinear networks (Appendix D). CIFAR and the GPT now run a full rate ladder. At every CIFAR rate up to 0.1, longer horizons never select fewer iterations. With the rate selected too, the update beats k=0k=0 at every horizon, each at its own rate, by 0.02–0.03 at H≤5H\le5 and 0.25–0.32 at the longest. Effective rank confirms the count as a flatness proxy.

5. Practical implication and limitations (Section 6, rewritten). The spectral shape should be chosen at the horizon it will be used for: at small step sizes, a short horizon understates the value of flattening, and the MLP optimizer's update is flatter the longer the horizon it was trained for. Section 6 now proposes scheduling the shape with the remaining horizon, like the learning rate. Two new experiments address the end of training:

  • Late starts (Table 4): from a state reached by GD, the preferred spectrum follows the remaining horizon.
  • Optimized schedule (Table 5): a separate power for every step, optimized in deep factorization, starts flat and rises gradually to GD, lowering held-out loss by a median of 21–30% against the best constant power.

6. Smaller changes. The clip on the rank-wise gains, which no trained rule reaches, is removed, so the gauge removes an exact symmetry (Appendix A). The typos are fixed. Section 5 is reorganized into one argument with an overview, and the appendix is deduplicated.

Sharpened claim. One set of prices, (1−ηq)2(H−t)−1(1-\eta q)^{2(H-t)-1}, covers every step size. At small rates, longer horizons favor flatter updates because they value directions that GD leaves in place, and in deep matrix factorization those directions are weak in the gradient. Near the edge, overshoot makes flat updates pay already at H=1H=1; with the step size tuned, longer horizons take a larger step together with an update at least as flat. We keep the title because it names what one-step accounts lack, flattening induced by the horizon; it is a qualified claim whose conditions the abstract now states, and the same prices also cover the large-step regime (Appendix I).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.