- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether spectral flattening emerges during optimization for long-horizon terminal objectives. It compares three classes: spectral-power updates, rank-wise gains, and a per-parameter MLP optimizer. Experiments primarily use deep linear factorization. The paper reports that longer horizons generally favor flatter updates at small step sizes.
Critical strengths
None
Critical weaknesses
-
The theoretical formulation does not match the normalized optimizer in the experiments. The empirical results in Section 4 are based on Equation 7, where rescaling via the Frobenius norm is applied at every optimization step. On the other hand, the quadratic model in Equation 15 applies that rescaling only at initialization; the subsequent steps use unnormalized updates. Applying the normalization with fixed yields $ \dot z_i(t) = -\kappa_iq_i z_i(t) \frac{\sqrt{\sum_j\kappa_j^2z_j(t)^2}}{\sqrt{\sum_jq_j^2\kappa_j^2z_j(t)^2}} $. In general, the normalization factor is not one (even in the continuous-time limit). Therefore, the terminal objective at T cannot be identified with that of the normalized optimizer employed in the experiments without further justification. This matters because Theorem 1 is the key finding explaining why longer horizons induce spectral flattening.
-
How practical is the favoring of flatter updates for longer horizons under tunable step sizes? In the experiments of deep linear factorization, longer horizons favor flatter updates at the three lowest step sizes . However, according to Section 4.3, every method chooses the largest available step sizes at every depth, width, and horizon. The update families of spectral power and rank-wise gain already favor flat updates at ; MLP optimizers no longer exhibit monotone flattening with ; the model may favor flat updates to keep updates safe yet large. Its practical implications are insufficiently explored: The searched step sizes are at most the stability edge for SGD, yet flatter updates could be stable even at larger step sizes.
Questions for authors
-
Could the authors derive Equation 15 while applying the Frobenius normalization at every step?
-
Could the authors extend the deep-linear step size search for each update family, including rates above SGD’s stability edge? How does the selected spectral shape vary with the horizon after jointly selecting the step size and update shape?
6 comments
Thank you for your response.
The revision considers two things: a theoretical setting with the normalization constraint at every step and experiments that jointly select the learning rate and the update shape over a wider range of learning rates. My concern is whether these results properly and directly support the central argument that a longer horizon favors a flatter update in meta-optimization.
W1, Q1. The original analysis characterized the spectrum of the optimal update as a function of the effective horizon in a simplified model, with normalization applied only at initialization. For spectral power-type updates, the revised analysis computes the derivative of terminal loss with respect to at GD and empirically examines the selected power (Appendices F.2 and G). The derivative at is positive on average over validation tasks, indicating that a small decrease in from 1 would reduce terminal validation loss. But this can hold at both short and long horizons even if the same minimizes loss at both horizons. Its sign alone does not determine how the optimal changes with .
W2, Q2. For the spectral power-type updates, joint tuning generally selected a learning rate above at longer horizons and an update at least as flat as that for (Table 6). This outcome is also consistent with flattening making a larger step usable (Section 5.4). But for the MLP optimizer, meta-training or evaluation was often unstable at the higher learning rate, and the effective rank of the MLP update relative to the gradient was lower at the longest horizon than at the shortest (Table 7).
In the experiments on deep matrix factorization, the trend toward flatter updates at longer horizons was most consistent across the three update types at small fixed learning rates. When the learning rate and the shape of the update were selected together, there was limited evidence that this trend extended beyond spectral power-type updates.
Follow-up: the claim is a spectral short-horizon bias
Thank you for the careful reading. We think the remaining concerns come from reading the central claim as a monotone trend in , so we restate the claim first.
W1/Q1. What the claim is. The title says that long-horizon meta-objectives induce spectral flattening. "Long" is relative to the shortest horizon; the title does not say that the spectrum flattens monotonically as grows. An objective that only asks how fast the current loss falls is minimized, to first order, by the gradient, which descends most among updates of its length. This is the update of standard training, and in practice it performs worse than Muon. We read this gap as a short-horizon bias of the spectrum, the counterpart of the short-horizon bias for step sizes (Wu et al., 2018, cited in Section 1). Both rest on one trade-off: their long horizon wins through the step size "by making more progress along the low curvature direction"; our prices show that the trade-off also acts on the shape of the update, which no step size can change. Their claim is also short against long: their theory derives the greedy optimum, and the long-horizon schedule is computed numerically at one horizon.
The positive derivative at therefore already supports the claim: at every horizon we evaluate, in all grid settings and up to , the meta-objective prefers an update flatter than the gradient. Its size also depends on the horizon: at the smallest rate it is below at and grows - to -fold by (Figure 6).
Our grid already shows that the trend in need not be monotone: at the top rate the update is flattest at (Section 4.4). A monotone fall of with is what the theory predicts in its persistence regime, . There the quadratic model, which fits no parameter, reproduces it at all grid rates of the three depths at which no direction overshoots (Figure 5a). The characterization in is kept: for small the price depends on and only through (Section 5.2), and the four smallest rates select the same power along one curve in (Figures 3 and 5).
W2/Q2, the power family with the step size tuned. The paper fixes the step size and constrains the update length (Section 3) to isolate one variable: how the horizon changes the shape of an update of a given size. When the step size is selected as well, the selection also trades shape against step size and stability, so several factors act at once. That is not the setting in which the claim is made. We added the joint selection at your request as a test beyond it, and we report what it shows rather than resting the claim on it.
The grid of Figure 1 can be read in three ways. Up a column, at a fixed horizon, the step size grows: at the selected power falls from at the smallest rate to – at the top rate. A one-step stability account sees only this reading, a larger step favoring flattening at once, which is the overshoot regime of the same price (Section 5.2, Corollary 2). Along a row, at a fixed step size at which no direction overshoots, the horizon grows; this is the reading our claim is about, and a stability account is silent on it, since the state and the step size already fix it. Joint selection moves along both at once, since each horizon picks its own row, so its path mixes the two effects: flattening along it can come from either, and so can its absence. It therefore cannot separate the horizon from the step size; the row can, and that is where the claim is stated.
The controls follow the same work. Wu et al. measure the bias against the greedy choice where it arises (stochastic, ill-conditioned problems), bound the learning rate to exclude unstable solutions, fix the momentum and the decay time constant so that only the learning-rate schedule is selected, and leave out their 20k-step meta-optimization runs, citing optimization difficulties outside their focus. Our fixed step size, length constraint, and treatment of meta-training beyond the edge follow the same logic.
Beyond the power family. At the fixed rates where the claim is made, the rank-wise gains and the MLP optimizer also flatten more at longer horizons, as you note (Figure 1), and the criterion of Theorem 4 covers any family meeting the Assumptions of Section 5, as all three do.
W2/Q2, the MLP optimizer. Making the meta-training of learned optimizers stable is not the topic of the paper. We use the MLP optimizer as a probe of what a long-horizon objective prefers when the update has no spectral prior. It is meta-trained by backpropagating the terminal loss through the full unroll. Beyond the stability edge, meta-gradients through an unstable inner loop are known to explode (Metz et al., 2019, Understanding and Correcting Pathologies in the Training of Learned Optimizers).
Response to Reviewer CB6g
We thank the reviewer; both points led to substantial changes in the paper.
W1/Q1. Can Eq. 15 be derived with Frobenius normalization at every step?
The revised theory imposes the normalization at every step, as in the experiments: every update map keeps ∥Dϕ(G)∥F=∥G∥F; differentiating gives ⟨G,B(G)⟩=0, so at GD a deformation only redistributes the update.
Relation to the former analysis. Its credit lemma (former Lemma 1, Corollary 1) priced mode i by ρi2H−1, ρi=1−ηκi at GD, so persistent modes gain relative to fast ones as H grows. This is the t=0 price of Theorem 3 on a quadratic diagonal in the gradient's singular basis. To first order, per-step normalization only removes the part of a deformation that rescales all modes together, so the price of moving energy between modes holds for the normalized optimizer. Only the closed-form optimum far from GD needed the constraint at initialization alone. The mechanism and its predictions (flattening grows with H; at small rates only ηH matters) carry over; we drop the closed-form p⋆.
Being first order, the results fix the side of GD on which the optimum lies, not p⋆; exact derivatives along the normalized rollouts connect them to the selected p⋆:
W2/Q2. Go beyond GD's stability edge and select step size and shape jointly.
We agree that flattening can pay by allowing a larger stable step: the large-step regime of the same price (Section 5.2; Corollary 2). Below the edge the one-step price gap is small and the horizon amplifies it; a direction with ηq>1 has a negative price at every horizon, so the gap is of order one already at H=1: the reviewer's "safe yet large" mechanism. The exact criterion at H=1 is 0.27–0.42 at the top rate and below 10−3 at the smallest.
The top rate of Figure 1. At a fixed top rate the update is flattest at H=1, and p⋆ rises by 0.10–0.15 by H=20. The two regimes of the price explain this: at H=1 overshoot makes the selection flatter than at any small rate, and at longer H it settles at the plateau every rate reaches at ηH≥0.4 (Appendix G). The pressure to flatten at GD does not fall: at the top rate the exact criterion is larger at H=20 than at H=1, and every selected update stays flatter than the gradient.
New experiment (Appendix H, Table 6, Figure 15). Rates extended to 16ηtop; (η,p) selected jointly at every depth and H≤100.
A real network. In a GPT on FineWeb from random initialization, joint selection picks p=0.25, 0.5 at H≤3 and the Muon direction (p=0) at every H≥10; at H=1, p=0 at its own rate loses to SGD on every seed (Appendix D, Table 3).
Learned families (Table 7). Beyond the edge meta-training often diverges; the joint choice mostly keeps ηtop and beats tuned GD, except at L=4, H≥5.
Revised claim. Below the edge, longer horizons flatten the update at a fixed rate; once η is tuned, as the reviewer asks, they choose the larger step with an update at least as flat. The abstract and introduction now state these conditions (title: see the general response).