- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how coupled spectral optimizers shape implicit curvature dynamics in a quadratic fast–slow matrix model. For persistent oscillations, it characterizes which fast–slow update pairs induce a scalar curvature objective, highlighting power-update compatibility and the special role of cube-root updates under decay. In a Gaussian small-step regime, it shows that the fast matrix has a positive averaged restoring response despite negative instantaneous spectral responses, and that finite covariance fluctuations can destroy scalar-potential representations that normalizing after averaging would preserve. Measurements from a Qwen3-0.6B trajectory motivate the analysis.
Critical strengths
The paper gives a precise account of when coupled optimizers follow a single implicit objective and when they do not. It classifies which pairs of spectral updates make the slow curvature dynamics equivalent to minimizing a scalar objective. Power updates are always compatible, and the cube-root update is uniquely robust once weight decay is added. Moreover, the paper shows that a stable, well-behaved fast equilibrium can still produce slow dynamics that no scalar objective describes (Theorem K.2). Together, these results set clear limits on the common practice of interpreting an optimizer’s implicit bias through a single objective.
Critical weaknesses
-
The results are not connected to how curvature actually evolves in a network. The model treats curvature as a matrix that is trained directly. In a real network, curvature is produced by other parameters, and the paper does not derive what its results imply for them. This substantially limits the interpretation of the results as an explanation of coupled Muon training.
-
The paper does not show that lacking a scalar objective changes training. The negative results establish that the averaged curvature force cannot be expressed as a gradient up to a positive scalar multiplier in the specified geometry, but the paper does not demonstrate how this obstruction affects curvature trajectories or eventual solution selection. The Gaussian effect is proven only for sufficiently wide matrices, and it shrinks in proportion to 1/width with a small constant. The NS5 example is a single hand-picked 3×3 case. The experiments only verify derivative formulas, on Qwen matrices far larger than the analyzed models. A controlled comparison of the two normalization orders, tracking their resulting curvature trajectories, would help establish whether the mathematical distinction produces an appreciable dynamical effect.
Questions for authors
Under what conditions on a parameterization and an optimizer acting on do the induced curvature dynamics satisfy the model analyzed in the paper? Does scalar compatibility or incompatibility in the independent coordinates transfer under those conditions? A concrete derivation for a coupled parameter model would materially strengthen the interpretation.