- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the implicit bias and generalization of spectral gradient descent, an idealized counterpart of Muon, in overparameterized matrix linear classification with corrupted labels. The data model combines a shared low-rank signal with sample-specific rank-one features used for memorization. The authors show that changing the geometry of these features from collapsed to dispersed can reverse the relative generalization performance of GD and SpecGD in certain parameter regimes. They further establish that weak correlations among random dispersed features preserve a small but statistically relevant signal component, whereas exact orthogonality eliminates the signal from the limiting direction. To characterize the direction selected by SpecGD despite non-uniqueness of spectral max-margin solutions, the paper analyzes the dynamics of normalized loss weights through the max-margin dual. Finally, it identifies conditions under which one SpecGD step interpolates the corrupted training labels and generalizes better than the late-stage solution.
Critical strengths
-
The paper identifies shortcut geometry as a concrete factor that can reverse the relative generalization of GD and SpecGD. This might be valuable to the community.
-
The analysis shows that weak random correlations among dispersed shortcuts preserve a small but statistically meaningful signal component, whereas exact orthogonality eliminates the signal from the limiting direction. This is nontrivial and may inspire future optimizer design.
-
Going beyond the spectral margin maximization, the paper characterizes the direction selected by SpecGD despite the non-uniqueness of the primal max-margin problem. The analysis combines normalized loss-weight dynamics, conditional dual convergence, and a model-specific direction-selection identity to determine the signal component needed for the generalization results.
Critical weaknesses
-
The data model assumes rank-one sample-specific shortcuts whose row and column spaces are orthogonal to those of the signal. While this structure enables the signal–memorization decomposition central to the analysis, it remains unclear which conclusions survive when these subspaces overlap or shortcuts have higher rank.
-
The analysis identifies shortcut geometry as a determinant of generalization, but does not provide an observable diagnostic connecting this geometry to practical training problems. Beyond motivating early stopping, it remains unclear how the results could guide optimizer selection or training decisions.
-
While the paper characterizes the first iterate and limiting direction, it does not establish when test risk deteriorates. Finite-time risk bounds would clarify how long favorable early-stage behavior persists and strengthen the early-stopping implications.
Questions for authors
-
Given the contrast between the good generalization of the first iterate and the potentially catastrophic overfitting of the limiting direction, is it feasible within the current dual-dynamical framework to estimate a deterioration timescale?
-
While I appreciate the theoretical focus of the paper, I would be interested in the authors’ perspective on whether these findings suggest any practical diagnostics or training adjustments for Muon users beyond early stopping. I view this as an exploratory question rather than a requirement for the present work.
2 comments
Thank you for the clarifications and additional experiments. I maintain my weak-accept recommendation.
We thank the reviewer for their constructive feedback. We address their comments and questions below.
1. Concern: Does the phenomenon survive overlapping subspaces and higher-rank shortcuts?
Response. Please see the general answer. We have added experiments with overlapping signal–shortcut subspaces (Section I.1). showing that our results persist with overlapping subspaces.
In addition, our experiment below suggests that the same phenomenon persists with shortcuts of relatively small rank R but with a different magnitude. Under CSM the performance of SpecGD degrades when R increases, but under DSM it improves. The performances of GD degrade in both cases when R increases. See Table 1 for the numerical results and below for the formal definition of the extended model:
Table 1. Clean test error (%) of GD and SpecGD under collapsed (CSM) and dispersed (DSM) shortcuts with varying shortcut rank R. Parameter: n=100, D=104, ρ=0.45, λ=50, r=1, σ1=1, T=800, and η=10−2.
Setting: We extend the shortcut model by considering rank-R perturbations
\qquad U_i^\top U_i=V_i^\top V_i=I_R,$$ where $U_i,V_i\in\mathbb R^{d\times R}$ are sampled independently and uniformly from the Stiefel manifold in the subspace orthogonal to the signal. In the collapsed model (CSM), we impose $V_i=V_0$ for all $i$, whereas in the dispersed model (DSM), the $V_i$ are sampled independently. We conjecture that the improved performance of SpecGD with large $R$ is due to the small correlations amplification. **2. Question: Can the onset of test-error deterioration be estimated?** **Response.** Figure 2 shows that deterioration can occur relatively quickly: under DSM, SpecGD interpolates after the first iteration, but its test error increases substantially within the first few hundred iterations. Our new MNIST experiments (Appendix I.2) also exhibit deterioration following interpolation. While our dual-dynamical framework provides a convergence rate toward the limiting classifier (Remark 2), it does not directly characterize the onset of deterioration. Establishing such a timescale would require additional finite-time analysis of the signal and shortcut components, which we leave for future work. **3. Question: What diagnostics or Muon training adjustments follow beyond early stopping?** **Response.** **One practical implication is that the choice of Newton–Schulz approximation may affect generalization, and closer approximation to the exact polar factor is not necessarily better.** Our additional experiments show that, while accurate polar approximations reproduce SpecGD's behavior, the popular quintic Newton–Schulz iteration avoids the deterioration observed under DSM, with only moderate degradation under CSM. This suggests that the approximation scheme itself may be a useful tuning choice for Muon. Regarding diagnostics, our results identify the organization of sample-specific features across singular directions as a potentially relevant property, although translating this insight into a practical diagnostic remains open.