- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies how training affects the gap between the product of spectral norms Lipschitz bound and empirical Jacobian lower bounds. It analyzes the effect of different optimizers and spectral-control methods, reporting observations on gap growth and rank collapse, and explores remedies like post-training repair and optimizer switching.
Critical strengths
The paper systematically investigates tightness across training settings and datasets, providing empirical evidence while acknowledging tuning sensitivity and accuracy costs.
Critical weaknesses
-
Limited theoretical novelty. Proposition 1 directly applies the standard margin-based certificate. Equation (4) gives an algebraic factorization, and Propositions 2–3 use familiar homogeneity and rescaling arguments. The paper should distinguish its new contributions from existing work, including Kuang et al. (2026).
-
Limited generalizable guidance. Optimizer ordering changes after learning-rate tuning, while collapse depends on architecture, temperature, and learning rate. The update-rank discussion in §3.3 does offer a partial explanation based on known properties of orthogonalized updates, but gives little guidance beyond experimental tuning.
-
Unclear practical benefit. The gap reduction from approximately 3.2 million to seven accompanies a clean-accuracy drop from approximately 88% to 55%, with only 0.15% certified accuracy at the reported radius. Practical claims need comparisons at similar accuracy and training budgets.
-
Some conclusions not sufficeintly justified. For example, “Never pay for post-hoc repair” rests on restricted rescaling arguments and selected experiments, without reporting clean accuracy after weight-only repair in the main text. Optimizer switching also lacks a clear comparison with using the destination optimizer from initialization. "Tightness alone is insufficient, as expected." The certificate depends on both margin and the Lipschitz bound. Even an exact global Lipschitz constant may give a conservative local classification certificate. The experiments should better separate limitations from bound estimation, the certificate itself, and weak robustness.
Questions for authors
-
Does optimizer switching improve clean accuracy, gap, and certified accuracy over training with either optimizer from initialization under comparable budgets? Is the benefit consistent across seeds and switching times?
-
How do clean and certified accuracy change after weight-only capping? Can repair preserve useful classification performance?
-
Beyond the known geometry of Muon updates, what new mechanism explains rank collapse? Can it predict when collapse occurs or tuning prevents it?
-
What supports the broad recommendation against post-training repair?