- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies spectral gradient clipping, which clamps individual singular values of a matrix-valued stochastic gradient instead of rescaling the entire gradient. It provides nonconvex convergence guarantees under a bounded centered alpha-moment assumption, and a strict asymptotic excess-loss constant comparison with Frobenius clipping on a quadratic problem with low-rank heavy-tailed noise. EMA and quantile thresholds and a randomized truncated-SVD implementation are proposed. Experiments cover CIFAR-10N, language modeling, and synthetic heavy-tailed regression.
Critical strengths
-
Theoretical comparison: Theorem 3.6 establishes a strict separation in actual asymptotic excess-loss constants under a specified low-rank heavy-tailed quadratic model, rather than merely comparing loose upper bounds. Theorem 3.4 also supplies an explicit stationarity guarantee.
-
Practical evidence: the evaluation spans real human label noise in CIFAR-10N, language modeling, and synthetic heavy-tailed regression. Appendix G documents substantial independent baseline tuning, including 200 CIFAR configurations with three seeds and a held-out validation set. Full/truncated-SVD and wall-clock comparisons address implementation costs.
Critical weaknesses
-
The exact theory and practical operator remain separated. The convergence results analyze exact spectral clipping with specified thresholds. Practical runs use randomized rank-r approximations and layerwise processing together with momentum or Adam or Muon. The paper acknowledges this distinction but does not quantify unprocessed spectral-tail violations or randomized approximation error. Theorem 3.6 establishes a strict advantage for a specified low-rank Haar and Pareto quadratic model. It does not establish a universal advantage for arbitrary noise. This limits the broad practical robustness claim rather than invalidating the theorem.
-
The experiments do not fully isolate spectral selectivity or establish language-model utility. It is unclear whether norm clipping uses matching layerwise granularity. Some gains could therefore arise from avoiding cross-layer rescaling. Language-model results mainly show training loss. Final-run variability and validation loss are not reported clearly and the GPT-2 equal-time comparison shown is for SGDM. The baseline tuning documented in Appendix G is substantial but does not resolve these specific controls. These gaps prevent a stronger acceptance recommendation.
Questions for authors
-
Please clarify whether norm clipping is global or layerwise in each experiment. Report whether the benefit persists with independently tuned and matched layerwise granularity and threshold adaptation. This would isolate spectral selectivity from cross-layer rescaling.
-
In the existing runs please report how often the first unprocessed singular value exceeds the clipping threshold. An approximation residual diagnostic or a small rank-sensitivity comparison would clarify whether the practical operator behaves like the analyzed operator.
-
For the existing GPT-2 runs please report validation loss at matched token and wall-clock budgets. Please specify the number of final evaluation seeds and the variability. These results could change my assessment of practical utility.
My current recommendation is weak acceptance because the strict asymptotic comparison and practical breadth are valuable. The above questions limit confidence in broader empirical claims. I am not requesting a new larger-model benchmark.
2 comments
Thank you for your reply, and I will keep my score.
We thank the Reviewer for their comments and suggestions. We address the concerns below.
We agree that Theorem 3.6 does not establish a universal advantage for arbitrary noise, and we do not claim one. Our method targets heavy-tailed gradient noise concentrated in a few singular directions (Figure 1), and Theorem 3.6 proves a separation in this regime, on a matrix quadratic with low-rank Haar–Pareto noise: we derive a lower bound on the asymptotic excess-loss constant of Frobenius clipping for any fixed positive threshold and any positive coefficient c in the step-size schedule c/(k+k0), and show that spectral clipping achieves a strictly smaller constant. This result concerns exact clipping under the stated assumptions. The practical randomized approximation is a separate question: we added a GPT-2 diagnostic (Appendix K, Table 7) that measures both unprocessed spectral-tail violations and the error relative to full SVD; the protocol and results are in our response to your diagnostic request below.
We thank the reviewer for this suggestion. In the original experiments, norm clipping is global, whereas spectral clipping is applied separately to each layer. The global norm is computed over all parameters for the MLP, NanoGPT and GPT-2, and over the convolution filters for CifarNet.
To check whether the improvement comes from clipping each layer separately, we added layer-wise norm clipping to the CIFAR-10N comparison. It clips the same six convolution-filter tensors as spectral clipping, using one threshold shared across layers. We tuned each method independently with the same budget of 200 random configurations and three seeds per configuration, selecting the best settings by mean clean-validation accuracy. This comparison uses constant thresholds for all three methods.
The table below reports test accuracy (%, mean ± standard deviation over three seeds).
Layer-wise norm clipping performs better than global norm clipping on Aggregate and Worst, but spectral clipping still achieves higher mean test accuracy than layer-wise norm clipping on all three label sets: by 0.56, 0.65 and 0.55 percentage points on Aggregate, Random-1 and Worst, respectively. We have added this baseline to Figure 2(a) and report the results and tuning details in Appendix G (Table 3).
We thank the reviewer for this suggestion. We added an approximation diagnostic on GPT-2 124M trained on FineWeb with SGDM. We ran three additional diagnostic runs of 2,000 updates each, using rank 64, oversampling 10, two power iterations, and a fixed clipping threshold of 0.0043421663. Every 200 updates, we checked all 48 attention and MLP weight-gradient matrices in each run, giving 1,440 matrix samples in total.
None of the sampled matrices had σ65>τ (0/1,440); at most 39 singular values exceeded the threshold in any sample. We also compared the actual clipped gradient with exact spectral clipping computed by full SVD in float64. The relative Frobenius error, normalized by the norm of the exact clipped gradient, had a median of 0.028%, a 95th percentile of 0.162%, and a maximum of 0.327% across all samples. Appendix K, Table 7 reports the results at each sampled update. These measurements concern the three additional 2,000-update diagnostic runs and their sampled points.
We thank the reviewer for this request. The validation loss is now reported in the paper (Figure 9, Appendix F). The table below gives it (mean ± std over three seeds) at matched tokens (after 2,000 updates) and at matched wall-clock time (when the norm-clipping run with the same seed finishes its 2,000 updates):
With SGDM and Adam, spectral clipping gives a lower validation loss at both budgets; at matched time the gain is smaller because an update with spectral clipping takes longer, so fewer updates fit into the same time. With Muon, the two methods are within seed-to-seed variation at matched tokens, and norm clipping is better at matched time. The matched-token values are the final values of Figure 9.
We hope this addresses all the concerns of the reviewer.