acceptodds
Under review as a conference paper at ICLR 2027

Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters

Abstract

Gradient clipping is a standard safeguard for training neural networks under noisy, heavy-tailed stochastic gradients; yet, most clipping rules treat all parameters as vectors and ignore the matrix structure of modern architectures. We show empirically that data outliers often amplify only a small number of leading singular values in layer-wise gradient matrices, while the rest of the spectrum remains largely unchanged. Motivated by this phenomenon, we propose spectral clipping, which stabilizes training by clamping singular values that exceed a threshold while preserving the singular directions. This framework generalizes classical gradient norm clipping and can be easily integrated into existing optimizers. We provide a convergence analysis for non-convex optimization with spectrally clipped SGD, yielding the optimal rate for heavy-tailed noise. To minimize hyperparameter tuning, we introduce layer-wise adaptive thresholds based on moving averages or sliding-window quantiles of the top singular values. Finally, we develop efficient implementations that clip only the top singular values via randomized truncated SVD, avoiding full decompositions for large layers. We demonstrate competitive performance across synthetic heavy-tailed settings and neural network training tasks.

Primary area:
optimization
Keywords:
gradient clipping, spectral clipping, heavy-tailed noise, stochastic optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5J2ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies spectral gradient clipping, which clamps individual singular values of a matrix-valued stochastic gradient instead of rescaling the entire gradient. It provides nonconvex convergence guarantees under a bounded centered alpha-moment assumption, and a strict asymptotic excess-loss constant comparison with Frobenius clipping on a quadratic problem with low-rank heavy-tailed noise. EMA and quantile thresholds and a randomized truncated-SVD implementation are proposed. Experiments cover CIFAR-10N, language modeling, and synthetic heavy-tailed regression.

Critical strengths
  1. Theoretical comparison: Theorem 3.6 establishes a strict separation in actual asymptotic excess-loss constants under a specified low-rank heavy-tailed quadratic model, rather than merely comparing loose upper bounds. Theorem 3.4 also supplies an explicit stationarity guarantee.

  2. Practical evidence: the evaluation spans real human label noise in CIFAR-10N, language modeling, and synthetic heavy-tailed regression. Appendix G documents substantial independent baseline tuning, including 200 CIFAR configurations with three seeds and a held-out validation set. Full/truncated-SVD and wall-clock comparisons address implementation costs.

Critical weaknesses
  1. The exact theory and practical operator remain separated. The convergence results analyze exact spectral clipping with specified thresholds. Practical runs use randomized rank-r approximations and layerwise processing together with momentum or Adam or Muon. The paper acknowledges this distinction but does not quantify unprocessed spectral-tail violations or randomized approximation error. Theorem 3.6 establishes a strict advantage for a specified low-rank Haar and Pareto quadratic model. It does not establish a universal advantage for arbitrary noise. This limits the broad practical robustness claim rather than invalidating the theorem.

  2. The experiments do not fully isolate spectral selectivity or establish language-model utility. It is unclear whether norm clipping uses matching layerwise granularity. Some gains could therefore arise from avoiding cross-layer rescaling. Language-model results mainly show training loss. Final-run variability and validation loss are not reported clearly and the GPT-2 equal-time comparison shown is for SGDM. The baseline tuning documented in Appendix G is substantial but does not resolve these specific controls. These gaps prevent a stronger acceptance recommendation.

Questions for authors
  1. Please clarify whether norm clipping is global or layerwise in each experiment. Report whether the benefit persists with independently tuned and matched layerwise granularity and threshold adaptation. This would isolate spectral selectivity from cross-layer rescaling.

  2. In the existing runs please report how often the first unprocessed singular value exceeds the clipping threshold. An approximation residual diagnostic or a small rank-sensitivity comparison would clarify whether the practical operator behaves like the analyzed operator.

  3. For the existing GPT-2 runs please report validation loss at matched token and wall-clock budgets. Please specify the number of final evaluation seeds and the variability. These results could change my assessment of practical utility.

My current recommendation is weak acceptance because the strict asymptotic comparison and practical breadth are valuable. The above questions limit confidence in broader empirical claims. I am not requesting a new larger-model benchmark.

2 comments

We thank the Reviewer for their comments and suggestions. We address the concerns below.

W1.

We agree that Theorem 3.6 does not establish a universal advantage for arbitrary noise, and we do not claim one. Our method targets heavy-tailed gradient noise concentrated in a few singular directions (Figure 1), and Theorem 3.6 proves a separation in this regime, on a matrix quadratic with low-rank Haar–Pareto noise: we derive a lower bound on the asymptotic excess-loss constant of Frobenius clipping for any fixed positive threshold and any positive coefficient cc in the step-size schedule c/(k+k0)c/(k + k_0), and show that spectral clipping achieves a strictly smaller constant. This result concerns exact clipping under the stated assumptions. The practical randomized approximation is a separate question: we added a GPT-2 diagnostic (Appendix K, Table 7) that measures both unprocessed spectral-tail violations and the error relative to full SVD; the protocol and results are in our response to your diagnostic request below.

W2 (1/2), Q1.

We thank the reviewer for this suggestion. In the original experiments, norm clipping is global, whereas spectral clipping is applied separately to each layer. The global norm is computed over all parameters for the MLP, NanoGPT and GPT-2, and over the convolution filters for CifarNet.

To check whether the improvement comes from clipping each layer separately, we added layer-wise norm clipping to the CIFAR-10N comparison. It clips the same six convolution-filter tensors as spectral clipping, using one threshold shared across layers. We tuned each method independently with the same budget of 200 random configurations and three seeds per configuration, selecting the best settings by mean clean-validation accuracy. This comparison uses constant thresholds for all three methods.

The table below reports test accuracy (%, mean ± standard deviation over three seeds).

Training labelsNorm Clip (global)Layer-wise Norm ClipSpectral Const
Aggregate89.10 ± 0.1089.36 ± 0.0389.92 ± 0.14
Random-188.27 ± 0.1488.12 ± 0.0988.77 ± 0.16
Worst82.04 ± 0.1382.31 ± 0.2082.86 ± 0.47

Layer-wise norm clipping performs better than global norm clipping on Aggregate and Worst, but spectral clipping still achieves higher mean test accuracy than layer-wise norm clipping on all three label sets: by 0.56, 0.65 and 0.55 percentage points on Aggregate, Random-1 and Worst, respectively. We have added this baseline to Figure 2(a) and report the results and tuning details in Appendix G (Table 3).

Q2.

We thank the reviewer for this suggestion. We added an approximation diagnostic on GPT-2 124M trained on FineWeb with SGDM. We ran three additional diagnostic runs of 2,000 updates each, using rank 64, oversampling 10, two power iterations, and a fixed clipping threshold of 0.0043421663. Every 200 updates, we checked all 48 attention and MLP weight-gradient matrices in each run, giving 1,440 matrix samples in total.

None of the sampled matrices had σ65>τ\sigma_{65} > \tau (0/1,440); at most 39 singular values exceeded the threshold in any sample. We also compared the actual clipped gradient with exact spectral clipping computed by full SVD in float64. The relative Frobenius error, normalized by the norm of the exact clipped gradient, had a median of 0.028%, a 95th percentile of 0.162%, and a maximum of 0.327% across all samples. Appendix K, Table 7 reports the results at each sampled update. These measurements concern the three additional 2,000-update diagnostic runs and their sampled points.

W2 (2/2), Q3.

We thank the reviewer for this request. The validation loss is now reported in the paper (Figure 9, Appendix F). The table below gives it (mean ± std over three seeds) at matched tokens (after 2,000 updates) and at matched wall-clock time (when the norm-clipping run with the same seed finishes its 2,000 updates):

OptimizerNorm ClipSpectral Clip, matched tokensSpectral Clip, matched time
SGDM6.314 ± 0.0056.215 ± 0.0016.221 ± 0.001
Adam4.134 ± 0.0094.076 ± 0.0054.114 ± 0.011
Muon4.241 ± 0.0084.236 ± 0.0024.258 ± 0.003

With SGDM and Adam, spectral clipping gives a lower validation loss at both budgets; at matched time the gain is smaller because an update with spectral clipping takes longer, so fewer updates fit into the same time. With Muon, the two methods are within seed-to-seed variation at matched tokens, and norm clipping is better at matched time. The matched-token values are the final values of Figure 9.

We hope this addresses all the concerns of the reviewer.

Thank you for your reply, and I will keep my score.

Reviewer_bhydOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SGD with spectral clipping and analyzes its convergence under standard smoothness and bounded-moment assumptions for non-convex setups. It also shows that spectral clipping can achieve a smaller asymptotic loss constant than norm clipping on a matrix quadratic with structured heavy-tailed noise. The experiments include CIFAR-10N, NanoGPT on Shakespeare, GPT-2 on FineWeb, and synthetic MLP regression, with comparisons involving SGDM, Adam, and Muon.

Critical strengths

The paper provides convergence guarantees for nonconvex setups under standard smoothness and bounded-moment assumptions. It extends the analysis to a matrix quadratic with structured heavy-tailed noise, where spectral clipping yields a smaller asymptotic loss constant than norm clipping. The experiments cover diverse tasks: image classification, language modeling, and synthetic regression, with comparisons involving SGDM, Adam, and Muon. The CIFAR-10N results look good.

Critical weaknesses
  • The dependence on dd in Theorem 3.4 seems loose. For example, in the bounded-variance case (α=2\alpha=2), the stochastic term includes an additional factor of dd, and the step size must satisfy η≤1/(4dL)\eta\leq 1/(4dL). By comparison, Theorem 3.3 of Koloskova et al. (2023) gives a dimension-free bound for standard norm clipping. Is this dependence necessary, or can the analysis be tightened? Theorem 3.6 helps explain the potential advantage of spectral clipping, but only for a quadratic objective with structured noise and still depends on the dimension dd.

  • SPECTRA seems closely related to the proposed method. Appendix J compares spectral clipping before and after momentum in a synthetic problem. A comparison on at least one deep-learning benchmark would help clarify the practical advantage over existing spectral methods.

  • The language-model experiments show training loss, but no validation loss or perplexity. It would be useful to see whether the lower training loss also leads to better performance on unseen data. Similarly, the synthetic regression experiment could report prediction error on a separate test set.

  • Figure 2b should also include Adam with spectral clipping; the current results show improvements within SGDM, but leave open whether Adam would benefit from the same clipping rule.

  • Could the authors report the number of seeds and provide std error bars for the language-model results to clarify whether the smaller improvements shown for Adam and Muon are consistent across runs?

Questions for authors

see weaknesses above

3 comments

We thank the Reviewer for their comments and suggestions. We address the concerns below.

W1.

We agree that the original dependence on dd was loose. We have revised Theorem 3.4 and its proof. A sharper second-moment estimate removes this dependence when α=2\alpha=2 and relaxes the step-size condition to 0<η≤1/L0<\eta\leq 1/L, under the same assumptions.

For a direct comparison with Koloskova et al. [1], the revised proof also gives, when α=2\alpha=2,

1K∑k=0K−1E∥∇f(Xk)∥F≤2ΔηK+4ΔηKτ+σ2τ+2σηL,Δ=f(X0)−f∗.\frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\Vert\nabla f(\mathbf X_k)\Vert_F \leq 2\sqrt{\frac{\Delta}{\eta K}}+\frac{4\Delta}{\eta K\tau}+\frac{\sigma^2}{\tau}+2\sigma\sqrt{\eta L},\qquad \Delta=f(\mathbf X_0)-f_\ast.

For τ≥max⁡2,8σ\tau\geq\max\\{2,8\sigma\\}, this matches the parameter dependence of their Theorem 3.3 with L0=LL_0=L and L1=0L_1=0, up to constants. Their theorem also covers arbitrary positive thresholds and generalized smoothness. We have added this comparison in Eq. (8), with the proof in Appendix C.2.

For 1<α<21<\alpha<2, the stochastic term now contains d1−α/2d^{1-\alpha/2} instead of dd. In the final rate, Eq. (6), the dimension enters only with exponent (2−α)(α−1)/(3α−2)≤1/9(2-\alpha)(\alpha-1)/(3\alpha-2) \leq 1/9. Remark C.2 shows that this factor is necessary for the second-moment bound, although this does not establish optimality of the dimension dependence in the convergence rate.

Regarding Theorem 3.6: Theorems 3.3 and 3.4 hold for arbitrary noise with a bounded α\alpha-moment and also cover norm clipping, so these worst-case bounds cannot show when spectral clipping is preferable. Theorem 3.6 instead gives a lower bound on the asymptotic excess-loss constant of Frobenius clipping for any fixed positive threshold and any positive coefficient cc in the step-size schedule c/(k+k0)c/(k + k_0), on a matrix quadratic with heavy-tailed low-rank noise. Spectral clipping achieves a strictly smaller constant in this setting, for every dimension d>rd > r. Thus, this is a separation result, not just a comparison of upper bounds. Its scope is the stated quadratic and structured-noise model; extending the comparison beyond this setting is a direction for future work.

W2.

We thank the reviewer for this suggestion. Our method clips the raw gradient before the optimizer step, whereas SPECTRA primarily clips the optimizer update, after momentum or adaptive scaling. SPECTRA also supports optional gradient clipping, but it is disabled by default in their LLM experiments. We clarify this distinction in Section 1.1. The lower-bound separation in Theorem 3.6, discussed above, is an additional theoretical contribution.

We compared our spectral clipping with SPECTRA on CIFAR-10N using the authors' implementation. We tuned SPECTRA with the same budget of 200 random configurations and three seeds per configuration, selecting the best hyperparameters by mean clean-validation accuracy on Aggregate labels. We then used these settings unchanged for Aggregate, Random-1 and Worst.

As shown in Figure 6(a–c), our spectral clipping achieves higher final mean test accuracy than SPECTRA on all three label sets. We have added these curves to Appendix F and described the SPECTRA implementation and hyperparameter tuning in Appendix G, with the selected settings reported in Table 2.

We also added SPECTRA to the MLP experiment with heavy-tailed gradient noise (Figure 10(b)), using the same tuning procedure as for the other methods. With SGDM, our spectral clipping reaches a lower final training loss at every noise probability, and the gap grows as the noise is injected more often: 0.0037 vs 0.0055 at probability 0.1 and 0.0125 vs 0.054 when noise is injected at every step. With Adam, SPECTRA is better at probability 0.1 (0.0078 vs 0.0165), and our clipping is better from probability 0.5 on (0.019 vs 0.136 at every step).

W3.

We thank the reviewer for this suggestion. For GPT-2, we added the validation loss on held-out FineWeb data (Figure 9, Appendix F); the results are consistent with the training loss. Validation loss at the end of training, mean ± std over three seeds:

OptimizerNorm ClipSpectral Clip
SGDM6.314 ± 0.0056.215 ± 0.001
Adam4.134 ± 0.0094.076 ± 0.005
Muon4.241 ± 0.0084.236 ± 0.002

With SGDM and Adam, spectral clipping gives a lower validation loss on each of the three seeds. With Muon, spectral clipping is ahead in the middle of training, and at the end the difference is within seed-to-seed variation.

The synthetic regression experiment is designed as a test of optimization under heavy-tailed gradient noise, like the synthetic experiment of Chezhegov et al. [2], so we did not add held-out results to the paper and report them here. Loss on the held-out 20% of the data, median over ten seeds:

MethodHeld-out lossTraining loss
SGDM Norm Clip0.440.0075
SGDM Spectral Const0.440.0042
SGDM Spectral EMA0.960.013
SGDM Spectral Quantile0.170.0011
Adam No Clip1.150.26
Adam Norm Clip0.280.016
Adam Spectral Const0.280.018

Quantile-based spectral clipping remains the best method on held-out data; the EMA variant is behind the other SGDM variants on both training and held-out data, and the remaining clipped methods overlap across seeds. The training set has only 8,000 points, and all methods fit it far better than they generalize.

W4.

This is an excellent question. We added Adam with spectral clipping to Figure 2b: in this setting it performs on par with Adam with norm clipping. We suspect that the way we currently combine spectral clipping with Adam is not optimal: SGDM has a single momentum buffer, whereas Adam tracks two exponential moving averages, and both are computed from the clipped gradient. How spectral clipping should be incorporated into Adam is not obvious to us; we are still investigating it and consider it an interesting direction for future research.

The benefit for Adam becomes clear when the noise is more frequent. We repeated the experiment for several probabilities of noise injection, tuning each method for each probability with the same search ranges and budget as in the main experiment (Figure 10(a), Appendix F). Final training loss, median over ten seeds (probability 0.1 is the setting of Figure 2b; since every method is re-tuned here, the values differ slightly from the table above):

Noise probabilityAdam + Norm ClipAdam + Spectral ClipSGDM + Norm ClipSGDM + Spectral Clip
0.10.0150.0160.0080.004
0.50.0230.0150.0250.008
0.750.0450.0160.0520.009
0.90.0930.0170.0960.011
1.00.2580.0190.1650.013

With norm clipping, the final loss grows by more than an order of magnitude as the noise becomes more frequent, for both optimizers, while spectral clipping is much less affected.

W5.

All GPT-2 results are now reported over three seeds: Figure 3 shows the mean training loss over the seeds, with the range across seeds shaded, and Figure 9 shows the validation loss (final values with standard deviations are in the table above). For Adam, the improvement is consistent: spectral clipping is better on each seed in both training and validation loss, and the gap is several times larger than the variation across seeds. For Muon, spectral clipping is ahead in the middle of training, and the difference at the end is within seed-to-seed variation; we revised the text of Section 6.2 accordingly.

References

[1] Anastasia Koloskova, Hadrien Hendrikx, & Sebastian U. Stich. (2023). Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees. ICML 2023.

[2] Savelii Chezhegov, Yaroslav Klyukin, Andrei Semenov, Aleksandr Beznosikov, Alexander Gasnikov, Samuel Horváth, Martin Takáč, & Eduard Gorbunov. (2025). Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed. ICML 2025.

We hope we have addressed all the questions and concerns of the reviewer. If you agree, please consider increasing your score. If not, please let us know so we can respond accordingly.

Thank you for the rebuttal and clarifications. Your response addresses my concerns, and I have increased my score to 3.

Reviewer_GBtVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes spectral gradient clipping for matrix-valued parameters to address heavy-tailed gradient noise. Instead of uniformly scaling the entire gradient matrix, the method clips individual singular values while preserving their corresponding singular directions. The authors provide convergence guarantees for non-convex optimization and establish a theoretical advantage over conventional norm clipping under a specific low-rank heavy-tailed noise model. Additionally, adaptive clipping thresholds and an efficient truncated-SVD implementation are introduced. Experiments on synthetic tasks, CIFAR-10N, and language-model training demonstrate promising improvements over conventional norm clipping.

Critical strengths
  1. Solid theoretical analysis. The paper provides convergence guarantees under heavy-tailed noise and a theoretical comparison demonstrating the potential advantages of spectral clipping over conventional norm clipping.
  2. Promising empirical results. The proposed method achieves competitive performance across several tasks and optimizers, supported by practical adaptive threshold strategies and an efficient implementation.
Critical weaknesses
  1. Limited methodological novelty. The central idea of clipping singular values is relatively straightforward and closely related to existing spectral clipping methods, particularly SPECTRA (Jiang et al., 2026). Although the theoretical analysis provides additional insights, the methodological contribution appears somewhat incremental. A clearer distinction from closely related work would strengthen the novelty claim.
  2. Concerns about experimental fairness. The paper primarily compares spectral clipping against conventional norm clipping. However, it is not entirely clear whether the baselines use comparable clipping granularity and equally thorough hyperparameter tuning. In particular, a matched layer-wise Frobenius norm clipping baseline would help determine whether the reported improvements arise from spectral clipping itself rather than differences in the experimental setup.
Questions for authors
  1. Could the authors further clarify the main methodological differences between the proposed approach and existing spectral clipping methods, particularly SPECTRA, and explain what additional practical advantages the proposed method offers?
  2. Are the norm-clipping baselines evaluated using comparable layer-wise clipping strategies and hyperparameter tuning budgets? Could the authors provide a matched layer-wise Frobenius norm clipping comparison to better demonstrate the effectiveness of spectral clipping?
1 comment

We thank the Reviewer for their comments and suggestions. We address the concerns below.

W1, Q1.

We thank the reviewer for this comment.

  • First, the main methodological difference is where clipping is applied. SPECTRA has two components: post-spectral clipping of the optimizer update, which enforces a spectral-norm constraint, and optional pre-spectral clipping of the raw gradient, which is disabled by default in their LLM experiments. We clip only the gradient, because our goal is robustness to heavy-tailed gradient noise, reported for attention models by Zhang et al. [1]. Figure 1 shows why this calls for spectral clipping: a few bad tokens in a mini-batch inflate the top singular values of a gradient matrix. Post-clipping acts too late for such a spike: by the time the update is clipped, the spike is already in the momentum buffer. Appendix J tests this in the simplest setting, with heavy-tailed rank-one spikes added to the gradients: clipping the gradient before momentum gives a lower gradient-estimation error than clipping after it (Figure 12). The SPECTRA authors also note that pre-clipping removes such spikes before they corrupt the momentum buffer (their Section 5.2). We have clarified this distinction in Section 1.1.

  • Second, the theory is different. SPECTRA's analysis assumes finite variance and compares spectral and global clipping through upper bounds (their Theorems 4.6 and 4.7). Our convergence guarantees (Theorems 3.3 and 3.4) allow infinite variance, and Theorem 3.6 goes beyond upper bounds: on a matrix quadratic with heavy-tailed low-rank noise, it gives a lower bound on the asymptotic excess-loss constant of Frobenius clipping for any fixed positive clipping threshold and any c>0c > 0 in the step sizes c/(k+k0)c/(k + k_0), and shows that spectral clipping attains a strictly smaller constant. This proves a separation between the methods rather than comparing their upper bounds.

  • Finally, to show the practical advantage, we added a direct comparison with SPECTRA on CIFAR-10N, using the authors' implementation with post-clipping only (pre-clipping is off by default) and the same hyperparameter tuning budget. As shown in Figure 6(a–c), our spectral clipping achieves higher final mean test accuracy than SPECTRA on all three label sets. We describe the implementation and tuning in Appendix G, with the selected settings in Table 2. We also added SPECTRA to the MLP experiment with heavy-tailed gradient noise (Figure 10(b)), again with post-clipping only and the same tuning procedure. With SGDM, our spectral clipping reaches a lower final training loss at every noise probability, and the gap grows as the noise is injected more often: 0.0037 vs 0.0055 at probability 0.1 and 0.0125 vs 0.054 when noise is injected at every step. With Adam, SPECTRA is better at probability 0.1 (0.0078 vs 0.0165), and our clipping is better from probability 0.5 on (0.019 vs 0.136 at every step).

W2, Q2.

We thank the reviewer for this suggestion.

  • First, we clarify that both methods received the same hyperparameter tuning budget which we discussed in the Appendix G. While spectral clipping indeed works independently for each layer, the constant-threshold variant used in these experiments has one clipping threshold shared across all clipped layers and we used the same number of trials to tune it.

  • Second, we added layer-wise norm clipping (Layer-wise Norm Clip) to the CIFAR-10N comparison and tuned it with the same budget of 200 configurations and three tuning seeds per configuration. The table below reports test accuracy (%, mean ± standard deviation over three seeds). Spectral clipping achieves higher mean test accuracy than layer-wise norm clipping on all three label sets. We have added Layer-wise Norm Clip to Figure 2(a), added this table to Appendix G (Table 3), and updated Appendix G to specify its hyperparameter search ranges.

Training labelsNorm Clip (global)Layer-wise Norm ClipSpectral Const
Aggregate89.10 ± 0.1089.36 ± 0.0389.92 ± 0.14
Random-188.27 ± 0.1488.12 ± 0.0988.77 ± 0.16
Worst82.04 ± 0.1382.31 ± 0.2082.86 ± 0.47

References

[1] Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J. Reddi, Sanjiv Kumar, & Suvrit Sra. (2020). Why are Adaptive Methods Good for Attention Models? NeurIPS 2020.

We hope this addresses all the concerns of the reviewer. Thanks again for investing time in the review process.

Comments on the paper

1 comment

Summary of the main contributions and of the revision.

We thank all reviewers for their careful reading and constructive feedback. Below we summarize the main contributions of the paper and the new results added in the revision. All changes in the revised PDF are marked in red.

Main contributions
  • We propose spectral clipping of gradients for matrix-valued parameters: instead of rescaling the whole gradient by its norm, we clip its singular values. This targets heavy-tailed gradient noise concentrated in a few singular directions (Figure 1).
  • We prove convergence guarantees under heavy-tailed noise with a bounded α\alpha-moment, which allow infinite variance (Theorems 3.3 and 3.4).
  • We prove a separation from norm clipping (Theorem 3.6): on a matrix quadratic with heavy-tailed low-rank noise, spectral clipping attains a strictly smaller asymptotic excess-loss constant than Frobenius clipping with any fixed threshold.
  • We give an efficient randomized implementation with adaptive thresholds and evaluate it on CIFAR-10N, NanoGPT, GPT-2 and a synthetic regression task.
New results in the revision
  • [Comparison with SPECTRA (Reviewers GBtV, bhyd)] We clarified the difference between the methods in Section 1.1: we clip the gradient before momentum, while SPECTRA clips the optimizer update. In a direct comparison with the same tuning budget, our clipping achieves higher final mean test accuracy on all three CIFAR-10N label sets (Figure 6(a–c)). In the MLP experiment with heavy-tailed gradient noise (Figure 10(b)), it reaches a lower final training loss with SGDM at every noise probability and with Adam from probability 0.5 on.
  • [Layer-wise norm clipping baseline (Reviewers GBtV, 5J2Z)] Tuned with the same budget on CIFAR-10N, it stays below spectral clipping on all three label sets, by 0.56, 0.65 and 0.55 percentage points (Figure 2(a), Table 3).
  • [Sharper theory (Reviewer bhyd)] The revised Theorem 3.4 has no dimension dependence when α=2\alpha = 2 and holds for all step sizes 0<η≤1/L0 < \eta \le 1/L. For 1<α<21 < \alpha < 2 the dimension enters the final rate only with an exponent of at most 1/91/9 (Eq. (6), Remark C.2).
  • [Validation loss on GPT-2 in Figure 9, Appendix F (Reviewers bhyd, 5J2Z)] Spectral clipping gives a lower validation loss with SGDM (6.215 ± 0.001 vs 6.314 ± 0.005) and with Adam (4.076 ± 0.005 vs 4.134 ± 0.009), also at matched wall-clock time. With Muon the two methods are within seed-to-seed variation.
  • [Seeds and variability (Reviewers bhyd, 5J2Z)] All GPT-2 results are now reported over three seeds: Figure 3 shows the mean training loss with the range across seeds shaded, and Figure 9 shows the mean validation loss. We revised the text of Section 6.2 accordingly.
  • [Adam with spectral clipping (Reviewer bhyd)] We added it to the synthetic regression experiment (Figure 2b), together with a study over noise probabilities (Figure 10(a)).
  • [Accuracy of the randomized approximation in Appendix K, Table 7 (Reviewer 5J2Z)] In 1,440 sampled gradient matrices of GPT-2, none had a singular value above the threshold left unprocessed, and the relative error against exact clipping with full SVD was at most 0.327%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.