acceptodds
Under review as a conference paper at ICLR 2027

Understanding Clipping in Zeroth-order Optimization

Abstract

Zeroth-order optimization has emerged as a resource-efficient alternative to Stochastic Gradient Descent (SGD) when fine-tuning LLMs, thanks to the inherent low dimensionality of the loss landscape. It relies on a two-point estimate of the gradient that only accesses the function value and not the gradient, bypassing the resource-heavy backpropagation. Recently, it was accidentally discovered that per-sample clipping can improve the performance of the trained model, while running a differentially private version of zeroth-order optimization zhang2024dpzero with a very large privacy parameter (corresponding to very little privacy). In this paper, we systematically investigate this phenomenon and demonstrate that () the optimal choice of learning rate is inversely proportional to the clipping threshold , satisfying constant; () clipping drives the zeroth-order optimization to a different solution with higher test accuracy; and () this phenomenon is only observed with per-sample clipped zeroth-order optimization and not other zeroth-order alternatives that also provide robust estimates of the gradient. Further analyses with artificially injected label noise reveals that clipping leads to more robust predictions, which stems from the implicit down weighting of hard-to-fit training examples.

Primary area:
optimization
Keywords:
zeroth-order optimization, fine-tuning LLMs, clipping

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wFXMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper empirically investigates why per-sample clipping can improve model performance in zero-th SGD. It first illustrates the limiting behavior of gradients when clipping threshold approaches either 0 or infinity, providing theoretical foundation. Then through extensive experiments on four RoBERTa classification tasks, the authors found that when learning rate η\eta and clipping threshold cc are inversely proportional, then clipping can robustly improve model performance by driving the model converges to a different solution. Label-corruption and data-pruning experiments suggest that the improved performance might originate from downweighting hard-to-fit examples.

Critical strengths
  1. Proposition 2.1 gives an intuitive explanation of the influence of clipping threshold on per-sample gradient, making later empirical observations easier to understand.
  2. Extensive experiments and complementary diagnostics demonstrate the performance gains from clipping.
  3. Label-corruption and data-pruning experiments are well-designed, properly supprting the downweighting claim.
Critical weaknesses
  1. Lack of clarity of the reported accuracy improvements. The paper currently in mixed-use of train and test accuracy: for example, Figure 1 reports test accuracy while Figure 2 reports train accuracy. Moreover, some statements refer broadly to improvements in “accuracy,” making it unclear whether they concern training “accuracy, test “accuracy, or both. Consistently specifying the metric would help readers understand the scope and implications of the claimed gains.
  2. The interpretation of convergence to different solutions needs stronger support. The distances from initialization and projected loss landscapes suggest that clipping changes the optimization trajectory. However, these finite-budget observations do not fully distinguish a lasting change in the solutions reached from differences in optimization speed or transient behavior. Stronger evidence for this distinction would substantiate a central interpretation of the paper.
Questions for authors
  1. Figure 1, right panel. Why c≤100c\leq 100 is identified as fully clipped regime, when less than half (20-29 out of 64) of gradients are actually clipped? What happens if cc is set to a small value (say c=1c=1 or c=0.001c=0.001) so that all gradients are truly clipped. Would the performance gain remain?
  2. How consistent are the comparisons in Figure 5 across different random seeds and few-shot training subsets? In particular, does the advantage over Winsorization persist? Reporting means and standard deviations would help assess the reliability of this finding.
4 comments

We thank you for the constructive review and for noting the clarity of Proposition 2.1 and the design of our label-corruption and data-pruning experiments. We have updated the manuscript (with revisions highlighted in blue) and address each of your questions and concerns below.

1. Why <50%<50\% of Gradients Are Clipped at c≤100c \le 100 & Behavior at Ultra-Small c∈{0.1,0.5,1.0}c \in \{0.1, 0.5, 1.0\} (Question 1)

Thank you for pointing this out. Our original phrasing was unclear: the optimizer does operate in a nearly fully clipped regime during the active learning phase, but the 2020–2929 statistic in Figure 1 (right) is averaged over the entire 10,00010,000-step training run. Zeroth-order optimization can be thought of like descending a steep slope blindfolded, where initial steps in random directions are highly volatile before flattening out near the bottom. As shown in our new per-iteration clip-count tracking in Appendix F.2 (Figure 24, left), during the first ∼4,000\sim 4,000 steps of training, almost all 6464 per-sample directional derivatives in each batch (6060–6464 out of 6464 for c≤5c \le 5) exceed cc and are clipped. This initial volatile phase drives the sharp separation in weight distance ∥wt−w0∥\Vert w_t - w_0\Vert (Figure 1, middle) and the rapid ascent in accuracy. After this initial ∼4,000\sim 4,000-step period, the model settles into the basin and adapts to the clean, easy examples (pushing them into the >0.9>0.9 confidence bin, Figure 7), so their gradient projections shrink below cc. In the second half of training, primarily the ambiguous or mislabeled examples continue to generate gradients large enough to trigger clipping (Figures 9–10). Averaging this long settling phase with the heavily clipped initial phase pulls the 10,00010,000-step average down to 2020–2929 clips per batch. We have clarified this in the caption of Figure 1 and updated our terminology throughout the paper to refer to c≤100c \le 100 as the clipping-dominated regime.

To test what happens when cc is set to even smaller values so that gradients continue to be clipped throughout training, we expanded our RoBERTa-SNLI sweep down to c∈{0.1,0.5,1.0}c \in \{0.1, 0.5, 1.0\} with correspondingly larger learning rates η∈{5×10−3,10−3,5×10−4}\eta \in \{5\times 10^{-3}, 10^{-3}, 5\times 10^{-4}\} along the ηc=5×10−4\eta c = 5\times 10^{-4} line in Appendix F.2 (Figures 22–24). Across all of these ultra-small cc settings, both the ηc=5×10−4\eta c = 5\times 10^{-4} scaling relationship and the test accuracy gains (82.7%82.7\%–83.8%83.8\% vs. 80.2%80.2\% at c=∞c = \infty) remain completely intact, and the weight distance ∥wt−w0∥\Vert w_t - w_0\Vert follows the exact same high-distance trajectory (Figures 22–23) while the 10,00010,000-step average clip count rises to 33.8933.89 out of 6464 at c=0.1c = 0.1 (Figure 24, right).

2. Stronger Evidence of Convergence to Distinct Solutions vs. Optimization Speed (Weakness 2)

You raised an important point regarding whether clipping genuinely drives the model to a different solution basin or simply alters optimization speed along the same trajectory. We have expanded Section 3.2 (lines 218–230) to rule out the optimization-speed explanation through three pieces of evidence. First, if unclipped ZO-SGD (w3w_3) were merely progressing more slowly along the same path toward the same minimum as clipped ZO-SGD (w1,w2w_1, w_2), then w3w_3 would have a higher training loss than w1w_1 and w2w_2 at step 10,00010,000. Instead, w3w_3 converges to a strictly lower training and test loss (0.450.45 unclipped vs. 0.700.70 at c=10c = 10, Figure 4 and Figure 13 right) while achieving lower train and test accuracy, whereas clipped models plateau at higher loss with higher accuracy. Second, our logit ensembling experiments (Figure 3) test functional diversity directly: ensembling two clipped checkpoints from different points on the ηc=constant\eta c = \text{constant} line (w1w_1 at c=10c = 10 and w2w_2 at c=50c = 50, trained from the same initialization and random seed) yields zero ensemble gain (Figure 3, middle), confirming that varying (η,c)(\eta, c) along the trade-off line reaches the same basin. In contrast, ensembling a clipped model (w2w_2) with an unclipped model (w3w_3) improves both test accuracy and test loss (Figure 3, left), demonstrating that they capture distinct solutions. Third, this matches our theoretical analysis in Proposition 2.1 and Appendix K.2: at any stationary point w∗w^\ast of the unclipped objective satisfying 1n∑i=1n∇fi(w∗)=0\frac{1}{n}\sum_{i=1}^n \nabla f_i(w^\ast) = 0, the expected clipped update 1n∑i=1nhc(∥∇fi(w∗)∥)∇fi(w∗)\frac{1}{n}\sum_{i=1}^n h_c(\Vert\nabla f_i(w^\ast)\Vert)\nabla f_i(w^\ast) remains non-zero whenever per-sample gradient norms vary across examples, driving clipped ZO-SGD to the minimizer of a distinct surrogate objective Fc(w)F_c(w) (derived in Appendix K.2.1, Eqs. 26–29).

Continued in next comment...

3. Consistency Across Random Seeds in Figure 5 (Question 2)

To verify the reliability of the comparisons in Figure 5 across random seeds, we ran 5 independent random seeds on SNLI and updated Figure 5 (Section 3.4) to report the mean ± standard error in every cell for per-sample clipping, batch clipping, and the Winsorized mean. The advantage of per-sample clipping is consistent and statistically significant across seeds: per-sample clipping achieves 82.62%82.62\%–82.96%82.96\% along c∈[5,100]c \in [5, 100] (peaking at 82.96%±0.35%82.96\% \pm 0.35\% at c=50c = 50 and 82.90%±0.25%82.90\% \pm 0.25\% at c=10c = 10), clearly outperforming unclipped ZO-SGD (80.92%±0.37%80.92\% \pm 0.37\%), batch clipping (peaking at 81.08%±0.24%81.08\% \pm 0.24\%), and Winsorized mean (peaking at 81.10%±0.49%81.10\% \pm 0.49\%). We are continuing to run multi-seed repetitions in the background for the remaining single-seed grids; however, generating multi-seed standard errors across every remaining figure in the paper, which spans four datasets (SNLI, MNLI, SST-2, TREC), two model families (RoBERTa and OPT), label corruption sweeps, and first-order grids, each requiring 10,00010,000 steps per grid cell, requires substantial compute time. We will include complete mean ± standard error bounds for all experiments in the camera-ready version.

4. Consistently Specifying "Train Accuracy" vs. "Test Accuracy" (Weakness 1)

Thank you for catching this. We have audited the entire manuscript (including Sections 3, 3.2, 3.3, 3.4, 3.5, and 4.3, as well as the figure captions) to explicitly specify "train accuracy" or "test accuracy" at every mention, and we have referenced the 2D test loss and test accuracy landscapes (Appendix G.1, Figure 25) directly alongside the training landscapes in Section 3.2 so readers can see that the test surface mirrors the training surface.

We have uploaded the revised manuscript with all updates highlighted in blue, including the expanded small-cc sweep and per-iteration clip-count trajectories in Appendix F.2, the expanded basin-separation analysis in Section 3.2, the 5-seed evaluations in Figure 5, and consistent train/test metric labels throughout the text. We really appreciate how much your feedback helped sharpen both the experimental evidence and the presentation of the paper, and we are happy to address any additional questions during the discussion phase. Thank you!

I sincerely thank the authors for their responses. My original concerns have been sufficiently addressing. I believe the empirical insights into the role of clipping in zeroth-order optimization in this work are interesting. Therefore, I am updating my score from 2 to 3.

We would like to thank the reviewer for their reply and constructive feedback. We will make sure to include all the revisions in the new version of the manuscript.

Reviewer_oLSEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper provides an empirical investigation to why, in LLM fine-tuning using Zeroth-order optimization that per-example clipping can improve the utility of the model. The paper argues that this is do to tight clipping driving to model to difference region that with with very loose clipping. The explanation, according to the authors, is that tight clipping suppresses the difficult/mislabeled examples and therefore the model generalizes better to the the usual examples, driving accuracy up. The authors also find that the learning rate inversely scales with the clipping bound, forcing ηC=constant\eta C = \text{constant}.

Critical strengths
  1. The experiments strongly demonstrate that clipped and unclipped settings drive the optimization to different regimes.
  2. The calibration experiments are effectively communicating the reason, also the plots off the loss landscapes clearly demonstrates the author's reasoning.
  3. The ηC=constant\eta C = \text{constant} finding is interesting on itself.
  4. The difference between loss and accuracy in terms of per-example clipping is very interesting.
Critical weaknesses
  1. The paper is mainly an exploration of the effect, providing no theory of what are the mechanism behind this. Adding even a rough theoretical explanation would make the paper more sound.
  2. Clipping (under DP) was recently empirically analyzed in [1] who also found that tight clipping can suppress the hard examples, leaving the optimization to be driven by the easier examples. This might limit the novelty of these results.
  3. There are no confidence intervals to indicate how much variability there is in the results.
  4. The Edge of Stability analysis does not seem to be relevant to paper results.

1 - "On Optimal Hyperparameters for Differentially Private Deep Transfer Learning" (Rehn et al., 2026) - https://arxiv.org/abs/2510.20616

Questions for authors
  1. Can you add confidence intervals to the results?
  2. Can you elaborate why you chose to keep the Edge of Stability analysis in the main text? It seems out of context and also switches from natural language to vision task.
4 comments

We thank you for your constructive feedback and for highlighting the strengths of our loss landscape visualizations, calibration experiments, ηc=constant\eta c = \text{constant} scaling discovery, and loss-versus-accuracy analysis. We have updated the manuscript (with changes highlighted in blue) to address each of your concerns as detailed below.

1. Moving the Edge of Stability (EoS) Analysis to the Appendix (Weakness 4 & Question 2)

We completely agree with your point. We originally ran the Edge of Stability (EoS) experiments to test whether the accuracy benefit of clipping in ZO fine-tuning could be explained by sharpness reduction along the ZO stability threshold (2/Tr(H)2/\text{Tr}(H)) recently identified by Song et al. (2026). To validate our spectral tracking pipeline, we first reproduced Song et al.'s random-initialization CIFAR-10 setting before evaluating pretrained ResNet and RoBERTa models. While our experiments revealed an interesting contrast—namely, that pretrained models already sit in flat basins relative to the fine-tuning objective and never reach the EoS threshold during fine-tuning (Appendix I)—including this in Section 4.4 disrupted the flow of the main text. In our revision, we have moved the EoS section and figures entirely to Appendix I, leaving only a brief two-sentence pointer in the Conclusion (Section 5) noting EoS dynamics in pretrained models as an avenue for future work.

2. Discussion and Differentiation from Rehn et al. (2026) (Weakness 2)

Thank you for bringing this relevant work, On Optimal Hyperparameters for Differentially Private Deep Transfer Learning, to our attention. We have added a citation and discussion of Rehn et al. in Appendix A.2 and connected it to our first-order evaluations in Section 3.5. Our study complements and differs from theirs in three main ways. First, Rehn et al. study first-order DP-SGD, where clipping thresholds the full dd-dimensional deterministic gradient norm ∥∇fi(w)∥\Vert\nabla f_i(w)\Vert to bound sensitivity against injected Gaussian privacy noise. In contrast, we study noise-free Zeroth-Order optimization (σ=0\sigma = 0), where backpropagation is unavailable and clipping acts on random 1D directional finite differences fi(w+λut)−fi(w−λut)2λ\frac{f_i(w+\lambda u_t)-f_i(w-\lambda u_t)}{2\lambda} along ut∼N(0,Id)u_t \sim \mathcal{N}(0, I_d). Second, because utu_t is random in ZO, scalar clipping does not hard-threshold ∥∇fi(w)∥\Vert\nabla f_i(w)\Vert; via Stein's identity, it induces a smooth Gaussian CDF reweighting hc(∥∇fi∥)=2Φ(c/∥∇fi∥)−1h_c(\Vert\nabla f_i\Vert) = 2\Phi(c/\Vert\nabla f_i\Vert) - 1 (Proposition 2.1). We further show that the accuracy gain and strict ηc=constant\eta c = \text{constant} trade-off are unique to per-sample ZO clipping: they do not occur with other ZO robust estimators such as Batch Clipping or Winsorized Mean (Section 3.4, Figure 5), nor do they appear in noise-free First-Order per-sample or batch clipping (Section 3.5, Figure 6), where FO-SGD achieves high accuracy across a wide band of hyperparameters and clipping provides only a marginal gain (+0.4%+0.4\% to +1.2%+1.2\%). Third, beyond showing via label corruption, data pruning, and subset ablations (Section 4.3, Appendix H.3) that clipping downweights mislabeled and ambiguous examples, we show that clipped ZO-SGD converges to a geometrically distinct basin with higher training and test cross-entropy loss and systematic overconfidence (higher ECE) alongside higher test accuracy.

Continued in next comment...

3. Theoretical Explanation of the Underlying Mechanisms (Weakness 1)

We agree that providing a theoretical foundation makes the paper much more sound. In the revised manuscript, we have added Appendix K to formally explain the non-asymptotic gradient behavior and why clipping improves accuracy.

To move beyond the asymptotic λ→0\lambda \to 0 identity in Proposition 2.1 and establish finite-smoothing convergence and accuracy gains, we added Appendix K (summarized in Section 2, p. 3). First, in Appendix K.1 (Proposition K.1, Eqs. 12–23), we analyze the expected clipped gradient estimate for finite smoothing λ>0\lambda > 0 under the assumption that each sample loss fi(w)f_i(w) is ℓ\ell-smooth. We prove that

Eu[gλ,c(w)]=1∣Bt∣∑i∈Bthc(∥∇fi(w)∥)∇fi(w)+bλ,c(w)\mathbb{E}_u[g_{\lambda,c}(w)] = \frac{1}{\vert B_t\vert}\sum_{i\in B_t} h_c(\Vert\nabla f_i(w)\Vert)\nabla f_i(w) + b_{\lambda,c}(w)

where the finite-smoothing bias is bounded by ∥bλ,c(w)∥≤ℓλ2d(d+2)\Vert b_{\lambda,c}(w)\Vert \le \frac{\ell\lambda}{2}\sqrt{d(d+2)}. This recovers Proposition 2.1 as λ→0\lambda \to 0 while quantifying how small λ\lambda needs to be in practice relative to ℓ\ell and dd.

Second, in Appendix K.2, we analyze clipped ZO-SGD in the canonical setting of dd-dimensional linear regression (yi=xi⊤w∗+ziy_i = x_i^\top w^* + z_i, with zi∼N(0,σ2)z_i \sim \mathcal{N}(0, \sigma^2)). In Theorem K.3 and Appendix K.2.1 (Eqs. 26–63), we analyze clipped ZO-SGD when an adversary corrupts an α~\tilde{\alpha} fraction of training labels to arbitrary values. This proves that clipped ZO-SGD matches the robust regression lower bound of Diakonikolas et al. (2019) up to the log⁡(1/α~)\sqrt{\log(1/\tilde{\alpha})} factor, achieving a near-optimal error rate. This theoretically supports out explanation on why clipped ZO-SGD achieves higher accuracy than unclipped ZO-SGD under corruption together with Proposition K.4. Proposition K.4 shows that unclipped ZO-SGD can suffer arbitrarily large error ((1/σ)∥w^−w∗∥>C′(1/\sigma)\Vert\hat{w} - w^\ast\Vert > C') for any C′>0C' > 0 from even a single corrupted label. This is consistent with out observation on the clipping gain growing with the corruption fraction α~\tilde{\alpha} (Figure 8 and Figure 28).

4. Quantifying Run-to-Run Variability (Weakness 3 & Question 1)

We agree that quantifying run-to-run variability is important given the stochasticity of the random Gaussian perturbations utu_t in ZO optimization. To ensure the statistical reliability of our core comparisons, we ran 5 independent random seeds for our central SNLI evaluations in Section 3.4 (Figure 5) and reported the mean ± standard error in every cell across the (η,c)(\eta, c) grids. These standard errors confirm that our observed trends are stable and statistically significant: peak per-sample clipped accuracy (82.96%±0.35%82.96\% \pm 0.35\% at c=50c = 50 and 82.90%±0.25%82.90\% \pm 0.25\% at c=10c = 10) clearly outperforms the unclipped baseline (80.92%±0.37%80.92\% \pm 0.37\%), batch clipping (81.08%±0.24%81.08\% \pm 0.24\%), and Winsorized mean (81.10%±0.49%81.10\% \pm 0.49\%). Generating multi-seed standard errors across every remaining figure in the paper, which spans four datasets (SNLI, MNLI, SST-2, TREC), two model families (RoBERTa and OPT), label corruption sweeps, and first-order grids, each requiring 10,00010,000 steps per grid cell, requires substantial compute time. We are continuing to run multi-seed repetitions in the background for the remaining single-seed grids and will include complete mean ± standard error bounds in the camera-ready version.

We have uploaded the revised manuscript with all changes highlighted in blue, including the relocation of the Edge of Stability experiments to Appendix I, the discussion of Rehn et al. (2026) in Appendix A.2, the 5-seed standard error evaluations in Figure 5, and the theoretical analyses in Appendix K. We hope these additions resolve your concerns regarding the theoretical grounding and statistical reliability of our findings, and we would be glad to answer any follow-up questions as you reconsider our work. Thank you!

Thank you for your hard work on the rebuttal!

You managed to address all my points, including replacing the out-of-context EoS stability section with much more relevant theory. I am satisfied by the results and will therefore raise my score.

We would like to thank you again for your reply and constructive feedback. We will make sure to include all the revisions in the new version of the manuscript.

Reviewer_FowzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates why clipping per-example directional derivatives can improve zeroth-order language-model fine-tuning, even without privacy noise. Its central observation is that clipping changes the relative influence of training examples rather than simply shrinking the update. The authors derive a gradient-norm-dependent weighting rule and connect heavy clipping to an approximately constant product of learning rate and clipping threshold. Experiments suggest that clipping downweights difficult or mislabeled examples, improving classification accuracy while increasing loss and overconfidence. The paper is primarily an investigation of an existing phenomenon, with label-noise, pruning, and calibration experiments used to explain it.

Critical strengths
  1. The reweighting identity provides a clear interpretation of the method. It makes the connection between clipping and normalized per-example gradients easy to understand.
  2. The empirical investigation is thoughtfully organized. Label corruption, data pruning, and alternative gradient estimators probe different aspects of the proposed explanation.
  3. I appreciated the calibration analysis. Showing that accuracy improves while confidence becomes less reliable gives a more complete account of what clipping actually changes.
Critical weaknesses
  1. The initial benefit was already reported in DPZero (2024). The contribution therefore rests on the explanation, which remains mostly empirical; the asymptotic identity does not establish finite-smoothing convergence or accuracy gains.
  2. Downweighting difficult examples is plausible, but competing explanations are not fully separated. Comparisons with per-example normalized first-order gradients and robust losses would clarify what is specific to zeroth-order clipping.
Questions for authors

Can you disentangle how downweighting mislabeled examples versus correctly labeled but difficult examples contributes to the accuracy improvement and calibration deterioration?

3 comments

We thank you for your constructive feedback and positive assessment of Proposition 2.1, our experimental design, and our calibration analysis. Your suggestions to disentangle the training data subsets, compare against First-Order normalized gradients, and establish non-asymptotic theoretical guarantees helped us significantly strengthen the paper. We have incorporated all of these additions into the revised manuscript (highlighted in blue) and detail how they address each of your points below.

1. Disentangling Mislabeled Examples vs. Clean-Hard Examples (Question 1)

To directly separate how downweighting mislabeled examples versus correctly labeled but difficult or ambiguous examples affects accuracy and calibration, we added a controlled nested-subset pruning experiment in Section 4.3 and Appendix H.3 (Table 1 and Figure 32). Starting from the 10%10\%-corrupted SNLI training set (N=1,536N = 1,536), we recorded per-sample clipping frequencies during Clipped ZO-SGD training (η=10−5,c=50\eta = 10^{-5}, c = 50 over 10,00010,000 steps) and partitioned the training data into three disjoint subsets:

  • Subset AA (∼\sim 10%): consisting of the synthetically mislabeled examples.
  • Subset BB (∼\sim 15%): consisting of uncorrupted but naturally difficult or ambiguous examples whose directional derivatives exceeded c=50c = 50 more than 375375 times (375375 was based on the magnitudes in Figure 9).
  • Subset CC (∼\sim 75%): consisting of the remaining clean, low-clip examples (≤375\le 375 clips).

We then trained both Unclipped ZO-SGD (η=10−6\eta = 10^{-6}) and Clipped ZO-SGD (η=10−5,c=50\eta = 10^{-5}, c = 50) from scratch on the subsets CC, C+BC+B, C+AC+A, and C+B+AC+B+A:

Configurationη=10−6\eta = 10^{-6} (No Clipping)η=10−5\eta = 10^{-5} (c=50c = 50 Clipping)
CC0.81500.81500.81100.8110
C+BC+B0.80400.80400.81300.8130
C+AC+A0.78300.78300.79100.7910
C+B+AC+B+A0.77600.77600.79400.7940

There are several insights we can extract from this table:

  • First, notice that unclipped(C)\text{unclipped}(C) achieves better performance than unclipped(C+B)\text{unclipped}(C+B), where C+BC+B is the original training data; using less data achieves better test performance. This supports our claim that there are examples (BB) in the original train data that hurt performance, perhaps due to being ambiguous or hard examples.
  • Next, notice that unclipped(C)\text{unclipped}(C), clipped(C)\text{clipped}(C), and clipped(C+B)\text{clipped}(C+B) have about the same test accuracy. This proves our claim that the performance gain of clipped ZO-SGD on the original data (C+BC+B) is due to robustness: i.e., it can naturally downweight bad examples, achieving the same performance as when trained on only good examples in the original train data.
  • Next, clipped(C+B+A)\text{clipped}(C+B+A) performs better than unclipped(C+B+A)\text{unclipped}(C+B+A), but not as good as clipped(C+B)\text{clipped}(C+B). This further shows the non-robustness of unclipped ZO-SGD (also supported by our new theoretical analysis in Appendix K, Proposition K.4). This also shows that clipped ZO-SGD is more robust to the injection of such corrupted data (AA) (also supported by our new theoretical analysis in Theorem K.3). However, even the clipped ZO-SGD suffers with enough corrupted data. We believe this suggests that, perhaps, clipped ZO-SGD is more robust against natural ambiguous data than maliciously mislabeled data. This is confirmed by how on C+AC+A with malicious 10%10\% corruption, both clipped and unclipped ZO-SGD do worse than on C+BC+B with 15%15\% natural hard data.

As shown in Table 1, the accuracy improvements stem from suppressing both subsets. When training with Unclipped ZO-SGD, test accuracy degrades progressively as naturally difficult examples (BB) and mislabeled examples (AA) are introduced, dropping from 81.5%81.5\% on CC to 80.4%80.4\% on C+BC+B (−1.1%-1.1\%), and finally to 77.6%77.6\% on C+B+AC+B+A (−2.8%-2.8\%). In contrast, Clipped ZO-SGD is unaffected by the addition of naturally difficult data (81.1%81.1\% on CC vs. 81.3%81.3\% on C+BC+B) and remains resilient when synthetic label noise is added (79.4%79.4\% on C+B+AC+B+A). Crucially, when both AA and BB are manually removed (Subset CC), Unclipped ZO-SGD (81.5%81.5\%) fully recovers and matches Clipped ZO-SGD (81.1%81.1\%), demonstrating that per-sample clipping improves generalization by acting as an automatic filter against gradient spikes from both mislabeled outliers and ambiguous boundary cases.

Continued in next comment.....

Examining the corresponding reliability diagrams in Figure 32 reveals that these two data types have opposite effects on calibration. Introducing mislabeled examples (C+B→C+B+AC+B \to C+B+A) actually pulls Unclipped ZO-SGD toward underconfidence (ECE=0.0700\text{ECE} = 0.0700), because memorizing contradictory labels flattens the model's output logits across classes. Removing AA (C+BC+B) eliminates this label contradiction and restores well-calibrated predictions (ECE=0.0320\text{ECE} = 0.0320, with only ∼23%\sim 23\% of predictions in the top >0.93>0.93 confidence bin). Conversely, the naturally difficult boundary examples in Subset BB exert a strong gradient penalty against overconfident misclassifications. When Subset BB is removed (C+B→CC+B \to C), even Unclipped ZO-SGD becomes systematically overconfident: the fraction of predictions in the highest confidence bin (>0.93>0.93) more than doubles to >53%>53\%, and its ECE nearly doubles from 0.03200.0320 to 0.05730.0573. Clipped ZO-SGD exhibits even stronger confidence concentration on Subset CC (ECE=0.1487\text{ECE} = 0.1487, with ∼86%\sim 86\% of predictions in the >0.93>0.93 bin) because the continuous reweighting hc(∥∇fi(w)∥)h_c(\Vert\nabla f_i(w)\Vert) from Proposition 2.1 continues to downweight moderate-gradient boundary examples within CC. Thus, while suppressing both AA and BB improves test accuracy, the calibration deterioration (overconfidence) is specifically driven by suppressing the clean-hard boundary examples in BB.

2. Comparison with First-Order Per-Sample Clipping, Normalization, and Robust Losses (Weakness 2)

To clarify what is specific to Zeroth-Order clipping versus First-Order (FO) per-example gradient normalization, we added Section 3.5 (Figure 6), which evaluates First-Order SGD on SNLI across a full (η,c)(\eta, c) grid (η∈[5×10−8,5×10−1],c∈[1,500]∪{∞}\eta \in [5\times 10^{-8}, 5\times 10^{-1}], c \in [1, 500]\cup\{\infty\}) under both FO per-sample clipping and FO batch clipping on the exact same model and dataset setup. In FO-SGD, performance is governed primarily by the learning rate rather than the clipping threshold: both FO per-sample clipping and FO batch clipping achieve high test accuracy (86.0%86.0\%–88.6%88.6\%) across a wide band of hyperparameters without following a strict ηc=constant\eta c = \text{constant} ridge. For instance, at η=10−5\eta = 10^{-5}, varying cc from 5050 to ∞\infty in FO per-sample clipping changes test accuracy very little (86.3%86.3\%–87.5%87.5\%). At c=1c = 1, where FO per-sample clipping reduces to normalized per-example FO gradients, test accuracy reaches at most 87.4%87.4\% (compared to 87.0%87.0\% unclipped at η=10−5\eta = 10^{-5} and 87.8%87.8\% at the overall unclipped peak), and FO per-sample clipping (88.2%88.2\%) and FO batch clipping (88.6%88.6\%) achieve nearly identical peak performance.

In contrast, ZO per-sample clipping only converges to its peak accuracy along a strict diagonal ridge (ηc=5×10−4\eta c = 5\times 10^{-4}) and yields a much larger relative gain over its unclipped baseline (80.7%→83.2%80.7\% \to 83.2\% on SNLI in Figure 6, and 82.77%→86.40%82.77\% \to 86.40\% averaged across four tasks in Figure 1), whereas ZO batch clipping and Winsorized mean fail to improve accuracy (Section 3.4, Figure 5). This difference arises because FO clipping normalizes the deterministic gradient vector ∇fi(wt)\nabla f_i(w_t), whereas ZO per-sample clipping thresholds the scalar directional derivative ⟨∇fi(wt),ut⟩\langle \nabla f_i(w_t), u_t \rangle before multiplying by the random perturbation ut∼N(0,Id)u_t \sim \mathcal{N}(0, I_d) of norm ≈d\approx \sqrt{d}. Clipping the scalar finite difference in ZO simultaneously (i) induces a smoother objective via Gaussian CDF reweighting (which we derive explicitly in Appendix K.2.1, Eqs. 26–29) and (ii) caps the O(d)O(d) variance injected by the random direction utu_t (Eq. 65), shifting the stability boundary and allowing ZO-SGD to use learning rates up to 100×100\times larger than unclipped ZO-SGD without diverging.

Continued in next comment....

3. Non-Asymptotic Finite-Smoothing (λ>0\lambda > 0) Analysis & Provable Accuracy Gains (Weakness 1)

To move beyond the asymptotic λ→0\lambda \to 0 identity in Proposition 2.1 and establish finite-smoothing convergence and accuracy gains, we added Appendix K (summarized in Section 2, p. 3). First, in Appendix K.1 (Proposition K.1, Eqs. 12–23), we analyze the expected clipped gradient estimate for finite smoothing λ>0\lambda > 0 under the assumption that each sample loss fi(w)f_i(w) is ℓ\ell-smooth. We prove that

Eu[gλ,c(w)]=1∣Bt∣∑i∈Bthc(∥∇fi(w)∥)∇fi(w)+bλ,c(w)\mathbb{E}_u[g_{\lambda,c}(w)] = \frac{1}{\vert B_t\vert}\sum_{i\in B_t} h_c(\Vert\nabla f_i(w)\Vert)\nabla f_i(w) + b_{\lambda,c}(w)

where the finite-smoothing bias is bounded by ∥bλ,c(w)∥≤ℓλ2d(d+2)\Vert b_{\lambda,c}(w)\Vert \le \frac{\ell\lambda}{2}\sqrt{d(d+2)}. This recovers Proposition 2.1 as λ→0\lambda \to 0 while quantifying how small λ\lambda needs to be in practice relative to ℓ\ell and dd.

Second, in Appendix K.2, we analyze clipped ZO-SGD in the canonical setting of dd-dimensional linear regression (yi=xi⊤w∗+ziy_i = x_i^\top w^* + z_i, with zi∼N(0,σ2)z_i \sim \mathcal{N}(0, \sigma^2)). In Theorem K.3 and Appendix K.2.1 (Eqs. 26–63), we analyze clipped ZO-SGD when an adversary corrupts an α~\tilde{\alpha} fraction of training labels to arbitrary values. This proves that clipped ZO-SGD matches the robust regression lower bound of Diakonikolas et al. (2019) up to the log⁡(1/α~)\sqrt{\log(1/\tilde{\alpha})} factor, achieving a near-optimal error rate. This theoretically supports out explanation on why clipped ZO-SGD achieves higher accuracy than unclipped ZO-SGD under corruption together with Proposition K.4. Proposition K.4 shows that unclipped ZO-SGD can suffer arbitrarily large error ((1/σ)∥w^−w∗∥>C′(1/\sigma)\Vert\hat{w} - w^\ast\Vert > C') for any C′>0C' > 0 from even a single corrupted label. This is consistent with out observation on the clipping gain growing with the corruption fraction α~\tilde{\alpha} (Figure 8 and Figure 28).

We have uploaded the revised manuscript with all of these additions highlighted in blue, including the First-Order clipping comparisons in Section 3.5, the nested-subset accuracy and calibration ablations in Section 4.3 and Appendix H.3, and the finite-smoothing and robust regression proofs in Appendix K. Thank you again for your thoughtful feedback. Your suggestions pushed us to run the exact experiments and theoretical analyses needed to make the paper's mechanisms concrete. Please let us know if you have any further questions during the discussion period. Thank you!

Comments on the paper

1 comment

We sincerely thank the reviewers for their time and constructive feedback. We are encouraged that the reviews highlighted the clarity of Proposition 2.1, our loss landscape and calibration analyses, and our label-corruption and data-pruning experiments, while offering concrete suggestions to strengthen the theoretical foundation, baseline comparisons, and statistical rigor of the work. We have uploaded a revised manuscript with all additions highlighted in blue that directly addresses every question and concern raised across the reviews.

We added two new theoretical results. First, we extend our previous asymptotic λ→0\lambda \to 0 result (Proposition 2.1) on the expected clipped gradient estimate to a non-asymptotic λ>0\lambda > 0 regime (Proposition K.1 in Appendix K). Second, we analyze clipped ZO-SGD under adversarial label perturbation in order to establish the test-time performance gain compared to unclipped ZO-SGD (Theorem K.2, Theorem K.3, and Proposition K.4). Concretely, we analyzed clipped ZO-SGD in the canonical setting of dd-dimensional linear regression. In Theorems K.2 and K.3, we establish finite-step convergence, the ηc=constant\eta c = \text{constant} scaling law, and minimax-optimal sample complexity under adversarial corruption for a fraction of the training data. This shows that clipped ZO-SGD is robust against such adversaries. Together with Proposition K.4, which shows that unclipped ZO-SGD is not robust, we establish that when data is not clean, clipped ZO-SGD achieves better performance than unclipped ZO-SGD.

On the empirical side, we added First-Order (FO) SGD baselines across a full (η,c)(\eta, c) grid on SNLI under both FO per-sample clipping (including per-example normalized FO gradients at c=1c = 1) and FO batch clipping in Section 3.5 (Figure 6). Unlike FO methods, which achieve high test accuracy across a broad range of hyperparameters with minimal difference between per-sample and batch clipping, the strict ηc=constant\eta c = \text{constant} trade-off and large relative test accuracy gains are unique to Zeroth-Order per-sample clipping. In Section 4.3 and Appendix H.3 (Table 1 and Figure 32), we also disentangled the effects of mislabeled examples versus clean-hard boundary examples by partitioning the 10%10\%-corrupted SNLI dataset into intentionally mislabeled (AA), original label but ambiguous (BB), and original label and clear (CC) subsets and training models from scratch on CC, C+BC+B, C+AC+A, and C+B+AC+B+A. These ablations demonstrate that suppressing both AA and BB drives the test accuracy improvements, with Unclipped ZO-SGD on CC fully recovering to match Clipped ZO-SGD, whereas suppressing the clean-hard boundary examples in BB removes the natural gradient penalty against overconfident misclassifications and directly drives the observed overconfidence.

Finally, we strengthened our statistical evaluations and presentation throughout the paper. We updated Figure 5 (Section 3.4) to report the mean ± standard error across 5 independent random seeds, expanded our RoBERTa-SNLI sweep down to ultra-small thresholds c∈{0.1,0.5,1.0}c \in \{0.1, 0.5, 1.0\} in Appendix F.2 (Figures 22–24) to confirm that ηc=5×10−4\eta c = 5\times 10^{-4} scaling and high test accuracy persist down to c=0.1c = 0.1, and plotted per-iteration clip counts over training to show how clipping transitions from nearly the entire batch during the first ∼4,000\sim 4,000 steps to selectively targeting hard and mislabeled examples as the model settles. We also relocated the Edge of Stability (EoS) analysis to Appendix I with a brief pointer in the Conclusion, added a discussion differentiating our noise-free 1D directional ZO clipping from First-Order DP-SGD clipping in Rehn et al. (2026) in Appendix A.2, expanded our stationary-point and ensembling analysis in Section 3.2, and audited the manuscript to consistently specify "train accuracy" and "test accuracy" at every mention. We have posted detailed responses to each reviewer below and welcome any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.