- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how different gradient clipping strategies influence adversarial robustness under the same differential privacy guarantee. The authors compare joint clipping, which bounds the combined classification and robustness gradients, with component-wise clipping, which bounds the two gradients separately. Although both approaches satisfy the same privacy calibration and optimize the same nominal objective, the paper demonstrates they can produce different optimization updates and consequently different robustness outcomes.
Critical strengths
-
The paper isolates a specific optimization mechanism under shared privacy calibration. By holding the nominal training objective and privacy guarantee constant, the authors investigate how gradient clipping construction alone can influence the resulting optimization process and adversarial robustness. This provides a controlled comparison of an often overlooked implementation choice in differentially private training.
-
The authors extend their theoretical analysis beyond the basic directional-distortion argument, investigates how preserving the relative weights of gradient components within each example does not necessarily preserve the aggregate gradient update. The analysis of example-dependent magnitudes and the resulting aggregate distortion provides a more detailed understanding of the underlying optimization behavior.
Critical weaknesses
-
The novelty and significance of the primary theoretical contribution are somewhat limited. The central observation that clipping gradients separately rather than jointly produces different optimization updates follows directly from the nonlinear nature of gradient clipping. Furthermore, the work follows closely with the previously studied aggregate gradient misalignment and its downstream effects in differentially private training (Esipova et al., 2023). While the authors extend these observations through a more specific analysis of composite objectives and example-dependent weighting, it is not clear how much fundamentally new theoretical insight is provided beyond existing work.
-
The experimental evaluation is limited in terms of model–dataset combinations and generalizability. The experiments are conducted on only two image-classification settings: CIFAR-10 with ResNet-18 and GTSRB with ConvNeXt-T. Given the relatively broad conclusions regarding gradient clipping and adversarial robustness, additional evaluations across different architectures, datasets, or training settings would significantly strengthen the work.
-
The paper provides limited practical guidance despite demonstrating different robustness outcomes across clipping strategies. While the authors empirically demonstrate that different clipping strategies can yield substantially different adversarial robustness, the paper does not establish a reliable criterion for when one strategy should be preferred over another, nor does it adequately connect these outcomes to specific model or dataset characteristics. Consequently, substantial further work would be required to translate these findings into practical guidance for selecting clipping strategies in new settings.
Esipova, M. S., Ghomi, A. A., Luo, Y., & Cresswell, J. C. (2023). Disparate impact in differential privacy from gradient misalignment. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2206.07737
Questions for authors
-
Theoretical novelty: How does the proposed theoretical analysis, particularly Theorem 2, extend existing work on clipping-induced gradient bias and misalignment (Esipova et al., 2023)?
-
Practical applicability: Can the authors identify specific model–dataset characteristics or gradient properties that explain the differing robustness outcomes across experimental settings, and clarify how these findings could guide the selection of clipping strategies?
Esipova, M. S., Ghomi, A. A., Luo, Y., & Cresswell, J. C. (2023). Disparate impact in differential privacy from gradient misalignment. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2206.07737
2 comments
Response to Reviewer cwPF: Theoretical Contribution and Practical Guidance
W3/Q2. Practical choice of bounding rule
We recommend evaluating Joint (J) and magnitude restoration (M) as the first candidates for private training with composite objectives. J preserves the direction of each nonzero full gradient. M retains Componentwise directions while assigning each nonzero Componentwise contribution its corresponding Joint magnitude.
Across all four original settings at , each of the other three rules was outperformed by J or M on both mean clean accuracy and mean AutoAttack accuracy at (Tables 9 and 15). On CIFAR-10, J achieved the highest robustness and M the highest clean accuracy under both hard clipping and AUTO-S. On GTSRB, M led the original five rules on both metrics at , while J achieved the highest robustness at .
Including the additional fixed-norm control N, M has the highest mean clean accuracy and lies on the observed clean–AA Pareto frontier among all six tested rules in all four frozen settings. For a new task, we recommend selecting the rule on public validation data at the intended attack radius and clean-accuracy requirement. N offers another trade-off worth evaluating when robustness at is the priority: on GTSRB, it improved mean AutoAttack accuracy over M by 0.30 points under hard clipping and 0.50 under AUTO-S, with lower clean accuracy in both cases (Appendix E.1).
We thank the reviewer again for the thoughtful questions.
[1] Yang, Kong and Kim. Improving Adversarial Robustness by Putting More Regularizations on Less Robust Samples. ICML, 2023. https://proceedings.mlr.press/v202/yang23h.html
[2] Waseda, Chang and Echizen. Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off. ICLR, 2025. https://arxiv.org/abs/2402.14648
Response to Reviewer cwPF: Theoretical Contribution and Practical Guidance
We thank the reviewer for the thorough and constructive review. We particularly appreciate your questions about the theoretical contribution and how the results can guide the choice of a clipping rule in practice.
W1/Q1. Theoretical contribution and relation to Esipova et al.
Esipova et al. (2023) analyze how clipping-induced changes in the batch gradient's direction and magnitude affect disparate impact. We quantify how much distortion remains when two loss gradients are clipped separately, even with the best shared allocation of the clipping bound. Two examples can have identical total gradients and thus identical Joint updates, yet require different allocations to preserve the relative weights of their loss components.
Theorem 2 makes this precise. Let E(q) be the mean squared component of each example's bounded contribution perpendicular to its full gradient. On any interval where both loss components saturate for every example under hard clipping, we obtain
E(q)=C2wˉ[Vw+(q−qw)2],
where qw and Vw are the weighted mean and variance of the record-specific balance-preserving allocations qi∗=∥hi∥/(∥gi∥+∥hi∥), and wˉ is the mean weight. Adjusting the shared allocation q can reduce the second term. The first term remains whenever the balance-preserving allocations have positive weighted variance. For AUTO-S, we derive a corresponding lower bound without a saturation assumption. On our public probes, the analytic floors range from 2.24% to 4.67% of the mean squared Joint contribution across the four settings (Table 11).
Lemma 3 then answers what happens if each example receives its own qi∗. This restores the relative component weights and preserves each example's full-gradient direction. Its scale, however, depends on ∥gi∥+∥hi∥, whereas Joint uses ∥gi+hi∥. Cancellation between the two gradients therefore affects how much weight each example receives relative to Joint. Equation (13) gives the condition under which this reweighting rotates the batch update.
W2. Generalizability of the findings
Our central contribution concerns how per-example gradient bounding changes optimization for composite objectives; its mathematical scope extends beyond the particular models, datasets, and TRADES regularizer used in the experiments. This question arises when private optimization is combined with a data-dependent robustness penalty. Both loss components depend on the same training example and share trainable parameters, so their gradients can be bounded jointly or separately. The two constructions can satisfy the same certified privacy bound while changing the balance between the components and the relative influence of examples.
TRADES provides one experimental instance of this broader setting. Our theoretical results do not require a particular functional form of the robustness regularizer. The analysis uses the norms and relative directions of the task and regularization gradients, so it applies to other differentiable per-example penalties under the stated assumptions. Recent examples of this composite-objective structure include vulnerability-aware regularization (Yang et al., ICML 2023 [1]) and invariance regularization (Waseda et al., ICLR 2025 [2]). The revised Appendix A.1 explains this scope and its relation to other private optimizers.
Empirically, we test whether these distinctions have measurable training consequences across changes in architecture, data, and bounding function. ResNet-18 on CIFAR-10 provides an established setting for controlled comparisons; ConvNeXt-T on GTSRB adds a different architecture and higher-resolution traffic-sign images across 43 classes. In each setting, we evaluate both hard clipping and AUTO-S with three training seeds per rule and full-test AutoAttack. Within every matched comparison, the objective, initialization, sampling, and noise calibration are fixed.
We have also extended the evaluation to a stronger privacy budget. At (ε,δ)=(2,10−5), we retrained CIFAR U/M under both clipping maps, changing only the noise multiplier and keeping the remaining hyperparameters fixed. The official-test M–U differences are:
Percentage points, mean ± sample SD across three training seeds (45–47):
M exceeds U in AA accuracy in all three seeds under both maps (Appendix E.2). These additional experiments support the magnitude-restoration result under stronger privacy. The geometric analysis applies under its stated assumptions, and the experiments establish its relevance to learned robustness in the evaluated private fine-tuning settings.