acceptodds
Under review as a conference paper at ICLR 2027

Same Privacy Guarantee, Different Robustness: How Gradient Clipping Shapes Adversarial Training

Abstract

Trustworthy machine learning requires addressing multiple threats, including privacy leakage and adversarial perturbations. Differential privacy and adversarial robustness provide complementary forms of protection against these risks. Combining techniques for these goals within the same training procedure can change how a model learns. In private gradient-based training, bounding each example's contribution calibrates privacy noise but can also alter the balance between classification and robustness. The loss coefficients and privacy budget alone therefore do not specify the update used to learn these goals. We compare two ways of limiting gradient contributions: bounding classification and robustness gradients jointly or separately, with the same objective and privacy calibration. We analyze the bounded updates before noise is added, then use controlled fine-tuning experiments in two image-classification settings to test their robustness consequences under hard and smooth clipping. Our analysis shows that preserving the balance between loss components within each example need not preserve the combined update across examples. Across both settings and clipping maps, matching magnitudes to joint bounding while retaining componentwise directions improves held-out AutoAttack accuracy; restoring directions alone is less consistent. Joint exceeds Componentwise by over three percentage points in one setting, but their ordering changes with the setting and attack radius. Contribution magnitudes affect learned robustness even when per-example gradient directions are preserved.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
differential privacy, adversarial robustness, gradient clipping, private optimization, adversarial training, gradient geometry, composite objectives, DP-SGD

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cwPFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how different gradient clipping strategies influence adversarial robustness under the same differential privacy guarantee. The authors compare joint clipping, which bounds the combined classification and robustness gradients, with component-wise clipping, which bounds the two gradients separately. Although both approaches satisfy the same privacy calibration and optimize the same nominal objective, the paper demonstrates they can produce different optimization updates and consequently different robustness outcomes.

Critical strengths
  1. The paper isolates a specific optimization mechanism under shared privacy calibration. By holding the nominal training objective and privacy guarantee constant, the authors investigate how gradient clipping construction alone can influence the resulting optimization process and adversarial robustness. This provides a controlled comparison of an often overlooked implementation choice in differentially private training.

  2. The authors extend their theoretical analysis beyond the basic directional-distortion argument, investigates how preserving the relative weights of gradient components within each example does not necessarily preserve the aggregate gradient update. The analysis of example-dependent magnitudes and the resulting aggregate distortion provides a more detailed understanding of the underlying optimization behavior.

Critical weaknesses
  1. The novelty and significance of the primary theoretical contribution are somewhat limited. The central observation that clipping gradients separately rather than jointly produces different optimization updates follows directly from the nonlinear nature of gradient clipping. Furthermore, the work follows closely with the previously studied aggregate gradient misalignment and its downstream effects in differentially private training (Esipova et al., 2023). While the authors extend these observations through a more specific analysis of composite objectives and example-dependent weighting, it is not clear how much fundamentally new theoretical insight is provided beyond existing work.

  2. The experimental evaluation is limited in terms of model–dataset combinations and generalizability. The experiments are conducted on only two image-classification settings: CIFAR-10 with ResNet-18 and GTSRB with ConvNeXt-T. Given the relatively broad conclusions regarding gradient clipping and adversarial robustness, additional evaluations across different architectures, datasets, or training settings would significantly strengthen the work.

  3. The paper provides limited practical guidance despite demonstrating different robustness outcomes across clipping strategies. While the authors empirically demonstrate that different clipping strategies can yield substantially different adversarial robustness, the paper does not establish a reliable criterion for when one strategy should be preferred over another, nor does it adequately connect these outcomes to specific model or dataset characteristics. Consequently, substantial further work would be required to translate these findings into practical guidance for selecting clipping strategies in new settings.

Esipova, M. S., Ghomi, A. A., Luo, Y., & Cresswell, J. C. (2023). Disparate impact in differential privacy from gradient misalignment. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2206.07737

Questions for authors
  1. Theoretical novelty: How does the proposed theoretical analysis, particularly Theorem 2, extend existing work on clipping-induced gradient bias and misalignment (Esipova et al., 2023)?

  2. Practical applicability: Can the authors identify specific model–dataset characteristics or gradient properties that explain the differing robustness outcomes across experimental settings, and clarify how these findings could guide the selection of clipping strategies?

Esipova, M. S., Ghomi, A. A., Luo, Y., & Cresswell, J. C. (2023). Disparate impact in differential privacy from gradient misalignment. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2206.07737

2 comments
AuthorsOpenReview ↗

Response to Reviewer cwPF: Theoretical Contribution and Practical Guidance

We thank the reviewer for the thorough and constructive review. We particularly appreciate your questions about the theoretical contribution and how the results can guide the choice of a clipping rule in practice.

W1/Q1. Theoretical contribution and relation to Esipova et al.

Esipova et al. (2023) analyze how clipping-induced changes in the batch gradient's direction and magnitude affect disparate impact. We quantify how much distortion remains when two loss gradients are clipped separately, even with the best shared allocation of the clipping bound. Two examples can have identical total gradients and thus identical Joint updates, yet require different allocations to preserve the relative weights of their loss components.

Theorem 2 makes this precise. Let E(q)E(q) be the mean squared component of each example's bounded contribution perpendicular to its full gradient. On any interval where both loss components saturate for every example under hard clipping, we obtain

E(q)=C2wˉ[Vw+(q−qw)2],E(q)=C^2\bar w\left[V_w+(q-q_w)^2\right],

where qwq_w and VwV_w are the weighted mean and variance of the record-specific balance-preserving allocations qi∗=∥hi∥/(∥gi∥+∥hi∥)q_i^*=\|h_i\|/(\|g_i\|+\|h_i\|), and wˉ\bar w is the mean weight. Adjusting the shared allocation qq can reduce the second term. The first term remains whenever the balance-preserving allocations have positive weighted variance. For AUTO-S, we derive a corresponding lower bound without a saturation assumption. On our public probes, the analytic floors range from 2.24% to 4.67% of the mean squared Joint contribution across the four settings (Table 11).

Lemma 3 then answers what happens if each example receives its own qi∗q_i^*. This restores the relative component weights and preserves each example's full-gradient direction. Its scale, however, depends on ∥gi∥+∥hi∥\|g_i\|+\|h_i\|, whereas Joint uses ∥gi+hi∥\|g_i+h_i\|. Cancellation between the two gradients therefore affects how much weight each example receives relative to Joint. Equation (13) gives the condition under which this reweighting rotates the batch update.

W2. Generalizability of the findings

Our central contribution concerns how per-example gradient bounding changes optimization for composite objectives; its mathematical scope extends beyond the particular models, datasets, and TRADES regularizer used in the experiments. This question arises when private optimization is combined with a data-dependent robustness penalty. Both loss components depend on the same training example and share trainable parameters, so their gradients can be bounded jointly or separately. The two constructions can satisfy the same certified privacy bound while changing the balance between the components and the relative influence of examples.

TRADES provides one experimental instance of this broader setting. Our theoretical results do not require a particular functional form of the robustness regularizer. The analysis uses the norms and relative directions of the task and regularization gradients, so it applies to other differentiable per-example penalties under the stated assumptions. Recent examples of this composite-objective structure include vulnerability-aware regularization (Yang et al., ICML 2023 [1]) and invariance regularization (Waseda et al., ICLR 2025 [2]). The revised Appendix A.1 explains this scope and its relation to other private optimizers.

Empirically, we test whether these distinctions have measurable training consequences across changes in architecture, data, and bounding function. ResNet-18 on CIFAR-10 provides an established setting for controlled comparisons; ConvNeXt-T on GTSRB adds a different architecture and higher-resolution traffic-sign images across 43 classes. In each setting, we evaluate both hard clipping and AUTO-S with three training seeds per rule and full-test AutoAttack. Within every matched comparison, the objective, initialization, sampling, and noise calibration are fixed.

We have also extended the evaluation to a stronger privacy budget. At (ε,δ)=(2,10−5)(\varepsilon,\delta)=(2,10^{-5}), we retrained CIFAR U/M under both clipping maps, changing only the noise multiplier and keeping the remaining hyperparameters fixed. The official-test M–U differences are:

Percentage points, mean ± sample SD across three training seeds (45–47):

SettingCleanAA 8/255
CIFAR-10, hard+1.230±0.558+1.230\pm0.558+2.833±0.345+2.833\pm0.345
CIFAR-10, AUTO-S+0.927±0.176+0.927\pm0.176+1.097±0.137+1.097\pm0.137

M exceeds U in AA accuracy in all three seeds under both maps (Appendix E.2). These additional experiments support the magnitude-restoration result under stronger privacy. The geometric analysis applies under its stated assumptions, and the experiments establish its relevance to learned robustness in the evaluated private fine-tuning settings.

AuthorsOpenReview ↗

Response to Reviewer cwPF: Theoretical Contribution and Practical Guidance

W3/Q2. Practical choice of bounding rule

We recommend evaluating Joint (J) and magnitude restoration (M) as the first candidates for private training with composite objectives. J preserves the direction of each nonzero full gradient. M retains Componentwise directions while assigning each nonzero Componentwise contribution its corresponding Joint magnitude.

Across all four original settings at ε=4\varepsilon=4, each of the other three rules was outperformed by J or M on both mean clean accuracy and mean AutoAttack accuracy at 8/2558/255 (Tables 9 and 15). On CIFAR-10, J achieved the highest robustness and M the highest clean accuracy under both hard clipping and AUTO-S. On GTSRB, M led the original five rules on both metrics at 8/2558/255, while J achieved the highest robustness at 4/2554/255.

Including the additional fixed-norm control N, M has the highest mean clean accuracy and lies on the observed clean–AA Pareto frontier among all six tested rules in all four frozen settings. For a new task, we recommend selecting the rule on public validation data at the intended attack radius and clean-accuracy requirement. N offers another trade-off worth evaluating when robustness at 8/2558/255 is the priority: on GTSRB, it improved mean AutoAttack accuracy over M by 0.30 points under hard clipping and 0.50 under AUTO-S, with lower clean accuracy in both cases (Appendix E.1).

We thank the reviewer again for the thoughtful questions.

[1] Yang, Kong and Kim. Improving Adversarial Robustness by Putting More Regularizations on Less Robust Samples. ICML, 2023. https://proceedings.mlr.press/v202/yang23h.html

[2] Waseda, Chang and Echizen. Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off. ICLR, 2025. https://arxiv.org/abs/2402.14648

Reviewer_NtJbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper focuses on optimizing jointly for privacy and robustness, analyzing different methods of combining DP-SGD with a robust auxiliary training objective such as TRADES. Specifically, the paper focuses on the bounding method under DP-SGD, exploring joint vs. component-wise clipping, restoring relative loss component weights, and distortion quantification beyond loss coefficients. The paper runs experiments comparing the different bounding methods, arguing that correcting magnitude can improve robustness in this setup.

Critical strengths
  • S1. Interesting mathematical arguments comparing bounding approaches are given.
  • S2. Figures and tables are well-made, and related work is described clearly.
  • S3. Experimental analysis of the results is thorough, and much information is given for experimental reproducibility.
Critical weaknesses
  • W1. There is insufficient motivation for comparing bounding approaches under DP-SGD with a robust auxiliary training objective.

    • Before exploring variations of (DP-SGD + a robust auxiliary training objective), it seems important to first establish this training regime as a relevant area of study, as (to my knowledge) it seems not to be explored in prior literature. May developers be tempted to combine DP-SGD with a TRADES-like objective in practice? While DP-SGD and TRADES are both commonly studied, there are many different methods proposed to improve privacy and robustness separately, depending on the application; some more motivation is needed before exploring this specific combination in depth.
    • The authors may consider first presenting empirical privacy and robustness results of the most simple combination of DP-SGD and TRADES in comparison to vanilla-DP (as run in the paper) and vanilla TRADES, to answer whether the combination appears able to preserve both privacy and robustness while maintaining utility. Ideally this can be compared against [1] and [2], or justification can be given as to why this comparison is unnecessary. An answer to this question could further motivate the authors' interesting comparisons in bounding techniques.
    • Alternatively, the authors may consider spending more time in the introduction motivating their proposed training regime with relevant related work, though this may not be as effective as the above motivating experiment.
  • W2. The writing and mathematical notation is at times difficult to follow. Below are a few examples/suggestions, though these are non-exhaustive and I suggest the authors search for similar clarity issues in other areas of the manuscript:

    • Differential privacy, adversarial robustness, TRADES, joint clipping, and componentwise clipping are all defined before motivation is given as to why they are relevant / worth consideration;
    • 'Joint' and 'Componentwise' labels should be explicitly defined before using, and they should not be used in the abstract if a definition is not provided earlier;
    • "branch", to my knowledge, is not a common term to describe a term within a compound loss, and should be defined/clarified somewhere;
    • xx is used as input to the gradient-clipping function, which is potentially confusing when xix_i refers to inputs, not gradients;
    • JiJ_i and UiU_i are defined as two separate quantities;
    • The Φ\Phi notation for AUTO-S obscures that it is just a modified clipping function;
    • Line 104 (defining qq) could be clearer, and "not realized update proportions" is likely unnecessary to mention.
Questions for authors
  • What other robust training regimes could this analysis, or parts of this analysis, apply to, besides TRADES?
  • Could the authors please share their motivation for the ResNet-18 and ConvNeXt-T architecture choices for experiments?

[1] Hayes, Jamie, Borja Balle, and M. Pawan Kumar. "Learning to be adversarially robust and differentially private." arXiv preprint arXiv:2201.02265 (2022).

[2] Zhang, Yuan, and Zhiqi Bu. "Differentially private optimizers can learn adversarially robust models." arXiv preprint arXiv:2211.08942 (2022).

3 comments
AuthorsOpenReview ↗

Response to Reviewer NtJb: Motivation, Scope, and Presentation

We thank the reviewer for the thoughtful and detailed feedback. We particularly appreciate the time devoted to providing specific suggestions on the presentation and have taken them into account in preparing the revised manuscript.

W1/Q1. Motivation and applicability beyond TRADES

Our setting lies at the intersection of two widely studied research directions: differentially private optimization (e.g., DP-SGD and DP-AdamW) and regularization-based adversarial training, which combines classification objectives with robustness regularizers (e.g., vulnerability-aware regularization by Yang et al. [3] and invariance regularization by Waseda et al. [4]).

Our theoretical contribution has a broader scope: it characterizes how per-record gradient bounding changes the component balance and aggregate update for composite objectives, independently of the specific robustness regularizer. TRADES provides a well-established experimental instance, but our geometric results depend on the component gradients rather than the particular KL divergence. They extend to other decomposable objectives when their components are bounded separately and the stated assumptions hold (Appendix A.1).

For such objectives, one can bound either the summed gradient or the individual components before aggregation. Although both strategies can satisfy the same certified privacy bound, they need not produce the same updates. Understanding this distinction motivates our analysis.

As discussed in our introduction, Hayes et al. [1] and Bu and Zhang [2] establish the broader relevance of private adversarial robustness. Our work investigates a different question: how gradient bounding changes optimization under a fixed objective and privacy calibration.

We also completed the suggested comparison with non-private robust training:

CIFAR-10 / ResNet-18, official test accuracy (%, mean ± sample SD across three seeds):

Training regimeCleanAA 8/255
Natural-DP, ε=4\varepsilon=468.83±0.2368.83\pm0.2322.16±0.2422.16\pm0.24
DP + robust objective (Joint), ε=4\varepsilon=467.24±0.0867.24\pm0.0830.03±0.2830.03\pm0.28
Non-private robust training81.30±0.2681.30\pm0.2632.64±0.1332.64\pm0.13

At these development-selected operating points, private robust training achieves substantially higher AA accuracy than Natural-DP, while the non-private result provides a reference for the remaining privacy–utility gap. All methods share the public initialization and evaluation protocol. The non-private baseline removes both bounding and Gaussian noise, and its learning rate was tuned separately. The historical private results use seeds 42–44; the new non-private results use 45–47.

This comparison motivates studying the optimization behavior of private robust objectives, rather than establishing superiority over existing private adversarial-training methods. The revised Appendix E.6 includes the baseline's full configuration, development selection and official-test results; Table 23 gives the individual seed results.

W2. Presentation and mathematical notation

We have substantially revised the manuscript to improve clarity, readability, and the logical flow of the presentation. Your detailed suggestions guided changes throughout the paper.

Q2. Architecture choices

Our aim is to support the theoretically motivated distinctions with empirical evidence across different architectures and datasets, rather than conduct an exhaustive architectural benchmark.

ResNet-18 on CIFAR-10 provides a standard, extensively studied setting for controlled comparisons. ConvNeXt-T is a more recent architecture, which we evaluate on GTSRB, a real-world traffic-sign recognition task with higher-resolution images. Together, these experiments examine the proposed geometric distinctions in two different training settings.

We thank the reviewer again for the careful and constructive feedback.

[1] Hayes, Balle and Kumar. Learning to Be Adversarially Robust and Differentially Private. 2022. https://arxiv.org/abs/2201.02265

[2] Bu and Zhang. Differentially Private Optimizers Can Learn Adversarially Robust Models. TMLR, 2023. https://openreview.net/forum?id=o8VgRNYh6n

[3] Yang, Kong and Kim. Improving Adversarial Robustness by Putting More Regularizations on Less Robust Samples. ICML, 2023. https://proceedings.mlr.press/v202/yang23h.html

[4] Waseda, Chang and Echizen. Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off. ICLR, 2025. https://arxiv.org/abs/2402.14648

Apologies for the late response; thank you for your thorough reply and the additional experiments. I will raise my score.

AuthorsOpenReview ↗

Response to Reviewer NtJb

Thank you for taking the time to revisit our work and for your thoughtful consideration of our response. We sincerely appreciate your feedback.

Reviewer_NuWcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how gradient clipping affects adversarial robustness in differentially private training. It compares clipping classification and robustness gradients jointly versus separately, showing that these choices can produce different updates despite sharing the same privacy guarantee. The paper is experimentally evaluated on CIFAR-10 and GTSRB. It demonstrates that preserving each example’s gradient direction does not necessarily preserve the aggregate update, because clipping also changes the relative weighting of examples. Adjusting the magnitudes of separately clipped gradients consistently improves robustness in the tested settings, while restoring their directions has less consistent benefits.

Critical strengths
  1. The analysis shows that preserving the relative weights of classification and robustness gradients within each example does not necessarily preserve the batch update, because example-dependent scaling can rotate the aggregate direction. This provides a useful explanation of how clipping choices affect optimization even under the same privacy calibration.

  2. The comparisons hold key training settings fixed and evaluate direction and magnitude interventions across two datasets (CIFAR-10 and GTSRB), hard and smooth clipping, and fresh training seeds. Magnitude restoration consistently improves held-out AutoAttack accuracy over Componentwise clipping in the tested settings, providing empirical support for the importance of contribution magnitudes alongside gradient directions.

Critical weaknesses
  1. It is not clear if a simpler adjustment would provide the same improvement. The paper shows that replacing Componentwise gradient magnitudes with Joint gradient magnitudes improves robustness (Table 1, equation 16). However, it does not compare this approach with simply normalizing the Componentwise gradients to a fixed length. In some cases, these two approaches might be identical. This missing comparison makes it difficult to determine if Joint’s magnitudes provide useful information beyond a simpler change in gradient length.

  2. It seems that the main findings are tested under a limited range of training conditions. The matched experiments use one privacy budget and one robustness-loss weight, and update only part of an already robust pretrained model. The results support the conclusions under these conditions, but it remains unclear whether the same improvements would occur with stronger privacy protection, a different robustness weight, or training the entire model.

A minor comment: It is a good practice to include the code as it improves reproducability of the work.

Questions for authors
  1. I was wondering if you have compared magnitude restoration with simply normalizing Componentwise contributions to a fixed norm or rescaling them before clipping? If so, does magnitude restoration still improve robustness?

  2. Can you please provide a focused comparison at another privacy budget or robustness-loss weight to assess whether the magnitude-restoration improvement persists?

  3. Across the intervention runs, how does the norm of the mean noiseless query, relative to Joint, evolve during training? Could the robustness improvements be explained largely by changes in aggregate signal strength under fixed noise, and what evidence distinguishes this explanation from the effects of which examples receive greater weight?

  4. How exactly was c=0.8140694935c=0.8140694935 selected for the uniform-attenuation control? During training, how closely does this control match the radial rule’s aggregate query norm?

2 comments
AuthorsOpenReview ↗

Response to Reviewer NuWc: Additional Experiments and Gradient Analysis

We sincerely thank the reviewer for the thoughtful and constructive review. We particularly appreciate the careful engagement with our analysis and the concrete suggestions for additional experiments, which helped us strengthen and clarify our findings.

W1/Q1. Fixed-norm normalization

We introduced magnitude restoration (M) to investigate the role of per-record gradient magnitudes in the robustness differences between Joint clipping (J) and Componentwise clipping (U). To isolate this factor, M preserves U's nonzero directions while replacing its per-record magnitudes with those of J: Mi=∥Ji∥ unit(Ui)M_i=\|J_i\|\,\mathrm{unit}(U_i), where unit(v)=v/∥v∥\mathrm{unit}(v)=v/\|v\| for v≠0v\neq0 and zero otherwise.

Beyond its role as a controlled intervention, M achieves one of the strongest clean–robustness trade-offs among the rules we investigate.

We additionally evaluated the reviewer's suggested fixed-norm normalization (N), Ni=B unit(Ui)N_i=B\,\mathrm{unit}(U_i) (B=CB=C for hard clipping; B=1B=1 for AUTO-S), at all four original ε=4\varepsilon=4 settings. N and M coincide for saturated hard Joint contributions when Ui≠0U_i\neq0.

M − N, official-test accuracy differences (percentage points, mean ± sample SD):

SettingCleanAA
CIFAR-10, hard+2.237±0.257+2.237\pm0.257+0.163±0.408+0.163\pm0.408
CIFAR-10, AUTO-S+2.853±0.206+2.853\pm0.206−0.080±0.113-0.080\pm0.113
GTSRB, hard+0.628±0.139+0.628\pm0.139−0.296±0.179-0.296\pm0.179
GTSRB, AUTO-S+0.148±0.036+0.148\pm0.036−0.504±0.180-0.504\pm0.180

On CIFAR-10, M attains substantially higher clean accuracy at comparable robustness. On GTSRB, N attains somewhat higher robustness, while M shows slightly higher clean accuracy.

Among all six evaluated rules (J, U, M, N, direction-restored D, and norm-proportional radial R), M achieves the highest mean clean accuracy and lies on the observed clean–AA Pareto frontier in all four settings. The suggested comparison thus further supports M as a competitive choice, while the original M–U intervention examines the consequences of changing per-record magnitudes without changing their nonzero directions. The revised Appendix E.1 includes this comparison; Table 23 gives all individual results. Differences align seed labels; N uses independent private randomness.

W2/Q2. Stronger privacy and training scope

We repeated CIFAR-10 U/M under hard clipping and AUTO-S at (ε,δ)=(2,10−5)(\varepsilon,\delta)=(2,10^{-5}), changing only the noise multiplier; all other hyperparameters remained fixed.

M − U at ε=2\varepsilon=2 (percentage points, mean ± sample SD):

SettingCleanAA
CIFAR-10, hard+1.230±0.558+1.230\pm0.558+2.833±0.345+2.833\pm0.345
CIFAR-10, AUTO-S+0.927±0.176+0.927\pm0.176+1.097±0.137+1.097\pm0.137

Both AA gains are positive across all three seeds, confirming that the improvement persists under stronger privacy in both CIFAR settings (Appendix E.2).

Appendix C.4 explores β∈{0,1,2,3}\beta\in\{0,1,2,3\} for hyperparameter selection; matched M–U tests fix β=1\beta=1 to isolate the bounding rule. Partial fine-tuning is our chosen controlled private-adaptation setting; full-network training remains outside the empirical scope.

Q3. Aggregate signal strength

We measured noiseless aggregate norms on 512 fixed public examples. Each entry gives U/Joint, M/Joint, respectively, on the methods' own training trajectories at ε=4\varepsilon=4 (three-seed means):

SettingStartMiddleEnd
CIFAR hard0.699 / 0.9780.625 / 1.0080.621 / 1.028
CIFAR AUTO-S0.776 / 0.9880.719 / 0.9980.699 / 0.988
GTSRB hard0.629 / 0.9100.834 / 1.0030.766 / 0.942
GTSRB AUTO-S0.770 / 1.0150.663 / 0.8890.646 / 0.841

The middle checkpoint is epoch 25 for CIFAR and approximately epoch 30 for GTSRB. The revised Appendix E.3 includes the complete trajectory plots (Figure 6) and numerical series (Table 20).

M has a larger public aggregate signal than U; signal strength may therefore contribute to its robustness gain. To investigate this, we trained cM, uniformly attenuating M using a coefficient calibrated at public initialization while preserving its relative example weights and the original Gaussian noise.

On CIFAR AUTO-S, the same-state public aggregate-norm ratio cM/U stayed within 0.972–1.041. cM still gained +0.350±0.070+0.350\pm0.070 points of AA over U (versus +0.917±0.110+0.917\pm0.110 for M), although its clean accuracy was 0.783 points lower than U's.

The separately trained trajectories are not perfectly scale-matched (final cM/U ratio 1.098±0.0491.098\pm0.049), so this does not establish a causal decomposition. The geometric distinction remains exact: ∥Mi∥=∥Ji∥\|M_i\|=\|J_i\| whenever Ui≠0U_i\neq0, yet at final GTSRB AUTO-S M states, ∥M‾∥/∥J‾∥≈0.694\|\overline M\|/\|\overline J\|\approx0.694. Appendix E.4 gives the calibration and results.

AuthorsOpenReview ↗

Response to Reviewer NuWc: Additional Experiments and Gradient Analysis

Q4. Calibration of the original uniform coefficient

For the original cJcJ versus radial R control (distinct from cM above), c=0.8140694935c=0.8140694935 was calibrated once on 512 public initialization examples to match the sums of per-record norms:

c=∑i∥Ri∥∑i∥Ji∥.c=\frac{\sum_i\|R_i\|}{\sum_i\|J_i\|}.

The coefficient remained fixed throughout training and was not designed to match aggregate norms. Along the R trajectory, the three-seed mean of ∥cJ‾∥/∥R‾∥\|c\overline J\|/\|\overline R\| changed from 0.934 to 0.838, while the angle between the aggregate vectors increased from approximately 7∘7^\circ to 28∘28^\circ. Even a dynamically adjusted scalar could not eliminate this directional discrepancy, illustrating why uniform attenuation cannot reproduce the aggregate geometry of example-dependent reweighting (Appendix E.5).

Code. Code and reproducibility materials are available in the repository linked in the submitted manuscript: https://github.com/quiet-gradient/clipping-shapes-robustness. We have expanded the repository to include the additional experiments, with configurations, per-seed results and public aggregate-norm trajectories; instructions are provided in the README under “Additional experiments”.

All accuracy results above use the development-selected base configurations in Table 6, three training seeds (45–47) per method, and full-test AutoAttack at ℓ∞\ell_\infty radius 8/2558/255.

We thank the reviewer again. These results reinforce our central conclusion: bounding rules change aggregate updates and learned robustness despite a shared objective and certified privacy bound.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We have uploaded a revised manuscript addressing the reviewers’ comments. We clarified the motivation, notation, and scope of the analysis, and added further experiments and diagnostics in Appendix E. Our responses to individual reviewers describe these additions and point to the relevant sections, figures, and tables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.