acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Certified Defense with Differential Privacy on Vision Transformers

Abstract

Certified defenses that incorporate differential privacy have proven effective on Convolutional Neural Networks (CNNs), furnishing rigorous robustness guarantees against norm-bounded adversaries. However, the certified robustness behavior of Pixel Differential Privacy (PixelDP) remains largely unexplored with the self-attention architecture now dominating the deep-learning landscape. Given that the Transformer has a profound impact on our daily applications from the digital world to the physical world, it is crucial to study certified robustness through differential-privacy-style stability. To fill this research gap, we revisit this construction in Vision Transformers and identify a failure mode that is largely hidden in the convolutional setting. When noise is injected after the patch embedding, the Laplace mechanism with the inherited grouped sensitivity bound collapses to chance-level accuracy across noise scales, whereas the Gaussian mechanism remains trainable. This contrast isolates the source of failure: not the injected noise itself, but the geometry of the sensitivity constraint. We show that the attenuation induced by the inherited projection increases with layer width and kernel size according to a random-matrix scale . Replacing the -type constraint with a spectral-norm constraint eliminates the collapse across datasets and architectures, but creates a fundamental obstacle: the repaired models no longer satisfy the sensitivity condition required by the standard Laplace certificate. We resolve this mismatch by deriving a dimension-free -privacy guarantee for the Laplace mechanism under sensitivity through concentration of the privacy loss. The resulting certificate applies directly to the repaired models without retraining and avoids the privacy degradation incurred by the naive conversion from to sensitivity. Experiments on CIFAR-10, CIFAR-100, and SVHN reproduce the failure-and-repair pattern, while ImageNet confirms that the inherited grouped-bound failure persists at scale. Finally, matched-budget experiments show that adversarial training improves certified accuracy at the evaluated positive radii.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
certified robustness, differential privacy, vision transformer

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DtvuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper extends Pixel DP from CNN to Vision Transformers, and proposes several solutions to this problem.

Critical strengths

None

Critical weaknesses
  1. Poor writing, for example, serveral symbol was first introduced without any explaination.
  2. The method/algorithm is not clear, where and how do you add laplace noise?
  3. No baseline is reported in the experiments.
Questions for authors
  1. Why extend PixelDP to ViT is an important research question?
  2. Could you please revise the paper with a focus on readability?
2 comments

Dear Reviewer Dtvu,

Thank you for pointing out the specific presentation issues. We have made targeted clarifications to address them, particularly regarding the noise-injection location, the sensitivity notation, and the organization of experimental baselines.

Our intention was to make the construction easier to reproduce and the mathematical distinctions easier to follow, while preserving the overall structure of the paper.

We now explicitly define the randomized representation immediately after the patch embedding, distinguish the inherited grouped and exact stride-aware sensitivity norms, and clarify how the Gaussian, Laplace, and input randomized-smoothing baselines relate to our construction.

1. Explicit noise-injection procedure.

We now state the mechanism directly in Section 3.1. Noise is injected immediately after the non-overlapping patch embedding and before the first self-attention block.

If the patch embedding produces g(x)g(x), the randomized representation is g~(x)=g(x)+Z\widetilde g(x)=g(x)+Z, where ZZ is sampled independently from the calibrated Laplace or Gaussian mechanism. The remaining ViT processes this randomized representation normally. During training, the selected sensitivity constraint is enforced after every optimizer step.

We also explain why this placement is used: the patch embedding is linear and admits an explicit pixel-space sensitivity bound, whereas standard dot-product self-attention does not provide the same simple global bound on an unbounded domain.

2. Clarified mathematical notation.

We revised the notation around the patch-embedding sensitivity in Appendix A.2. In particular, we explicitly distinguish the inherited grouped bound Δˉ1,1\bar{\Delta}_{1,1} from the exact stride-aware induced norm Δ1,1exact\Delta_{1,1}^{\mathrm{exact}}. The spectral sensitivity constraint is denoted separately by Δ2,2\Delta_{2,2}.

We also clarify the meaning of the construction scale, inference-time noise scale, and candidate certification radius to avoid conflating these quantities.

3. More explicit experimental baselines.

We have reorganized the comparisons to make the competing constructions easier to identify.

The main comparison includes Gaussian and Laplace PixelDP configurations. The constraint-only study in Table 5 now explicitly compares inherited grouped ℓ1\ell_1, exact stride-aware ℓ1\ell_1, and spectral ℓ2\ell_2 constraints across CIFAR-10, CIFAR-100, and SVHN and four construction scales.

At L=b=0.03L=b=0.03, the three constraints achieve 27.91%, 69.65%, and 86.59% on CIFAR-10 in the three-seed experiment. We additionally report a larger ViT-Small control and the full exact-ℓ1\ell_1 certification results.

We retain the input Gaussian randomized-smoothing comparison and matched-budget adversarial-training baselines, now with clearer explanations of what each comparison measures.

4. Expanded related work and clearer motivation.

We have added discussion of SPLITZ, latent-space randomized smoothing, Smoothed ViTs, diffusion-denoised smoothing, and prior PixelDP extensions.

The revised motivation is also more precise: we do not argue that Vision Transformers lack certified defenses. Instead, we investigate whether PixelDP's intermediate randomized-representation construction can be transferred to self-attention architectures while retaining pixel-space perturbation semantics and a valid post-processing certificate.

We have revised the Introduction, Related Work, method description, and experimental discussion accordingly.

We appreciate your emphasis on readability. These changes were intended to make the noise placement, sensitivity constraints, and baseline comparisons explicit without changing the underlying experimental methodology.

We hope these targeted changes resolve the readability and methodological concerns raised in your review. We would sincerely appreciate your reconsideration of the current evaluation in light of the revised presentation and additional experimental evidence.

Thank you again for your helpful feedback.

Yours Sincerely,

Submission 17899 Team

Reviewer Dtvu,

Thank you again for emphasizing the presentation and methodological-clarity issues.

We have addressed these concerns directly in the final manuscript. The revised method section now states explicitly that noise is injected immediately after the non-overlapping patch embedding and before the first self-attention block. If the patch embedding produces g(x)g(x), the deployed randomized representation is g(x)+Zg(x)+Z, and the corresponding sensitivity constraint is enforced after every optimizer step.

We also revised the notation to distinguish the inherited grouped ℓ1\ell_1 bound, the exact stride-aware ℓ1\ell_1 operator norm, and the spectral ℓ2\ell_2 constraint, and we clarified the roles of the construction scale, inference-time noise scale, and certification radius.

The experimental comparisons have likewise been reorganized. The paper now includes explicit grouped-ℓ1\ell_1, exact-ℓ1\ell_1, and spectral-ℓ2\ell_2 constraint controls, Gaussian and Laplace PixelDP configurations, a matched input randomized-smoothing baseline, multi-seed validation, a larger ViT-Small control, and matched adversarial-training controls. We therefore believe the method and the relevant baselines are now stated unambiguously.

Finally, the motivation has been sharpened. The paper does not claim that ViTs lack certified defenses. It asks a more specific question: whether PixelDP's internal randomized-representation construction can be transferred to a ViT while preserving an enforceable pixel-space sensitivity bound and a valid certificate.

We appreciate that the original presentation did not make all of these distinctions sufficiently explicit. The final revision addresses the specific readability concerns while preserving the underlying technical structure of the work.

Thank you again for the feedback, which helped us make the manuscript considerably clearer and easier to reproduce.

Submission17899 Authors

Reviewer_k6cQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

From my understanding, the paper tries to apply PixelDP to Vision Transformers. Because self-attention has no simple pixel-space Lipschitz bound, noise is injected right after the patch embedding, with that layer’s sensitivity renormalized to 1 after each step.

The main empirical observation is that the Laplace arm, using PixelDP’s inherited grouped ℓ1 bound (Eq. 10, the max over input channels of the summed |W| across output channels and kernel positions), falls to near chance on CIFAR-10/100 and SVHN with ViT-Tiny even at L=0.03. The Gaussian arm under a spectral constraint trains normally.

Experiments cover an ImageNet run using PixelDP’s autoencoder front end with Inception-v3 and ViT-B backbones; a comparison to input randomized smoothing; a matched-budget ablation in which ℓ2 adversarial training raises certified accuracy.

Critical strengths
  1. Careful and honest diagnostics as the paper doesn’t stop at “Laplace fails” and runs several controls includinga constraint-only control (Table 5: grouped 28.87% → exact stride-aware ℓ1 70.83% → spectral 85.50%); a placement control showing that noise after attention is simply absorbed (Fig. 3); tracking of ρ during training (Fig. 5), which honestly shows the initialization law stops being quantitatively accurate after about 20 epochs; a direct comparison to input randomized smoothing that the method loses (Table 6).
Critical weaknesses
  1. Missing related work for an ICLR paper on certified ViTs.: Diffusion-denoised smoothing (Carlini et al., ICLR 2023) already certifies off-the-shelf transformer classifiers against ℓ2 at scale, with 71% ImageNet certified accuracy at radius 0.5. Smoothed ViTs (Salman et al., CVPR 2022) exploit ViT-specific structure for certification. Yang et al. (ICML 2020, “Randomized Smoothing of All Shapes and Sizes”) analyze Laplace smoothing and the dimension dependence of ℓp certificates. That is directly relevant to the √d discussion and to the claim that the 2-norm Laplace row was “absent for exactly this reason”. Phan et al. (IJCAI 2019, Heterogeneous Gaussian Mechanism) extend PixelDP with first-hidden-layer noise and a tighter robustness bound.
  2. The certified radii are also very small. Everything is ≤0.05 for the repaired Laplace models, and the Gaussian AT models drop to 0 at T=0.2, whereas the smoothing literature reports CIFAR-10 radii of 0.25–1.0.
  3. The claimed link between ρ and collapse is inconsistent across the paper’s own settings: ResNet-50’s first layer has a measured ρ≈38 (Fig. 2) and trains fine, while the ViT’s trained ρ≈55 collapses to chance. It is not clear why a 1.4× difference in ratio would mean the difference between 85% and 29%. On ImageNet, the PixelDP front end (32 filters, 10×10, 3 input channels) gives, by the paper’s own formula, ρ ≈ 3200·√(2/π)/(√32+√300) ≈ 111, about the same as the ViT-Tiny patch embedding (118). Yet Δ̄1,1 Laplace reaches 58–60% at L=0.03 there (Table 2), not chance. Gaussian under the spectral constraint also collapses to the majority class on SVHN with ViT at L≥0.3 (DP), and so does the spectrally repaired Laplace. Collapse is therefore not specific to the grouped bound.
Questions for authors
  1. Andersson et al.already give (ε,δ)-DP guarantees for the Laplace mechanism in terms of both ℓ1 and ℓ2 sensitivity. Is your theorem 3.1 rederivation? How does Thm. 3.1’s ε compare numerically with Andersson et al.’s ℓ2-sensitivity Laplace bound, and with a tight numerical privacy-loss accountant, at your (ε0, δ)? If either is tighter, using it would directly enlarge the certified radii.
  2. Why is the exact stride-aware ℓ1 bound (Table 5) not the main repair? It keeps the pure ε-DP ℓ1 certificate with no change of threat model. What are its full Table 1 sweep and certified accuracy across L and datasets? If it closes most of the gap, what does the ℓ2/approximate-DP detour add?
5 comments

Dear Reviewer k6cQ,

Thank you again for the detailed technical review. We have carefully reconsidered the main claims and completed the additional experiments suggested in your review. The revised manuscript reflects several substantive changes.

1. Exact stride-aware ℓ1\ell_1 is now a first-class repair.

We agree that the exact stride-aware ℓ1\ell_1 operator norm should not be treated merely as an ablation. We extended the comparison to the full three-dataset, four-scale grid.

At L=0.03L=0.03, grouped ℓ1\ell_1 / exact ℓ1\ell_1 / spectral ℓ2\ell_2 achieve:

  • CIFAR-10: 28.87% / 70.83% / 85.50%.
  • CIFAR-100: 7.51% / 44.36% / 59.92%.
  • SVHN: 19.59% / 88.92% / 96.36%.

A fresh three-seed CIFAR-10 experiment at L=b=0.03L=b=0.03 further gives:

27.91±0.03,69.65±1.36,86.59±1.40.27.91\pm0.03,\quad 69.65\pm1.36,\quad 86.59\pm1.40.

The same ordering persists on ViT-Small, where the corresponding single-seed results are 24.53%, 57.03%, and 84.57%.

Thus, the inherited grouped bound explains a substantial part of the original degradation, but the additional spectral trainability gain remains evident after this conservatism is removed. We have reorganized the discussion around these two distinct effects rather than presenting spectral control as the unique repair.

2. Exact-ℓ1\ell_1 certified accuracy.

We completed the requested certification over all three datasets and four construction scales. The exact-ℓ1\ell_1 construction retains its native pure-DP guarantee, ϵ(T)=T/L\epsilon(T)=T/L and δ=0\delta=0.

For example, at L=0.1L=0.1, the certified accuracies at T=0.03,0.05,0.10T=0.03,0.05,0.10 are:

  • CIFAR-10: 46.29%, 41.52%, 29.59%.
  • CIFAR-100: 20.87%, 17.96%, 11.60%.
  • SVHN: 73.60%, 67.72%, 49.75%.

The full certification table is now included as Table 7 in Appendix D.4. We explicitly distinguish the ℓ1\ell_1 and ℓ2\ell_2 threat models and avoid interpreting equal numerical radii as equivalent guarantees.

3. Andersson et al. and numerical privacy-loss accounting.

We verified that our closed-form ℓ2\ell_2-sensitive Laplace expression is algebraically identical to the corresponding Andersson et al. guarantee. We have removed the implication that this inequality constitutes an independent theoretical contribution and now explicitly present it as an existing result specialized to our PixelDP construction.

Following your suggestion, we also evaluated privacy-loss distributions for representative equal-coordinate shift geometries. For b=0.1b=0.1, δ=0.05\delta=0.05, and T=0.02,0.03,0.05,0.10T=0.02,0.03,0.05,0.10, the closed-form values are:

0.5095,0.7793,1.3489,2.9477,0.5095,\quad 0.7793,\quad 1.3489,\quad 2.9477,

while the largest values over the tested shift family are:

0.0974,0.2039,0.5143,1.5689.0.0974,\quad 0.2039,\quad 0.5143,\quad 1.5689.

These results indicate substantial conservatism for the tested geometries. Importantly, we do not present the latter values as improved certificates because we have not established that the tested family contains the worst-case shift over the entire ℓ2\ell_2 ball. They are now explicitly labeled as a numerical tightness diagnostic.

4. Clarifying the role of the norm ratio and the empirical scope.

We have revised the interpretation of ρgrp\rho_{\rm grp}. It quantifies initialization-scale attenuation induced by the grouped projection, rather than serving as a universal threshold for optimization collapse. The discussion now distinguishes grouped-bound-induced degradation from generic high-noise failure.

We also expanded Related Work to cover the certified-ViT and intermediate-representation methods identified in your review, and clarified the distinction between fixed-mechanism re-accounting and changing the deployed Laplace noise scale.

Taken together, these revisions substantially change the paper's positioning. Its central empirical result is now the separation between inherited-bound conservatism and a residual sensitivity-geometry effect, supported by full-grid experiments, three-seed validation, a larger-backbone control, and certification under the respective threat models.

We appreciate that your review identified the most important weaknesses in the original presentation. We hope that the additional evidence and the revised theoretical positioning address the central soundness concerns. If you find these changes sufficient, we would be grateful if you would consider reconsidering your current rating.

Thank you for your time and detailed feedback.

Yours Sincerely,

Submission 17899 Team

I re-read the paper and other reviews (with their responses once again). The rebuttal is honest and ran the experiments I (and other reviewers) asked for. But the response makes the paper smaller, not stronger. Especially the theoretical contribution is identical to Andersson et al.

More generally, I don't understand why why use PixelDP on ViTs when input smoothing on the same ViT already (Table 6 in previous version, Table 8 in current). I'm keeping my score

Thank you for re-reading the revised paper and the discussion. We appreciate that you recognize that the requested experiments were carried out. We would like to clarify one point that may still be causing a mismatch in how we view the contribution.

We fully agree that the ℓ2\ell_2-sensitive Laplace accountant itself is not novel. The revised manuscript explicitly states that the closed-form guarantee is algebraically identical to the corresponding result of Andersson et al., and we no longer present it as an independent theoretical contribution.

However, this accountant is not the entirety of the theoretical analysis in the paper. Our original analysis is the norm-ratio and attenuation result in Proposition 3.1 and Corollary 3.1, which characterizes how the inherited grouped PixelDP constraint interacts with the geometry of a ViT patch embedding. The new exact-ℓ1\ell_1 experiments further show that this distinction is not merely analytical: removing grouped-bound conservatism recovers a large fraction of the lost accuracy, while a residual gap remains under a different sensitivity geometry. We will make this separation between the existing accountant and our attenuation analysis even more explicit in the revised text.

Regarding input smoothing, we agree with your empirical observation: if the sole objective is the largest pure-ℓ2\ell_2 certified radius, the input Gaussian smoothing baseline in Table 8 is clearly stronger. We included this comparison precisely so that the paper would not imply otherwise.

The reason for studying PixelDP on ViTs is that it poses a different certification question. Input smoothing randomizes the raw input. PixelDP instead certifies a randomized internal representation, after which the downstream network is covered by post-processing. The scientific question is therefore whether this internal-representation construction remains valid and trainable when the sensitivity-controlled front end changes from a convolution to a ViT patch embedding. Input smoothing does not answer that question.

This distinction is also what motivates the ImageNet post-processing experiment: the certificate is attached to the noisy front end, while the downstream backbone can change without changing the underlying sensitivity guarantee. We do not claim that this structural flexibility translates into a larger certified radius than specialized input smoothing.

We therefore agree with the narrower statement that the Laplace accountant is existing theory and that PixelDP does not outperform input smoothing on pure-ℓ2\ell_2 radius. We do not, however, think these observations make the research question redundant. The contribution is the diagnosis of what changes when certification is moved to an internal ViT representation, the identification of the inherited grouped-bound conservatism, the exact-ℓ1\ell_1 repair that preserves the original certificate, and the remaining sensitivity-geometry effect revealed by the spectral intervention.

We hope this clarification resolves the apparent tension between Table 8 and the motivation of the paper, and more clearly separates the prior privacy accountant from the original analysis in the manuscript.

Reviewer_qMADOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits PixelDP-based certified robustness for ViT. The authors observe that applying the original grouped ℓ1\ell_1 sensitivity constraint to ViT patch embeddings results in severe accuracy degradation when using Laplace noise. They attribute this to excessive signal attenuation caused by conservative sensitivity bounds and provide a theoretical analysis to explain this phenomenon. To mitigate this, the authors propose spectral-norm constraints and leverage existing approximate differential privacy guarantees to certify the resulting Laplace models under ℓ2\ell_2 sensitivity. The authors evaluate their approach on several datasets, including CIFAR-10, CIFAR-100, SVHN, and ImageNet, and further explore the impact of adversarial training on certified accuracy.

Critical strengths
  1. The paper is generally well-written and clearly organized, making it easy to follow the main ideas.
  2. The research problem is well-motivated.
Critical weaknesses
  1. Limited theoretical novelty. The dimension-free Laplace guarantee under ℓ2\ell_2 sensitivity in Theorem 3.1 largely follows the existing result in [1*], which the authors have already cited. It remains unclear what the main theoretical contribution is beyond adapting this guarantee to PixelDP. I encourage the authors to better distinguish their original contributions from existing theoretical results and clarify what new insights the analysis provides.

  2. The significance of the identified failure mode is unclear. The reported accuracy collapse appears to be largely caused by an overly conservative sensitivity bound. As shown in Table 5, simply replacing the inherited grouped ℓ1\ell_1 bound with the exact stride-aware bound already improves accuracy from 28.87% to 70.83%. It would be helpful to extend this comparison to other datasets and noise levels, and report the corresponding certified accuracy under the appropriate threat models. This would help establish how much additional benefit the proposed spectral repair actually provides.

  3. Insufficient discussion of closely related work. Prior studies, such as [2*] and [3*], have already explored certified robustness through intermediate-layer noise injection and Lipschitz-constrained representations. These works appear closely related to the overall framework adopted in this paper. I suggest that the authors discuss these studies in more detail, clarify the methodological differences, and explain what new challenges or insights arise specifically from extending PixelDP to ViTs.

  4. Limited experimental validation and practical advantages. The main ViT experiments are restricted to small models and low-resolution images, while the ImageNet experiments use a convolutional autoencoder as the noise-injection front end rather than the ViT patch embedding. Moreover, Table 6 shows substantially lower certified accuracy than standard Gaussian randomized smoothing. I encourage the authors to evaluate the proposed repair on larger ViTs, report results across multiple random seeds, and better demonstrate scenarios where the proposed approach offers practical advantages over existing certified defenses.

References

[1*] Joel Daniel Andersson, Rasmus Pagh, Teresa Anna Steiner, and Sahel Torkamani. Count on Your Elders: Laplace vs Gaussian Noise. FORC, 2025.

[2*] Meiyu Zhong and Ravi Tandon. SPLITZ: Certifiable Robustness via Split Lipschitz Randomized Smoothing. IEEE TIFS, 2025.

[3*] Huimin Zeng, Jiahao Su, and Furong Huang. Certified Defense via Latent Space Randomized Smoothing with Orthogonal Encoders. arXiv:2108.00491, 2021.

Questions for authors
  1. Could the authors clarify the key technical contributions beyond the existing ℓ2\ell_2-sensitive Laplace guarantee in [1*] and prior latent-space certification methods such as [2*] and [3*]? In particular, which aspects of the proposed framework are fundamentally new rather than adaptations of existing results?

  2. Table 5 shows that using the exact stride-aware ℓ1\ell_1 bound already substantially improves accuracy. Does the identified failure remain significant when this tighter bound is adopted? How does this alternative compare with the proposed spectral repair in terms of certified robustness under their respective threat models?

  3. The paper claims that repaired Laplace models can be re-certified without retraining. However, the new calibration in Eq. (6) requires a different noise scale. Could the authors clarify whether the noise distribution is changed during inference or only the privacy accounting is updated? How are the training, inference, and certification settings kept consistent?

References

[1*] Joel Daniel Andersson, Rasmus Pagh, Teresa Anna Steiner, and Sahel Torkamani. Count on Your Elders: Laplace vs Gaussian Noise. FORC, 2025.

[2*] Meiyu Zhong and Ravi Tandon. SPLITZ: Certifiable Robustness via Split Lipschitz Randomized Smoothing. IEEE TIFS, 2025.

[3*] Huimin Zeng, Jiahao Su, and Furong Huang. Certified Defense via Latent Space Randomized Smoothing with Orthogonal Encoders. arXiv:2108.00491, 2021.

4 comments

Dear Reviewer qMAD,

Thank you again for your careful review and constructive suggestions. We have substantially revised the manuscript and completed the additional experiments requested in the discussion. We would like to summarize the changes.

1. Full evaluation of the exact stride-aware ℓ1\ell_1 alternative.

Following your suggestion, we extended the constraint-only comparison to CIFAR-10, CIFAR-100, and SVHN over the complete construction-scale grid L∈{0.03,0.1,0.3,1.0}L\in\{0.03,0.1,0.3,1.0\}.

At L=0.03L=0.03, the grouped ℓ1\ell_1, exact stride-aware ℓ1\ell_1, and spectral ℓ2\ell_2 constructions achieve:

  • CIFAR-10: 28.87%, 70.83%, and 85.50%.
  • CIFAR-100: 7.51%, 44.36%, and 59.92%.
  • SVHN: 19.59%, 88.92%, and 96.36%.

The complete sweep is now reported in Table 5 (Appendix D.4). These results confirm that the inherited grouped bound accounts for a substantial fraction of the original degradation, while a further trainability gap remains between exact ℓ1\ell_1 and spectral ℓ2\ell_2 control.

2. Multi-seed validation and larger ViT.

We repeated the key CIFAR-10 experiment at L=b=0.03L=b=0.03 using three independent training seeds. The grouped ℓ1\ell_1, exact stride-aware ℓ1\ell_1, and spectral ℓ2\ell_2 constructions achieve:

27.91±0.03%,69.65±1.36%,86.59±1.40%.27.91\pm0.03\%,\quad 69.65\pm1.36\%,\quad 86.59\pm1.40\%.

We also evaluated a larger ViT-Small backbone under the same constraint-only setting. The corresponding accuracies are 24.53%, 57.03%, and 84.57%, respectively. The ViT-Small result is a single-seed control, whereas the ViT-Tiny result above uses three seeds. Both are reported explicitly in Table 6.

3. Full exact-ℓ1\ell_1 certification.

We certified all 12 exact-ℓ1\ell_1 checkpoints across the three datasets and four construction scales under their native pure-DP guarantee, ϵ(T)=T/L\epsilon(T)=T/L and δ=0\delta=0.

For example, at L=0.1L=0.1, the certified accuracies at T=0.03,0.05,0.10T=0.03,0.05,0.10 are:

  • CIFAR-10: 46.29%, 41.52%, 29.59%.
  • CIFAR-100: 20.87%, 17.96%, 11.60%.
  • SVHN: 73.60%, 67.72%, 49.75%.

The complete results are provided in Table 7. The CIFAR-10 L=0.03L=0.03 certification additionally reports three-seed mean and standard deviation; the other grid points use their corresponding full-sweep checkpoints.

4. Theoretical positioning and related work.

We agree that the ℓ2\ell_2-sensitive Laplace guarantee in Theorem 3.1 is not an independent theoretical result. We now explicitly attribute it to Andersson et al. and present it as a specialization to the PixelDP construction. We also distinguish fixed-noise re-accounting from recalibration of the inference-time noise scale.

We expanded Related Work to discuss SPLITZ, latent-space randomized smoothing, Smoothed ViTs, diffusion-denoised smoothing, Yang et al., and the Heterogeneous Gaussian Mechanism. We also clarify that our method is not intended to outperform specialized input-space Gaussian smoothing in certified radius. Instead, we investigate the sensitivity and trainability consequences of placing randomized noise at an internal representation.

These changes have led us to revise the central interpretation of the paper: exact stride-aware ℓ1\ell_1 is a first-class repair that preserves the original pure-DP guarantee, while spectral ℓ2\ell_2 control provides a further trainability improvement under a different sensitivity geometry.

We hope that the additional experiments and the substantially revised presentation address the main concerns raised in your review. If so, we would sincerely appreciate your reconsideration of the current assessment and rating.

Thank you again for helping us strengthen the work.

Yours Sincerely,

Submission 17899 Team

Thanks for the additional results and clarifications. While these strengthen the empirical evaluation, my main concerns regarding the methodological novelty and the practical advantages in certified robustness remain. I will therefore maintain my score.

Reviewer qMAD,

Thank you for the follow-up and for considering the additional evidence.

We understand that your remaining concerns are primarily about methodological novelty and the practical advantage in certified robustness. We have revised the manuscript accordingly to make the scope more precise: we do not position the work as a new state-of-the-art certification method, nor do we treat the existing ℓ2\ell_2-sensitive Laplace guarantee as independent theoretical novelty.

The revised contribution is centered on diagnosing the architecture-specific sensitivity mismatch that arises when PixelDP is transferred to ViTs, disentangling the conservatism of the inherited grouped bound from the remaining sensitivity-geometry effect, and characterizing the certification consequences of the corresponding repairs.

Thank you again for the detailed feedback. It has helped us substantially sharpen both the claims and the empirical presentation of the paper.

Best Regards!

Yours Sincereley,

Submission17899 Authors

Reviewer qMAD,

Thank you again for the careful review and for reconsidering the paper after the additional experiments.

We appreciate your acknowledgement that the new results strengthen the empirical evaluation. We also understand that your remaining concerns concern methodological novelty and practical advantage in certified robustness.

The final manuscript has been revised to make our position on both points precise. We do not present the existing ℓ2\ell_2-sensitive Laplace guarantee as an independent theoretical contribution, nor do we claim that PixelDP outperforms specialized input randomized smoothing in pure-ℓ2\ell_2 certified radius.

At the same time, we believe the revised paper establishes a distinct contribution. The central question is what happens when PixelDP-style randomization is moved from a convolutional setting to a ViT internal representation. The analysis identifies the attenuation induced by the inherited grouped constraint, the exact stride-aware ℓ1\ell_1 intervention shows that a substantial part of the degradation comes from bound conservatism while retaining the original certificate, and the spectral intervention exposes a remaining sensitivity-geometry effect. These conclusions are now supported by the full three-dataset sweep, exact-ℓ1\ell_1 certification, three-seed validation, and a ViT-Small control.

We therefore do not position the work as a new state-of-the-art certification method. Its contribution is instead a mechanism-level analysis of how architecture and sensitivity geometry interact when an existing certified-defense principle is transferred to ViTs.

We understand that the significance of this contribution may ultimately be judged differently, but we believe the final manuscript now states its scope and evidence accurately. Thank you again for the constructive feedback, which materially improved the paper.

Submission17899 Authors

Comments on the paper

1 comment

We thank all reviewers for the careful reading, technical comments, and constructive discussion during the rebuttal period. The feedback led us not only to add experiments, but also to substantially sharpen the scope and interpretation of the paper. We have incorporated these changes into the revised manuscript.

The main revisions are as follows:

1. We made the exact stride-aware ℓ1\ell_1 constraint a first-class repair rather than a secondary ablation. We completed the full comparison of inherited grouped ℓ1\ell_1, exact stride-aware ℓ1\ell_1, and spectral ℓ2\ell_2 constraints across CIFAR-10, CIFAR-100, and SVHN over all four construction scales. We also added the corresponding pure-DP/ℓ1\ell_1 certified-accuracy results.

2. We strengthened the empirical validation. The key CIFAR-10 constraint comparison is now repeated over three independent training seeds, and we added a ViT-Small experiment as a larger-backbone control. These experiments confirm that the inherited grouped bound explains a substantial part of the original degradation, while a further trainability gap remains after replacing it with the exact stride-aware norm.

3. We revised the theoretical positioning. We verified that the closed-form ℓ2\ell_2-sensitive Laplace bound used in the spectral construction is algebraically identical to the corresponding existing guarantee of Andersson et al. The revised manuscript therefore does not present this privacy inequality as independent theorem-level novelty. Its role is now stated more precisely: it provides an accounting rule whose sensitivity geometry is compatible with the spectral repair.

4. We narrowed the interpretation of the norm-ratio analysis. The ratio ρgrp\rho_{\rm grp} is now treated as an attenuation diagnostic, particularly near initialization, rather than as a universal predictor or threshold for optimization collapse. The revised discussion separates conservatism of the inherited grouped bound from more general high-noise and optimization effects.

5. We clarified certification and mechanism consistency. The manuscript now explicitly distinguishes fixed-mechanism re-accounting from recalibration of the inference-time noise scale. We also added a numerical privacy-loss diagnostic to study the conservatism of the closed-form accountant, while clearly separating this diagnostic from rigorous worst-case certification.

6. We expanded the related-work discussion and clarified the scope of the contribution. The revision now discusses closely related intermediate-representation certification, certified ViT methods, randomized smoothing, and prior PixelDP extensions in substantially greater detail. We also make clear that the goal is not to claim state-of-the-art certified radii relative to specialized input randomized smoothing, but to understand the sensitivity, trainability, and certification consequences of transferring a PixelDP-style randomized representation to ViTs.

7. We improved the presentation and reproducibility of the method. The revised manuscript explicitly states where the noise is injected, defines the randomized representation and sensitivity constraints, distinguishes the grouped, exact stride-aware, and spectral norms, and reorganizes the experimental baselines and certification protocols to make the comparisons easier to follow and reproduce.

Taken together, these changes also modify the central message of the paper. Rather than attributing the observed behavior to a single “collapse” mechanism, the revised manuscript separates two effects: conservatism of the inherited grouped sensitivity bound and a remaining sensitivity-geometry effect. Exact stride-aware ℓ1\ell_1 control addresses the former while retaining the original pure-DP certificate; spectral ℓ2\ell_2 control provides a further trainability improvement under a different certification geometry.

We are grateful that the reviewers' comments helped us identify where the original manuscript overstated some claims, under-emphasized important alternatives, or did not explain the experimental protocol clearly enough. We believe the revised manuscript is substantially more precise, complete, and better supported as a result.

We understand that reviewers may still differ in their assessment of the methodological novelty and practical significance of the resulting contribution. We therefore leave the final evaluation to the reviewers and the Area Chair, and sincerely appreciate their time and consideration of the revised manuscript and the additional evidence provided during the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.