- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a test-time adversarial defense method based on early residual curvature. The authors find experimentally that adversarial perturbations reduce the curvature of early residual trajectories in ViT and amplify representation discrepancies through deep MLP branches. CURB uses a gating mechanism and gradient-based input pixel updates to increase early residual curvature for abnormal samples, thereby curbing adversarial propagation. Experiments show that the method effectively improves adversarial robustness on image captioning, visual question answering, and zero-shot classification tasks while largely preserving clean performance.
Critical strengths
- By comparing the clean–adversarial representation discrepancies across different layers of ViT-based visual encoders, the authors observe that the representation discrepancies caused by adversarial perturbations gradually increase toward deeper layers. The authors also find that the early residual curvature of adversarial samples tends to be lower than that of clean samples, and attempt to explain the connection between this phenomenon and deep-layer representation discrepancy growth through an analysis of the common component and differential component. The overall research motivation is clear and has some value for mechanistic analysis.
- The authors propose CURB, a test-time defense method that does not require retraining the model. It increases early residual curvature by optimizing the input image to alleviate representation discrepancies caused by adversarial perturbations. The method is independent of the downstream output space and introduces selective activation to avoid unnecessary modifications to all inputs.
Critical weaknesses
- The current experiments are mainly based on statistical analysis of curvature distributions and do not explicitly report the proportions of successful adversarial examples whose curvature decreases or increases. Therefore, I am uncertain whether the phenomenon proposed in the paper is general.
- The gating threshold proposed in the paper should be a very important hyperparameter. However, the paper only states that the 5th percentile of the clean curvature distribution is used as the gating threshold, without providing a detailed explanation or discussing robustness to the choice of this gating threshold.
Questions for authors
- Among successful adversarial examples, what proportions exhibit a decrease and an increase in early residual curvature, respectively? Furthermore, how effective is CURB on successful adversarial examples whose curvature does not decrease or that do not trigger the CURB gating mechanism?
- What is the specific rationale for choosing the 5th percentile rather than other percentiles as the gating threshold? When the threshold is set to the 1st, 5th, or 10th percentile, how do clean performance, adversarial robustness, and the gating activation rate change?
4 comments
W2: Sensitivity to the gating threshold
Thanks for your comments. We evaluated LLaVA on the same APGD-500 pairs, using one cross-dataset threshold defined by the 1st, 5th, or 10th percentile of the clean-curvature distribution.
| Percentile | Clean gate | Adv. gate | COCO C/A | Flickr30k C/A | TextVQA C/A | VQAv2 C/A |
|---|---|---|---|---|---|---|
| 1st | 1.0% | 15.1% | 115.5 / 23.3 | 77.8 / 18.1 | 37.2 / 6.7 | 76.1 / 12.0 |
| 5th | 5.0% | 88.9% | 115.4 / 116.0 | 77.2 / 73.9 | 36.9 / 28.0 | 74.5 / 65.3 |
| 10th | 10.0% | 91.0% | 115.2 / 116.3 | 76.9 / 74.5 | 36.7 / 29.4 | 74.2 / 65.7 |
Here, C/A denotes clean/adversarial CIDEr for captioning and accuracy for VQA; gate rates are pooled across the four datasets. The 1st percentile is overly conservative and misses most adversarial inputs. Increasing the threshold from the 5th to the 10th percentile doubles clean activation, while adversarial robustness improves only marginally.
Notably, this sensitivity analysis shows that activating CURB for the small subset of low-curvature clean inputs has only a marginal effect on clean performance, consistent with our finding in Sec. 4.4, Ablation of Gating Threshold and Optimization Direction, that gate activation on clean inputs does not necessarily lead to clean performance degradation. This is why a lower percentile of the clean distribution can serve as the gating threshold. And we will add this sensitivity analysis to the Sec. 4.4
Dear Reviewer JaXM,
We would be grateful if you could review our rebuttal at your convenience. We have provided additional experiments and revised the manuscript accordingly, with the changes highlighted in blue. we would be grateful if you could let us know whether our response and revisions adequately address your concerns, and we are happy to provide additional clarification for further questions.
Thank you for your time and consideration.
Summary of Concerns Addressed and Request for Reassessment
Dear reviewer,
Thank you again for your thoughtful review. As the discussion period closes soon, we would appreciate your feedback on whether our additional response resolve your concerns:
-
W1: Our paired analysis shows decreased curvature in 3,369/3,382 (99.6%) successfully attacked samples. Among the 13 exceptions, 12 still activate the gate and all 12 obtain improved task performance, directly quantifying both prevalence and exceptional cases.
-
W2: Our sensitivity analysis over the 1st, 5th, and 10th percentiles shows that the 1st percentile misses most adversarial inputs, whereas increasing from the 5th to the 10th percentile doubles clean activation with only marginal robustness activation. This supports the 5th percentile as the best trade-off.
Both analyses have been incorporated into the revised manuscript. Could you please let us know whether these additions resolve your concerns? If so, we would greatly appreciate your reconsideration of the assessment and score.
W1: Prevalence of decreased curvature among successful attacks
Thanks for your comments. We conducted a paired analysis on the same 500 APGD examples per dataset used in Table 1. For every successfully attacked pair, we compute ΔC=C(xadv)−C(xclean). An attack is considered successful when its per-sample task score is lower than that of the corresponding clean input.
For Q1.1: Overall, curvature decreases for 1,594/1,594 (100%) successful attacks on LLaVA and 1,775/1,788 (99.3%) on InternVL. Thus, the observation is not merely distributional, it holds for 3,369/3,382 (99.6%) successfully attacked clean–adversarial pairs.
For Q1.2: Importantly, ΔC>0 does not necessarily prevent CURB from working, because the gate depends on the adversarial input’s absolute curvature, rather than its change relative to the unavailable clean counterpart. Among the 13 InternVL exceptions with ΔC>0, 12 still activate the gate and all 12 obtain improved task scores:
This analysis clarifies that the paired curvature observation is highly prevalent, while the remaining robustness gap is primarily attributable to the conservative gate rather than violations of the curvature observation. We will add these paired statistics and the exception analysis to the revised appendix.