- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Gradual Adversarial Transition Alignment (GATA) to improve zero-shot adversarial robustness in vision-language models. Motivated by the observation that adversarial attack trajectories exhibit locally fluctuating updates but relatively stable cumulative displacement directions, GATA constructs an endpoint-induced interpolated surrogate path and minimizes KL divergence between adjacent predictive distributions. The method reports modest robustness improvements over endpoint alignment with negligible additional computational overhead.
Critical strengths
-
The paper addresses the important problem of improving adversarial robustness while preserving the zero-shot generalization of pretrained VLMs. The proposed method avoids additional encoder passes for real intermediate adversarial states, achieving low computational efficiency.
-
The paper reports extensive empirical evaluations across multiple VLM backbones, downstream datasets and adversarial attacks including comparisons with endpoint supervision and real-trajectory alignment.
Critical weaknesses
-
Insufficient justification for transition-level supervision / Insufficient geometric justification for the surrogate path: The paper argues that endpoint supervision leaves intermediate semantic transitions unconstrained, but does not establish why this degrades adversarial robustness. The lack of empirical or causal evidence linking unconstrained intermediate transitions to robustness failures weakens the motivation for transition-level alignment. Also, stable cumulative displacement directions do not necessarily imply that a linear interpolation between endpoints accurately approximates real adversarial trajectories. Moreover, the trajectory analysis uses a frozen encoder, whereas GATA constructs its surrogate path using representations from frozen and fine-tuned encoders. Direct approximation-error measurements or theoretical bounds are missing, leaving it unclear whether the surrogate preserves robustness-relevant trajectory geometry across different attack configurations.
-
Limited methodological novelty and marginal improvements: GATA primarily combines endpoint-based representation interpolation with existing KL-based consistency regularization. The improvements over endpoint KL are modest, and the experiments do not adequately distinguish the effects of gradual alignment from changes in effective loss and gradient scales. This raises questions about the methodological novelty of the proposed approach. These concerns could be alleviated by establishing the limitations of endpoint supervision, demonstrating a causal relationship between intermediate semantic drift and robustness failures, and deriving a sufficiently tight approximation error bound that supports the surrogate geometry.
Questions for authors
-
What empirical evidence demonstrates that endpoint-only supervision causes robustness-critical failures at intermediate attack states? Can the authors establish that controlling these states provides benefits beyond conventional endpoint regularization?
-
How do the authors establish that directional fluctuations in feature space correspond to semantic information degradation? Is there a measurable relationship between intermediate representation drift, classification-margin degradation, and actual robustness failures?
-
For CLIP's linear image-text similarity classifier, normalized positive interpolation between two endpoints predicting the same class preserves that predicted class along the entire path. Under this condition, what additional robustness-relevant failure does intermediate KL alignment address beyond changes in predictive confidence or distribution?
-
How accurately does the endpoint-induced virtual path approximate actual adversarial trajectories across different attacks, including stochastic or non-gradual perturbations? Can the authors quantify the representation or prediction discrepancy and derive an mathematical upper bound under explicit assumptions? How does this approximation error relate to robustness improvements?
-
Does GATA consistently outperform a properly tuned endpoint KL baseline after controlling for effective loss/gradient scale, regularization coefficients, and random seeds?
8 comments
Response to Weaknesses 2 (a)
Limited methodological novelty and marginal improvements: GATA primarily combines endpoint-based representation interpolation with existing KL-based consistency regularization. The improvements over endpoint KL are modest, and the experiments do not adequately distinguish the effects of gradual alignment from changes in effective loss and gradient scales. This raises questions about the methodological novelty of the proposed approach. These concerns could be alleviated by establishing the limitations of endpoint supervision, demonstrating a causal relationship between intermediate semantic drift and robustness failures, and deriving a sufficiently tight approximation error bound that supports the surrogate geometry.
Thank you for these comments. We address the concerns in four parts.
(1) Contribution: trajectory-informed gradual supervision.
-
Our goal is not to introduce a new form of KL divergence. We ask whether the structure of real attack trajectories can guide low-cost transition supervision for robust fine-tuning.
-
Our analysis reveals substantial local directional fluctuations but relatively stable cumulative displacement directions across the five attacks and settings tested. This motivates an endpoint-induced surrogate path with consistency constraints between adjacent predictive states.
-
Our contribution connects trajectory analysis, surrogate construction, and gradual predictive supervision. Interpolation and KL are implementation tools; the trajectory analysis provides an empirical basis for their use.
(2) Limitations of endpoint supervision.
-
The ablation results (Tables 3, 7, 15, and 18) show that our method improves both clean and robust accuracy over Endpoint KL across different perturbation budgets.
-
Figure 6 provides empirical support: compared with Endpoint KL, GATA maintains higher image–correct-label similarity and classification accuracy along the constructed clean-to-adversarial path. This suggests that gradual alignment helps preserve semantic information during the transition.
-
Our method’s advantage over Endpoint KL becomes more pronounced as attack strength increases. We added experiments using 10-step PGD fine-tuning at (\epsilon=4/255) and evaluation at (\epsilon=1/255) . The results show that our method significantly improved the clean accuracy and robust accuracy of Endpoint KL in this experiment. | | Method | TinyImageNet | CIFAR10 | CIFAR100 | STL10 | SUN397 | Food101 | OxfordPets | Flowers102 | DTD | EuroSAT | FGVC Aircraft | ImageNet | Caltech101 | Caltech256 | StanfordCars | PCAM | Average | H | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Clean | Endpoint KL | 67.49 | 76.24 | 47.73 | 88.77 | 34.00 | 41.28 | 62.80 | 35.18 | 26.54 | 12.49 | 7.80 | 34.05 | 76.14 | 63.41 | 26.71 | 50.01 | 46.92 | — | | Clean | GATA | 66.52 | 74.79 | 47.33 | 88.56 | 33.53 | 43.67 | 64.51 | 38.05 | 28.24 | 17.54 | 9.21 | 35.42 | 77.84 | 65.56 | 30.06 | 50.01 | 48.18 | — | | PGD | Endpoint KL | 50.73 | 50.82 | 29.35 | 74.86 | 20.54 | 22.78 | 42.52 | 23.69 | 18.72 | 4.88 | 3.69 | 21.51 | 64.32 | 48.73 | 13.68 | 47.92 | 33.67 | 39.20 | | PGD | GATA | 48.75 | 48.93 | 28.57 | 75.09 | 20.08 | 23.41 | 43.88 | 25.08 | 19.31 | 7.91 | 4.26 | 21.87 | 65.43 | 50.19 | 14.67 | 49.62 | 34.19 | 40.00 | | AutoAttack | Endpoint KL | 48.74 | 49.33 | 27.13 | 74.33 | 18.27 | 20.65 | 41.21 | 21.97 | 16.86 | 4.30 | 2.91 | 19.48 | 62.76 | 46.83 | 11.48 | 47.41 | 32.10 | 38.12 | | AutoAttack | GATA | 47.18 | 47.21 | 26.42 | 74.50 | 18.15 | 21.00 | 42.27 | 22.75 | 17.23 | 6.21 | 3.60 | 20.02 | 63.74 | 48.51 | 12.70 | 49.47 | 32.56 | 38.86 | | CW | Endpoint KL | 49.08 | 49.80 | 27.26 | 74.25 | 18.33 | 20.69 | 41.95 | 22.08 | 16.33 | 4.24 | 2.82 | 19.73 | 62.82 | 46.95 | 12.19 | 47.32 | 32.24 | 38.22 | | CW | GATA | 47.54 | 47.67 | 26.76 | 74.52 | 18.33 | 21.26 | 42.93 | 22.75 | 16.97 | 6.25 | 3.78 | 20.25 | 63.90 | 48.74 | 13.87 | 49.46 | 32.81 | 39.04 |
-
These gains come with little additional cost. Table 5 shows FLOPs and peak memory nearly identical to Endpoint KL.
Response to Weaknesses 2 (b)
Limited methodological novelty and marginal improvements: GATA primarily combines endpoint-based representation interpolation with existing KL-based consistency regularization. The improvements over endpoint KL are modest, and the experiments do not adequately distinguish the effects of gradual alignment from changes in effective loss and gradient scales. This raises questions about the methodological novelty of the proposed approach. These concerns could be alleviated by establishing the limitations of endpoint supervision, demonstrating a causal relationship between intermediate semantic drift and robustness failures, and deriving a sufficiently tight approximation error bound that supports the surrogate geometry.
(3) Loss scale control.
We agree that varying the number of intermediate states can also change the scale of the summed KL objective. To examine whether the gains mainly arise from this change, we conducted additional experiments using:
Under a local quadratic approximation, when adjacent predictive distributions are sufficiently close, the summed KL decreases approximately as (1/K) as the path is subdivided. Multiplication by (K) compensates for this leading-order scale change. The supplementary results are as follows:
\begin{array}{c|c|c|c|c} \hline n & \text{PGD} & \text{CW} & \text{AutoAttack} & \text{Clean} \\\\ \begin{array}{c} \vphantom{H} \\\\ \hline 0 \\\\ 1 \\\\ 3 \\\\ 5 \\\\ 7 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 31.94 & 39.47 \\\\ 32.05 & 39.47 \\\\ 32.01 & 39.43 \\\\ \mathbf{32.13} & \mathbf{40.13} \\\\ 32.00 & 39.45 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 30.41 & 38.28 \\\\ 30.69 & 38.43 \\\\ 30.69 & 38.41 \\\\ \mathbf{30.74} & \mathbf{39.03} \\\\ 30.59 & 38.35 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 29.97 & 37.93 \\\\ 30.23 & 38.06 \\\\ 30.29 & 38.10 \\\\ \mathbf{30.71} & \mathbf{39.00} \\\\ 30.14 & 38.00 \end{array} & \begin{array}{c} \text{Avg.} \\\\ \hline 51.63 \\\\ 51.37 \\\\ 51.34 \\\\ \mathbf{53.44} \\\\ 51.41 \end{array} \\\\ \hline \end{array} \end{gathered}$$ The following are the results shown in Table 3 of the paper (without loss scale control). $$\begin{gathered} \begin{array}{c|c|c|c|c} \hline n & \text{PGD} & \text{CW} & \text{AutoAttack} & \text{Clean} \\\\ \begin{array}{c} \vphantom{H} \\\\ \hline 0 \\\\ 1 \\\\ 3 \\\\ 5 \\\\ 7 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 31.94 & 39.47 \\\\ 32.24 & 39.98 \\\\ \mathbf{32.30} & \mathbf{40.21} \\\\ 31.94 & 39.98 \\\\ 31.66 & 39.76 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 30.41 & 38.28 \\\\ \mathbf{31.07} & 39.07 \\\\ \mathbf{31.07} & \mathbf{39.25} \\\\ 30.71 & 39.01 \\\\ 30.38 & 38.74 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 29.97 & 37.93 \\\\ \mathbf{30.46} & 38.59 \\\\ \mathbf{30.46} & \mathbf{38.75} \\\\ 30.06 & 38.47 \\\\ 29.71 & 38.19 \end{array} & \begin{array}{c} \text{Avg.} \\\\ \hline 51.63 \\\\ 52.64 \\\\ 53.26 \\\\ 53.43 \\\\ \mathbf{53.44} \end{array} \\\\ \hline \end{array} \end{gathered}$$ Taken together, the results in the two tables show that: - **Robustness gains persist after scale compensation.** With this compensation, all settings with intermediate states still achieve higher average robust accuracy than the endpoint baseline under all three attacks. - **Controlling the loss scale does not materially change the results.** The best results in the two tables are comparable, suggesting that the loss scale has little effect on performance. (4) **Approximation-error bounds.** - **Accurate trajectory reconstruction is not our objective.** We aim to construct effective transition supervision at low cost, rather than minimize geometric distance to the real trajectory. - **Directly using real trajectory states does not improve performance in our experiments.** PGD-Traj uses actual intermediate attack states instead of interpolated substitutes. Nevertheless, Table 4 reports lower average robust accuracy than GATA, while Table 5 shows higher computational and memory costs. **Thus, retaining real trajectory states does not necessarily yield better training outcomes.** - Existing results show no clear robustness benefit from more accurate trajectory fitting (Tables 4, 5). - Fitting accuracy lacks a standardized evaluation framework. Its scale varies across backbones and datasets, and no established framework links the degree of fitting accuracy to robustness. **Therefore, a tight approximation error bound is not aligned with our method’s objective and does not directly guarantee robustness.**
Response to Weaknesses 1 and Questions 1
Thank you for these detailed comments. We address each concern below.
(1) Relationship between intermediate semantic drift and robustness failures.
The gains of our method (constrains intermediate semantic transitions) over endpoint KL(unconstrained) provide evidence of a link between intermediate semantic drift and robustness failures.
Intermediate semantic transitions progressively undermine robustness: iterative attacks gradually induce these shifts until misclassification occurs. Endpoint supervision constrains only a single point along the path, whereas robustness degradation unfolds along the entire trajectory.
Figure 6 provides empirical support: compared with Endpoint KL, GATA maintains higher image–correct-label similarity and classification accuracy along the constructed clean-to-adversarial path. This suggests that gradual alignment helps preserve semantic information during the transition.
The ablations across backbones in Tables 7, 15, 18 provide further evidence of its effectiveness. Constraining the path improves model robustness compared with leaving it unconstrained.
Stronger attacks lead to greater representation drift (semantic transitions). We measure the shift between clean and adversarial representations across increasing perturbation budgets using three metrics. The results are as follow | Dataset | eps | Mean L2 | Normalized L2 | Cosine distance | |---|---|---:|---:|---:| | TinyImageNet | 1/255 | 7.76 | 0.70 | 0.25 | | | 2/255 | 9.02 | 0.84 | 0.36 | | | 4/255 | 9.85 | 0.93 | 0.44 | | | 6/255 | 10.13 | 0.96 | 0.47 | | | 8/255 | 10.28 | 0.98 | 0.49 | | ImageNet | 1/255 | 5.61 | 0.54 | 0.15 | | | 2/255 | 7.53 | 0.73 | 0.28 | | | 4/255 | 8.63 | 0.85 | 0.37 | | | 6/255 | 8.95 | 0.89 | 0.40 | | | 8/255 | 9.11 | 0.91 | 0.42 |
Our experiments show that constraining semantic transitions yields greater robustness gains as attack strength increases (Tables 1, 8, 9, 13, and 14, and Figure 10). Together with the drift analysis above, these results suggest that leaving semantic drift unconstrained leads to greater robustness degradation as attack strength increases.
(2) Stable cumulative directions do not guarantee that endpoint interpolation accurately approximates real trajectories.
We agree. Directional consistency motivates the surrogate construction but does not prove pointwise approximation accuracy.
Our goal is to retain useful overall transition information for supervision, rather than reconstruct every attack update. GATA therefore constructs a low-cost surrogate and aligns adjacent predictive states.
Tables 4, 5 evaluate this choice through performance and efficiency. Under the tested settings, GATA achieves higher average robust accuracy than PGD-Traj, which directly uses real intermediate attack states, while requiring substantially less computation and memory. This supports the surrogate’s value for training.
(3) The original analysis uses a frozen encoder, whereas GATA connects frozen and fine-tuned representations.
We added trajectory analysis during fine-tuning at epochs 3, 6, and 9. Clean reference representations come from the frozen original CLIP, while intermediate and final adversarial representations come from the current fine-tuned model.
Figure 7 shows that cumulative displacements remain directionally consistent under this setting, which matches the representation sources used by GATA.
(4) Approximation-error measurements or theoretical bounds are missing.
Accurate trajectory reconstruction is not our objective. We aim to construct effective transition supervision at low cost, rather than minimize geometric distance to the real trajectory.
Directly using real trajectory states does not improve performance in our experiments. PGD-Traj uses actual intermediate attack states instead of interpolated substitutes. Nevertheless, Table 4 reports lower average robust accuracy than GATA, while Table 5 shows higher computational and memory costs. Thus, retaining real trajectory states does not necessarily yield better training outcomes.
Existing results show no clear robustness benefit from more accurate trajectory fitting (Tables 4, 5).
Fitting accuracy lacks a standardized evaluation framework. Its scale varies across backbones and datasets, and no established framework links the degree of fitting accuracy to robustness.
Therefore, a tight approximation error bound is not aligned with our method’s objective and does not directly guarantee robustness.