acceptodds
Under review as a conference paper at ICLR 2027

Gradual Adversarial Transition Alignment for Robust Vision-Language Models

Abstract

Vision-language models such as CLIP exhibit strong zero-shot generalization but remain highly vulnerable to adversarial perturbations. Most existing adversarial fine-tuning methods primarily constrain the clean/reference state and the final adversarial state, leaving the progressive semantic drift between them unsupervised. A natural remedy is to supervise the intermediate states of iterative attacks, but this requires an additional forward–backward pass for every state, and real trajectories carry attack-specific local fluctuations. We therefore investigate whether attack trajectories exhibit a compact geometric structure that can be efficiently approximated. Our analysis reveals that, despite substantial local fluctuations, the cumulative representation displacement follows a stable dominant direction across attacks and perturbation settings. This finding motivates an endpoint-induced surrogate path that approximates the dominant clean-to-adversarial transition using only the two endpoints. Inspired by gradual domain adaptation, we discretize this path into a sequence of ordered intermediate states and enforce KL-based prediction consistency between adjacent states. The resulting Gradual Adversarial Transition Alignment (GATA) replaces a coarse endpoint constraint with a sequence of locally smaller predictive transitions, providing efficient transition-level supervision without explicitly modeling the full attack trajectory. Extensive experiments across diverse attacks, perturbation strengths, downstream datasets, and vision-language backbones demonstrate improved adversarial robustness while preserving a favorable clean–robust trade-off.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial robustness, adversarial fine-tuning, attack trajectory, gradual alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sGyMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Gradual Adversarial Transition Alignment (GATA) to improve zero-shot adversarial robustness in vision-language models. Motivated by the observation that adversarial attack trajectories exhibit locally fluctuating updates but relatively stable cumulative displacement directions, GATA constructs an endpoint-induced interpolated surrogate path and minimizes KL divergence between adjacent predictive distributions. The method reports modest robustness improvements over endpoint alignment with negligible additional computational overhead.

Critical strengths
  1. The paper addresses the important problem of improving adversarial robustness while preserving the zero-shot generalization of pretrained VLMs. The proposed method avoids additional encoder passes for real intermediate adversarial states, achieving low computational efficiency.

  2. The paper reports extensive empirical evaluations across multiple VLM backbones, downstream datasets and adversarial attacks including comparisons with endpoint supervision and real-trajectory alignment.

Critical weaknesses
  1. Insufficient justification for transition-level supervision / Insufficient geometric justification for the surrogate path: The paper argues that endpoint supervision leaves intermediate semantic transitions unconstrained, but does not establish why this degrades adversarial robustness. The lack of empirical or causal evidence linking unconstrained intermediate transitions to robustness failures weakens the motivation for transition-level alignment. Also, stable cumulative displacement directions do not necessarily imply that a linear interpolation between endpoints accurately approximates real adversarial trajectories. Moreover, the trajectory analysis uses a frozen encoder, whereas GATA constructs its surrogate path using representations from frozen and fine-tuned encoders. Direct approximation-error measurements or theoretical bounds are missing, leaving it unclear whether the surrogate preserves robustness-relevant trajectory geometry across different attack configurations.

  2. Limited methodological novelty and marginal improvements: GATA primarily combines endpoint-based representation interpolation with existing KL-based consistency regularization. The improvements over endpoint KL are modest, and the experiments do not adequately distinguish the effects of gradual alignment from changes in effective loss and gradient scales. This raises questions about the methodological novelty of the proposed approach. These concerns could be alleviated by establishing the limitations of endpoint supervision, demonstrating a causal relationship between intermediate semantic drift and robustness failures, and deriving a sufficiently tight approximation error bound that supports the surrogate geometry.

Questions for authors
  1. What empirical evidence demonstrates that endpoint-only supervision causes robustness-critical failures at intermediate attack states? Can the authors establish that controlling these states provides benefits beyond conventional endpoint regularization?

  2. How do the authors establish that directional fluctuations in feature space correspond to semantic information degradation? Is there a measurable relationship between intermediate representation drift, classification-margin degradation, and actual robustness failures?

  3. For CLIP's linear image-text similarity classifier, normalized positive interpolation between two endpoints predicting the same class preserves that predicted class along the entire path. Under this condition, what additional robustness-relevant failure does intermediate KL alignment address beyond changes in predictive confidence or distribution?

  4. How accurately does the endpoint-induced virtual path approximate actual adversarial trajectories across different attacks, including stochastic or non-gradual perturbations? Can the authors quantify the representation or prediction discrepancy and derive an mathematical upper bound under explicit assumptions? How does this approximation error relate to robustness improvements?

  5. Does GATA consistently outperform a properly tuned endpoint KL baseline after controlling for effective loss/gradient scale, regularization coefficients, and random seeds?

8 comments
AuthorsOpenReview ↗

Response to Weaknesses 1 and Questions 1

Thank you for these detailed comments. We address each concern below.

(1) Relationship between intermediate semantic drift and robustness failures.

  • The gains of our method (constrains intermediate semantic transitions) over endpoint KL(unconstrained) provide evidence of a link between intermediate semantic drift and robustness failures.

  • Intermediate semantic transitions progressively undermine robustness: iterative attacks gradually induce these shifts until misclassification occurs. Endpoint supervision constrains only a single point along the path, whereas robustness degradation unfolds along the entire trajectory.

  • Figure 6 provides empirical support: compared with Endpoint KL, GATA maintains higher image–correct-label similarity and classification accuracy along the constructed clean-to-adversarial path. This suggests that gradual alignment helps preserve semantic information during the transition.

  • The ablations across backbones in Tables 7, 15, 18 provide further evidence of its effectiveness. Constraining the path improves model robustness compared with leaving it unconstrained.

  • Stronger attacks lead to greater representation drift (semantic transitions). We measure the shift between clean and adversarial representations across increasing perturbation budgets using three metrics. The results are as follow | Dataset | eps | Mean L2 | Normalized L2 | Cosine distance | |---|---|---:|---:|---:| | TinyImageNet | 1/255 | 7.76 | 0.70 | 0.25 | | | 2/255 | 9.02 | 0.84 | 0.36 | | | 4/255 | 9.85 | 0.93 | 0.44 | | | 6/255 | 10.13 | 0.96 | 0.47 | | | 8/255 | 10.28 | 0.98 | 0.49 | | ImageNet | 1/255 | 5.61 | 0.54 | 0.15 | | | 2/255 | 7.53 | 0.73 | 0.28 | | | 4/255 | 8.63 | 0.85 | 0.37 | | | 6/255 | 8.95 | 0.89 | 0.40 | | | 8/255 | 9.11 | 0.91 | 0.42 |

  • Our experiments show that constraining semantic transitions yields greater robustness gains as attack strength increases (Tables 1, 8, 9, 13, and 14, and Figure 10). Together with the drift analysis above, these results suggest that leaving semantic drift unconstrained leads to greater robustness degradation as attack strength increases.

(2) Stable cumulative directions do not guarantee that endpoint interpolation accurately approximates real trajectories.

  • We agree. Directional consistency motivates the surrogate construction but does not prove pointwise approximation accuracy.

  • Our goal is to retain useful overall transition information for supervision, rather than reconstruct every attack update. GATA therefore constructs a low-cost surrogate and aligns adjacent predictive states.

  • Tables 4, 5 evaluate this choice through performance and efficiency. Under the tested settings, GATA achieves higher average robust accuracy than PGD-Traj, which directly uses real intermediate attack states, while requiring substantially less computation and memory. This supports the surrogate’s value for training.

(3) The original analysis uses a frozen encoder, whereas GATA connects frozen and fine-tuned representations.

  • We added trajectory analysis during fine-tuning at epochs 3, 6, and 9. Clean reference representations come from the frozen original CLIP, while intermediate and final adversarial representations come from the current fine-tuned model.

  • Figure 7 shows that cumulative displacements remain directionally consistent under this setting, which matches the representation sources used by GATA.

(4) Approximation-error measurements or theoretical bounds are missing.

  • Accurate trajectory reconstruction is not our objective. We aim to construct effective transition supervision at low cost, rather than minimize geometric distance to the real trajectory.

  • Directly using real trajectory states does not improve performance in our experiments. PGD-Traj uses actual intermediate attack states instead of interpolated substitutes. Nevertheless, Table 4 reports lower average robust accuracy than GATA, while Table 5 shows higher computational and memory costs. Thus, retaining real trajectory states does not necessarily yield better training outcomes.

  • Existing results show no clear robustness benefit from more accurate trajectory fitting (Tables 4, 5).

  • Fitting accuracy lacks a standardized evaluation framework. Its scale varies across backbones and datasets, and no established framework links the degree of fitting accuracy to robustness.

Therefore, a tight approximation error bound is not aligned with our method’s objective and does not directly guarantee robustness.

AuthorsOpenReview ↗

Response to Weaknesses 2 (a)

Limited methodological novelty and marginal improvements: GATA primarily combines endpoint-based representation interpolation with existing KL-based consistency regularization. The improvements over endpoint KL are modest, and the experiments do not adequately distinguish the effects of gradual alignment from changes in effective loss and gradient scales. This raises questions about the methodological novelty of the proposed approach. These concerns could be alleviated by establishing the limitations of endpoint supervision, demonstrating a causal relationship between intermediate semantic drift and robustness failures, and deriving a sufficiently tight approximation error bound that supports the surrogate geometry.

Thank you for these comments. We address the concerns in four parts.

(1) Contribution: trajectory-informed gradual supervision.

  • Our goal is not to introduce a new form of KL divergence. We ask whether the structure of real attack trajectories can guide low-cost transition supervision for robust fine-tuning.

  • Our analysis reveals substantial local directional fluctuations but relatively stable cumulative displacement directions across the five attacks and settings tested. This motivates an endpoint-induced surrogate path with consistency constraints between adjacent predictive states.

  • Our contribution connects trajectory analysis, surrogate construction, and gradual predictive supervision. Interpolation and KL are implementation tools; the trajectory analysis provides an empirical basis for their use.

(2) Limitations of endpoint supervision.

  • The ablation results (Tables 3, 7, 15, and 18) show that our method improves both clean and robust accuracy over Endpoint KL across different perturbation budgets.

  • Figure 6 provides empirical support: compared with Endpoint KL, GATA maintains higher image–correct-label similarity and classification accuracy along the constructed clean-to-adversarial path. This suggests that gradual alignment helps preserve semantic information during the transition.

  • Our method’s advantage over Endpoint KL becomes more pronounced as attack strength increases. We added experiments using 10-step PGD fine-tuning at (\epsilon=4/255) and evaluation at (\epsilon=1/255) . The results show that our method significantly improved the clean accuracy and robust accuracy of Endpoint KL in this experiment. | | Method | TinyImageNet | CIFAR10 | CIFAR100 | STL10 | SUN397 | Food101 | OxfordPets | Flowers102 | DTD | EuroSAT | FGVC Aircraft | ImageNet | Caltech101 | Caltech256 | StanfordCars | PCAM | Average | H | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Clean | Endpoint KL | 67.49 | 76.24 | 47.73 | 88.77 | 34.00 | 41.28 | 62.80 | 35.18 | 26.54 | 12.49 | 7.80 | 34.05 | 76.14 | 63.41 | 26.71 | 50.01 | 46.92 | — | | Clean | GATA | 66.52 | 74.79 | 47.33 | 88.56 | 33.53 | 43.67 | 64.51 | 38.05 | 28.24 | 17.54 | 9.21 | 35.42 | 77.84 | 65.56 | 30.06 | 50.01 | 48.18 | — | | PGD | Endpoint KL | 50.73 | 50.82 | 29.35 | 74.86 | 20.54 | 22.78 | 42.52 | 23.69 | 18.72 | 4.88 | 3.69 | 21.51 | 64.32 | 48.73 | 13.68 | 47.92 | 33.67 | 39.20 | | PGD | GATA | 48.75 | 48.93 | 28.57 | 75.09 | 20.08 | 23.41 | 43.88 | 25.08 | 19.31 | 7.91 | 4.26 | 21.87 | 65.43 | 50.19 | 14.67 | 49.62 | 34.19 | 40.00 | | AutoAttack | Endpoint KL | 48.74 | 49.33 | 27.13 | 74.33 | 18.27 | 20.65 | 41.21 | 21.97 | 16.86 | 4.30 | 2.91 | 19.48 | 62.76 | 46.83 | 11.48 | 47.41 | 32.10 | 38.12 | | AutoAttack | GATA | 47.18 | 47.21 | 26.42 | 74.50 | 18.15 | 21.00 | 42.27 | 22.75 | 17.23 | 6.21 | 3.60 | 20.02 | 63.74 | 48.51 | 12.70 | 49.47 | 32.56 | 38.86 | | CW | Endpoint KL | 49.08 | 49.80 | 27.26 | 74.25 | 18.33 | 20.69 | 41.95 | 22.08 | 16.33 | 4.24 | 2.82 | 19.73 | 62.82 | 46.95 | 12.19 | 47.32 | 32.24 | 38.22 | | CW | GATA | 47.54 | 47.67 | 26.76 | 74.52 | 18.33 | 21.26 | 42.93 | 22.75 | 16.97 | 6.25 | 3.78 | 20.25 | 63.90 | 48.74 | 13.87 | 49.46 | 32.81 | 39.04 |

  • These gains come with little additional cost. Table 5 shows FLOPs and peak memory nearly identical to Endpoint KL.

AuthorsOpenReview ↗

Response to Weaknesses 2 (b)

Limited methodological novelty and marginal improvements: GATA primarily combines endpoint-based representation interpolation with existing KL-based consistency regularization. The improvements over endpoint KL are modest, and the experiments do not adequately distinguish the effects of gradual alignment from changes in effective loss and gradient scales. This raises questions about the methodological novelty of the proposed approach. These concerns could be alleviated by establishing the limitations of endpoint supervision, demonstrating a causal relationship between intermediate semantic drift and robustness failures, and deriving a sufficiently tight approximation error bound that supports the surrogate geometry.

(3) Loss scale control.

We agree that varying the number of intermediate states can also change the scale of the summed KL objective. To examine whether the gains mainly arise from this change, we conducted additional experiments using:

Lscaled=K∑k=1KDKL(pk−1∥pk),K=n+1.\mathcal{L}_{\mathrm{scaled}} = K \sum_{k=1}^{K} D_{\mathrm{KL}}(p_{k-1} \Vert p_k), \qquad K=n+1.

Under a local quadratic approximation, when adjacent predictive distributions are sufficiently close, the summed KL decreases approximately as (1/K) as the path is subdivided. Multiplication by (K) compensates for this leading-order scale change. The supplementary results are as follows:

\begin{array}{c|c|c|c|c} \hline n & \text{PGD} & \text{CW} & \text{AutoAttack} & \text{Clean} \\\\ \begin{array}{c} \vphantom{H} \\\\ \hline 0 \\\\ 1 \\\\ 3 \\\\ 5 \\\\ 7 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 31.94 & 39.47 \\\\ 32.05 & 39.47 \\\\ 32.01 & 39.43 \\\\ \mathbf{32.13} & \mathbf{40.13} \\\\ 32.00 & 39.45 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 30.41 & 38.28 \\\\ 30.69 & 38.43 \\\\ 30.69 & 38.41 \\\\ \mathbf{30.74} & \mathbf{39.03} \\\\ 30.59 & 38.35 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 29.97 & 37.93 \\\\ 30.23 & 38.06 \\\\ 30.29 & 38.10 \\\\ \mathbf{30.71} & \mathbf{39.00} \\\\ 30.14 & 38.00 \end{array} & \begin{array}{c} \text{Avg.} \\\\ \hline 51.63 \\\\ 51.37 \\\\ 51.34 \\\\ \mathbf{53.44} \\\\ 51.41 \end{array} \\\\ \hline \end{array} \end{gathered}$$ The following are the results shown in Table 3 of the paper (without loss scale control). $$\begin{gathered} \begin{array}{c|c|c|c|c} \hline n & \text{PGD} & \text{CW} & \text{AutoAttack} & \text{Clean} \\\\ \begin{array}{c} \vphantom{H} \\\\ \hline 0 \\\\ 1 \\\\ 3 \\\\ 5 \\\\ 7 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 31.94 & 39.47 \\\\ 32.24 & 39.98 \\\\ \mathbf{32.30} & \mathbf{40.21} \\\\ 31.94 & 39.98 \\\\ 31.66 & 39.76 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 30.41 & 38.28 \\\\ \mathbf{31.07} & 39.07 \\\\ \mathbf{31.07} & \mathbf{39.25} \\\\ 30.71 & 39.01 \\\\ 30.38 & 38.74 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 29.97 & 37.93 \\\\ \mathbf{30.46} & 38.59 \\\\ \mathbf{30.46} & \mathbf{38.75} \\\\ 30.06 & 38.47 \\\\ 29.71 & 38.19 \end{array} & \begin{array}{c} \text{Avg.} \\\\ \hline 51.63 \\\\ 52.64 \\\\ 53.26 \\\\ 53.43 \\\\ \mathbf{53.44} \end{array} \\\\ \hline \end{array} \end{gathered}$$ Taken together, the results in the two tables show that: - **Robustness gains persist after scale compensation.** With this compensation, all settings with intermediate states still achieve higher average robust accuracy than the endpoint baseline under all three attacks. - **Controlling the loss scale does not materially change the results.** The best results in the two tables are comparable, suggesting that the loss scale has little effect on performance. (4) **Approximation-error bounds.** - **Accurate trajectory reconstruction is not our objective.** We aim to construct effective transition supervision at low cost, rather than minimize geometric distance to the real trajectory. - **Directly using real trajectory states does not improve performance in our experiments.** PGD-Traj uses actual intermediate attack states instead of interpolated substitutes. Nevertheless, Table 4 reports lower average robust accuracy than GATA, while Table 5 shows higher computational and memory costs. **Thus, retaining real trajectory states does not necessarily yield better training outcomes.** - Existing results show no clear robustness benefit from more accurate trajectory fitting (Tables 4, 5). - Fitting accuracy lacks a standardized evaluation framework. Its scale varies across backbones and datasets, and no established framework links the degree of fitting accuracy to robustness. **Therefore, a tight approximation error bound is not aligned with our method’s objective and does not directly guarantee robustness.**
Reviewer_WjXHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GATA, an adversarial fine-tuning method for vision-language models. It interpolates between clean and adversarial image features and aligns predictions at neighboring points with KL divergence. Experiments report robustness gains across classification tasks and a downstream vision-language model.

Critical strengths
  1. The method is simple and adds little computation. It introduces intermediate supervision without encoding extra intermediate images.

  2. The experiments cover several attacks, datasets, and backbones.

Critical weaknesses
  1. The baseline comparison is not convincing yet. FARE and PMG-AFT perform much worse than TeCoA in Table 1, despite competitive or stronger results in their original papers. The unexplained gap makes GATA's advantage hard to judge, particularly given its substantial loss of clean accuracy. GATA's gains in zero-shot robustness also appear modest.

  2. The gains over endpoint KL in the ablation are small. For a fixed smooth prediction path, each adjacent KL term is second order in the step size 1/K, so summing K terms in Equation (14) gives a loss that scales roughly as 1/K. Increasing K may therefore lead to smaller updates at the same learning rate. The ablation does not separate this effect from the benefit of transition supervision.

Questions for authors

Please see the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Weakness 1

Thank you for raising this concern. All methods in Table 1 were rerun under a common training and evaluation protocol rather than copied from their original papers. Regarding the reproduction of TeCoA, FARE, and PMG-AFT, we further checked their implementations and configurations, as detailed below.

(1) PMG-AFT.

(a) Running the official code did not reproduce the performance shown in the paper. We used the official PMG-AFT repository, setting fine-tuning to PGD-10 with (ϵ=2/255)(\epsilon=2/255) and a step size of (1/255):

python PMG_AFT.py --batch_size 256 --root ./data --dataset tinyImageNet --name wangsibo --train_eps 2 --train_numsteps 10 --train_stepsize 1

Under PGD-10 evaluation at (ϵ=2/255)(\epsilon=2/255), robust accuracy was 23.56% on CIFAR-10, 5.78% on CIFAR-100 2.02 on DTD, and 0.03 on EuroSAT, below the performance shown in Figure 4 of the original paper. Thus, low performance was also observed when running the official code, not only our adapted implementation.

(b) Our adapted implementation closely reproduced the reported robustness under the weaker attack setting. At (ϵ=1/255)(\epsilon=1/255), average robust accuracy was 31.63%, compared with 31.95% in the original paper (PMG-AFT Table 1). Average AutoAttack accuracy was 17.88%, compared with 17.91% (PMG-AFT Table 3). Average clean accuracy was 53.72%, compared with 55.71% (PMG-AFT Table 2). These results are reported in Table 10.

(c) We identified implementation differences that may affect the results, but have not established their causal role. Specifically:

  • The relevant logit computation in the official implementation omits CLIP’s temperature scaling.
  • The official implementation did not utilize the CLIP’s ImageNet normalization statistics.

These differences may affect training and evaluation, but we have not isolated their effects through controlled ablations. We therefore report them as implementation observations, not as confirmed errors in the official code.

(2) FARE: the original paper and our comparison use different settings.

SettingFARE, Table 4Our paper, Table 1
BackboneViT-L/14ViT-B/32
Fine-tuning datasetImageNetTinyImageNet
Number of evaluation datasets1416

Consequently, the numerical results and method rankings in FARE’s original paper cannot be directly transferred to our setting. Our results come from retraining and evaluating the methods under our common protocol.

(3) TeCoA: implementation source and configuration. We reproduced TeCoA following the implementation and instructions provided by the official TGA-ZSR repository. For Table 1, fine-tuning uses PGD-10 with (ϵ=2/255)(\epsilon=2/255) and a step size of (1/255); PGD evaluation uses PGD-100 with the same budget and step size.

(4) Clean accuracy. We acknowledge GATA’s clean-accuracy loss relative to the original CLIP. However, its average clean accuracy is comparable to UCAT’s—45.31% versus 45.21%—and exceeds that of TeCoA and FARE. Thus, its robustness improvement over UCAT does not come with an additional loss in average clean accuracy. Although PMG-AFT exhibits the smallest drop in clean accuracy, its robust accuracy is low, which is consistent with the trade-off between clean performance and adversarial robustness.

(5) Robustness gains and practical advantages. GATA achieves the highest average robust accuracy under every evaluated attack in Table 1. Figure 9 further shows that GATA’s relative advantage becomes more apparent as the fine-tuning attack strength increases.

(6) Additional advantages.

  • Simple implementation: GATA requires only a few additional lines of code in the KL loss.
  • Low overhead: Its computational and memory costs are nearly identical to those of Endpoint KL.
  • Transferability: Experiments across different datasets, backbone and vision-language model (VLM) architectures further demonstrate its transferability.
AuthorsOpenReview ↗

Response to Weakness 2 (a)

Thank you for this comment. To further investigate the advantages of our method over endpoint KL, we conducted an additional experiment in which both methods were fine-tuned at (ϵ=4/255)(\epsilon=4/255) and evaluated on clean images and adversarial examples generated at (ϵ=1/255)(\epsilon=1/255), across 16 datasets. The results are as follows

\begin{array}{c|l|cccccccccccccccc|cc} \hline & \text{Method} & \substack{\text{Tiny}\text{ImageNet}} & \text{CIFAR10} & \text{CIFAR100} & \text{STL10} & \text{SUN397} & \text{Food101} & \substack{\text{Oxford}\\\\\text{Pets}} & \substack{\text{Flowers}\\\\\text{102}} & \text{DTD} & \text{EuroSAT} & \substack{\text{FGVC}\\\\\text{Aircraft}} & \text{ImageNet} & \substack{\text{Caltech}\\\\\text{101}} & \substack{\text{Caltech}\\\\\text{256}} & \substack{\text{Stanford}\\\\\text{Cars}} & \text{PCAM} & \text{Average} & H \\\\ \hline \text{Clean} & \substack{\text{Endpoint KL}\\\\\text{GATA (Ours)}} & \substack{\mathbf{67.49}\\\\66.52} & \substack{\mathbf{76.24}\\\\74.79} & \substack{\mathbf{47.73}\\\\47.33} & \substack{\mathbf{88.77}\\\\88.56} & \substack{\mathbf{34.00}\\\\33.53} & \substack{41.28\\\\\mathbf{43.67}} & \substack{62.80\\\\\mathbf{64.51}} & \substack{35.18\\\\\mathbf{38.05}} & \substack{26.54\\\\\mathbf{28.24}} & \substack{12.49\\\\\mathbf{17.54}} & \substack{7.80\\\\\mathbf{9.21}} & \substack{34.05\\\\\mathbf{35.42}} & \substack{76.14\\\\\mathbf{77.84}} & \substack{63.41\\\\\mathbf{65.56}} & \substack{26.71\\\\\mathbf{30.06}} & \substack{\mathbf{50.01}\\\\\mathbf{50.01}} & \substack{46.92\\\\\mathbf{48.18}} & \substack{\text{—}\\\\\text{—}} \\\\ \hline \text{PGD} & \substack{\text{Endpoint KL}\\\\\text{GATA (Ours)}} & \substack{\mathbf{50.73}\\\\48.75} & \substack{\mathbf{50.82}\\\\48.93} & \substack{\mathbf{29.35}\\\\28.57} & \substack{74.86\\\\\mathbf{75.09}} & \substack{\mathbf{20.54}\\\\20.08} & \substack{22.78\\\\\mathbf{23.41}} & \substack{42.52\\\\\mathbf{43.88}} & \substack{23.69\\\\\mathbf{25.08}} & \substack{18.72\\\\\mathbf{19.31}} & \substack{4.88\\\\\mathbf{7.91}} & \substack{3.69\\\\\mathbf{4.26}} & \substack{21.51\\\\\mathbf{21.87}} & \substack{64.32\\\\\mathbf{65.43}} & \substack{48.73\\\\\mathbf{50.19}} & \substack{13.68\\\\\mathbf{14.67}} & \substack{47.92\\\\\mathbf{49.62}} & \substack{33.67\\\\\mathbf{34.19}} & \substack{39.20\\\\\mathbf{40.00}} \\\\ \hline \text{AutoAttack} & \substack{\text{Endpoint KL}\\\\\text{GATA (Ours)}} & \substack{\mathbf{48.74}\\\\47.18} & \substack{\mathbf{49.33}\\\\47.21} & \substack{\mathbf{27.13}\\\\26.42} & \substack{74.33\\\\\mathbf{74.50}} & \substack{\mathbf{18.27}\\\\18.15} & \substack{20.65\\\\\mathbf{21.00}} & \substack{41.21\\\\\mathbf{42.27}} & \substack{21.97\\\\\mathbf{22.75}} & \substack{16.86\\\\\mathbf{17.23}} & \substack{4.30\\\\\mathbf{6.21}} & \substack{2.91\\\\\mathbf{3.60}} & \substack{19.48\\\\\mathbf{20.02}} & \substack{62.76\\\\\mathbf{63.74}} & \substack{46.83\\\\\mathbf{48.51}} & \substack{11.48\\\\\mathbf{12.70}} & \substack{47.41\\\\\mathbf{49.47}} & \substack{32.10\\\\\mathbf{32.56}} & \substack{38.12\\\\\mathbf{38.86}} \\\\ \hline \text{CW} & \substack{\text{Endpoint KL}\\\\\text{GATA (Ours)}} & \substack{\mathbf{49.08}\\\\47.54} & \substack{\mathbf{49.80}\\\\47.67} & \substack{\mathbf{27.26}\\\\26.76} & \substack{74.25\\\\\mathbf{74.52}} & \substack{\mathbf{18.33}\\\\\mathbf{18.33}} & \substack{20.69\\\\\mathbf{21.26}} & \substack{41.95\\\\\mathbf{42.93}} & \substack{22.08\\\\\mathbf{22.75}} & \substack{16.33\\\\\mathbf{16.97}} & \substack{4.24\\\\\mathbf{6.25}} & \substack{2.82\\\\\mathbf{3.78}} & \substack{19.73\\\\\mathbf{20.25}} & \substack{62.82\\\\\mathbf{63.90}} & \substack{46.95\\\\\mathbf{48.74}} & \substack{12.19\\\\\mathbf{13.87}} & \substack{47.32\\\\\mathbf{49.46}} & \substack{32.24\\\\\mathbf{32.81}} & \substack{38.22\\\\\mathbf{39.04}} \\\\ \hline \end{array}$$ From the results, we can see that **Consistent improvements in average performance.** GATA improves average clean accuracy from 46.92% to 48.18% (+1.26 percentage points). It also improves average robust accuracy under all three evaluated attacks: from 33.67% to 34.19% under PGD, from 32.10% to 32.56% under AutoAttack, and from 32.24% to 32.81% under CW. Thus, the robustness gains are accompanied by improved clean accuracy. **Benefits across multiple datasets.** Under each attack, GATA outperforms Endpoint KL on 12 of the 16 datasets. These results indicate that the improvements in this additional setting are spread across multiple datasets. Because our method is simple to implement and the additional computational and memory costs are negligible, these improvements make it practically applicable.
AuthorsOpenReview ↗

Response to Weakness 2 (b)

Loss scale control.

We agree that varying the number of intermediate states can also change the scale of the summed KL objective. To examine whether the gains mainly arise from this change, we conducted additional experiments using:

Lscaled=K∑k=1KDKL(pk−1∥pk),K=n+1.\mathcal{L}_{\mathrm{scaled}} = K \sum_{k=1}^{K} D_{\mathrm{KL}}(p_{k-1} \Vert p_k), \qquad K=n+1.

Under a local quadratic approximation, when adjacent predictive distributions are sufficiently close, the summed KL decreases approximately as (1/K) as the path is subdivided. Multiplication by (K) compensates for this leading-order scale change. The supplementary results are as follows:

\begin{array}{c|c|c|c|c} \hline n & \text{PGD} & \text{CW} & \text{AutoAttack} & \text{Clean} \\\\ \begin{array}{c} \vphantom{H} \\\\ \hline 0 \\\\ 1 \\\\ 3 \\\\ 5 \\\\ 7 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 31.94 & 39.47 \\\\ 32.05 & 39.47 \\\\ 32.01 & 39.43 \\\\ \mathbf{32.13} & \mathbf{40.13} \\\\ 32.00 & 39.45 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 30.41 & 38.28 \\\\ 30.69 & 38.43 \\\\ 30.69 & 38.41 \\\\ \mathbf{30.74} & \mathbf{39.03} \\\\ 30.59 & 38.35 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 29.97 & 37.93 \\\\ 30.23 & 38.06 \\\\ 30.29 & 38.10 \\\\ \mathbf{30.71} & \mathbf{39.00} \\\\ 30.14 & 38.00 \end{array} & \begin{array}{c} \text{Avg.} \\\\ \hline 51.63 \\\\ 51.37 \\\\ 51.34 \\\\ \mathbf{53.44} \\\\ 51.41 \end{array} \\\\ \hline \end{array} \end{gathered}$$ The following are the results shown in Table 3 of the paper (without loss scale control). $$\begin{gathered} \begin{array}{c|c|c|c|c} \hline n & \text{PGD} & \text{CW} & \text{AutoAttack} & \text{Clean} \\\\ \begin{array}{c} \vphantom{H} \\\\ \hline 0 \\\\ 1 \\\\ 3 \\\\ 5 \\\\ 7 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 31.94 & 39.47 \\\\ 32.24 & 39.98 \\\\ \mathbf{32.30} & \mathbf{40.21} \\\\ 31.94 & 39.98 \\\\ 31.66 & 39.76 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 30.41 & 38.28 \\\\ \mathbf{31.07} & 39.07 \\\\ \mathbf{31.07} & \mathbf{39.25} \\\\ 30.71 & 39.01 \\\\ 30.38 & 38.74 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 29.97 & 37.93 \\\\ \mathbf{30.46} & 38.59 \\\\ \mathbf{30.46} & \mathbf{38.75} \\\\ 30.06 & 38.47 \\\\ 29.71 & 38.19 \end{array} & \begin{array}{c} \text{Avg.} \\\\ \hline 51.63 \\\\ 52.64 \\\\ 53.26 \\\\ 53.43 \\\\ \mathbf{53.44} \end{array} \\\\ \hline \end{array} \end{gathered}$$ Taken together, the results in the two tables show that: **Robustness gains persist after scale compensation.** With this compensation, all settings with intermediate states still achieve higher average robust accuracy than the endpoint baseline under all three attacks. **Controlling the loss scale does not materially change the results.** The best results in the two tables are comparable, suggesting that the loss scale has little effect on performance.
Reviewer_6FoSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper proposes Gradual Adversarial Transition Alignment to improve the adversarial robustness of vision-language models. It analyzes attack trajectories and observes relatively stable global displacement despite local fluctuations. GATA constructs virtual intermediate states between frozen clean features and fine-tuned adversarial features, then aligns adjacent predictions using KL divergence. Experiments cover multiple datasets, attacks, model backbones, and transfer to LLaVA.

Critical strengths
  1. The method is simple and has low reported additional cost. Virtual states are constructed in feature space.

  2. The evaluation covers multiple datasets and attacks.

Critical weaknesses
  1. GATA combines feature interpolation with adjacent KL alignment, but provides no direct comparison with related methods such as AdvSimplex [1]. The trajectory analysis motivates the design, yet does not establish that this construction offers a distinct advantage over existing alternatives. The paper does not sufficiently establish its contribution beyond existing path-based methods.

  2. The central ablation does not isolate the claimed mechanism. Varying the number of intermediate states changes both the alignment structure and the scale of the summed KL objective. Without a loss-scale control, the results cannot distinguish the benefit of gradual alignment from changes in optimization strength. This weakens the evidence for the paper’s central explanation.

  3. In Table 13, although three intermediate states increase the average AutoAttack accuracy from 29.97% to 30.46%, excluding PCAM reverses this trend (29.51% to 29.45%). The reported improvement is largely driven by a single dataset rather than a consistent benefit of intermediate states across datasets.

  4. The LLaVA evaluation relies heavily on an overly permissive keyword-based success criterion, where merely mentioning or quoting the target content may be counted as a successful attack without demonstrating semantic success. Moreover, the subsequently added human evaluation lacks clearly defined criteria and basic reliability measures, such as inter-annotator agreement and disagreement resolution.

  5. The paper leaves the limits of its effectiveness insufficiently explained. GATA falls below UCAT in average AutoAttack accuracy under both weaker adversarial training and ImageNet fine-tuning. The suggested explanation based on smaller representation shifts is not directly tested.

[1]Dong, J., Koniusz, P., Zhang, Y., Zhu, H., Liu, W., Qu, X., & Ong, Y. S. (2025). Improving zero-shot adversarial robustness in vision-language models by closed-form alignment of adversarial path simplices. In proceedings of the 42th International Conference on Machine Learning.

Questions for authors
  1. What evidence establishes GATA’s distinct advantage over AdvSimplex and other related path-based methods? Can you provide a comparison under matched training conditions?

  2. When intermediate states change, the loss scale and gradient magnitude may also change. The performance gains should be disentangled from these effects to establish whether they can be attributed to progressive alignment.

  3. Why does PCAM account for most of the average AutoAttack improvement in Table 13?

  4. The evaluation criteria and human assessment details for LLaVA defense remain vague. Please clarify the evaluation protocol and expand the evaluation scope.

  5. Can you directly test whether representation-shift magnitude explains GATA’s varying gains, including the ImageNet fine-tuning results?

12 comments
AuthorsOpenReview ↗

Response to Weaknesses 1 and Questions 1

GATA combines feature interpolation with adjacent KL alignment, but provides no direct comparison with related methods such as AdvSimplex. The trajectory analysis motivates the design, yet does not establish that this construction offers a distinct advantage over existing alternatives. The paper does not sufficiently establish its contribution beyond existing path-based methods.

What evidence establishes GATA’s distinct advantage over AdvSimplex and other related path-based methods? Can you provide a comparison under matched training conditions?

Thank you for your great comments.

(1) Regarding direct comparison.

  • Among all the path-based methods cited in our paper, only AdvSimplex addresses the same research problem and allows a meaningful comparison with our method.

  • Official code unavailable. In Section 2.2, we cite studies in adversarial robustness that use the concept of attack paths. Among them, AdvSimplex is the most closely related work in our subfield. However, AdvSimplex does not provide official code, preventing us from conducting direct empirical comparisons.

(2) The main distinctions between GATA and these path-based methods are as follows:

  • Modeling informed by actual attack trajectories. Our design is grounded in the geometric properties of actual attack trajectories. Using local and global directional consistency metrics, we observe relatively consistent cumulative displacement directions across trajectories generated by five widely used attacks. This observation motivates an endpoint-based surrogate path for gradual prediction alignment, without explicitly processing intermediate adversarial samples.

  • Low computational overhead. GATA constructs intermediate states directly in representation space, requiring no additional image encoding or Jacobian/Hessian-vector product computations (AdvSimplex). Table 5 shows that its FLOPs and peak memory are nearly identical to those of Endpoint KL. Table 4 further shows higher average robust accuracy than PGD-Traj, which directly uses real PGD intermediate states, under both fine-tuning settings. These results support GATA’s effectiveness and efficiency.

  • Simple implementation. Our method requires only a few additional lines of code beyond the standard KL loss.

  • Transferability. We fine-tune and evaluate our method across different attack settings, backbones, and datasets, and further evaluate it on an LVLM. The results demonstrate its effectiveness across these varied settings.

AuthorsOpenReview ↗

Response to Weaknesses 2 and Questions 2

The central ablation does not isolate the claimed mechanism. Varying the number of intermediate states changes both the alignment structure and the scale of the summed KL objective. Without a loss-scale control, the results cannot distinguish the benefit of gradual alignment from changes in optimization strength. This weakens the evidence for the paper’s central explanation.

Do the gains over endpoint KL remain after controlling for the overall loss scale? Without this control, how can the improvement be attributed to gradual alignment?

Thank you for identifying this confounding factor.

We agree that varying the number of intermediate states can also change the scale of the summed KL objective. To examine whether the gains mainly arise from this change, we conducted additional experiments using:

Lscaled=K∑k=1KDKL(pk−1∥pk),K=n+1.\mathcal{L}_{\mathrm{scaled}} = K \sum_{k=1}^{K} D_{\mathrm{KL}}(p_{k-1} \Vert p_k), \qquad K=n+1.

Under a local quadratic approximation, when adjacent predictive distributions are sufficiently close, the summed KL decreases approximately as (1/K) as the path is subdivided. Multiplication by (K) compensates for this leading-order scale change. The supplementary results are as follows:

\begin{array}{c|c|c|c|c} \hline n & \text{PGD} & \text{CW} & \text{AutoAttack} & \text{Clean} \\\\ \begin{array}{c} \vphantom{H} \\\\ \hline 0 \\\\ 1 \\\\ 3 \\\\ 5 \\\\ 7 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 31.94 & 39.47 \\\\ 32.05 & 39.47 \\\\ 32.01 & 39.43 \\\\ \mathbf{32.13} & \mathbf{40.13} \\\\ 32.00 & 39.45 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 30.41 & 38.28 \\\\ 30.69 & 38.43 \\\\ 30.69 & 38.41 \\\\ \mathbf{30.74} & \mathbf{39.03} \\\\ 30.59 & 38.35 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 29.97 & 37.93 \\\\ 30.23 & 38.06 \\\\ 30.29 & 38.10 \\\\ \mathbf{30.71} & \mathbf{39.00} \\\\ 30.14 & 38.00 \end{array} & \begin{array}{c} \text{Avg.} \\\\ \hline 51.63 \\\\ 51.37 \\\\ 51.34 \\\\ \mathbf{53.44} \\\\ 51.41 \end{array} \\\\ \hline \end{array} \end{gathered}$$ The following are the results shown in Table 3 of our paper (without loss scale control). $$\begin{gathered} \begin{array}{c|c|c|c|c} \hline n & \text{PGD} & \text{CW} & \text{AutoAttack} & \text{Clean} \\\\ \begin{array}{c} \vphantom{H} \\\\ \hline 0 \\\\ 1 \\\\ 3 \\\\ 5 \\\\ 7 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 31.94 & 39.47 \\\\ 32.24 & 39.98 \\\\ \mathbf{32.30} & \mathbf{40.21} \\\\ 31.94 & 39.98 \\\\ 31.66 & 39.76 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 30.41 & 38.28 \\\\ \mathbf{31.07} & 39.07 \\\\ \mathbf{31.07} & \mathbf{39.25} \\\\ 30.71 & 39.01 \\\\ 30.38 & 38.74 \end{array} & \begin{array}{cc} \text{Avg.} & H \\\\ \hline 29.97 & 37.93 \\\\ \mathbf{30.46} & 38.59 \\\\ \mathbf{30.46} & \mathbf{38.75} \\\\ 30.06 & 38.47 \\\\ 29.71 & 38.19 \end{array} & \begin{array}{c} \text{Avg.} \\\\ \hline 51.63 \\\\ 52.64 \\\\ 53.26 \\\\ 53.43 \\\\ \mathbf{53.44} \end{array} \\\\ \hline \end{array} \end{gathered}$$ Taken together, the results in the two tables show that: - **Robustness gains persist after scale compensation.** With this compensation, all settings with intermediate states still achieve higher average robust accuracy than the endpoint baseline under all three attacks. - **Controlling the loss scale does not materially change the results.** The best results in the two tables are comparable, suggesting that the loss scale has little effect on performance. - With the effect of loss scale ruled out, the classification gains can be more confidently attributed to **gradual alignment**.
AuthorsOpenReview ↗

Response to Weaknesses 3 and Questions 3

The reported robustness improvement in Table 13 is largely driven by a single dataset. Three intermediate states raise average AutoAttack accuracy from 29.97% to 30.46%. Excluding PCAM, the average instead changes from approximately 29.51% to 29.45%. Although clean accuracy improves, this ablation provides weak evidence that intermediate states broadly improve robustness across datasets.

Why does PCAM account for most of the average AutoAttack improvement in Table 13? Are the gains stable across repeated training runs?

Thank you for pointing out this particular result.

  1. We agree that PCAM contributes most of the average AutoAttack gain in Table 13. We provide the following clarifications and additional results.
  • Improvements are not limited to PCAM. From (n=0) to (n=3), AutoAttack accuracy increases on 10 of the 16 datasets, remains unchanged on one, and decreases on five.

  • Clean accuracy improves across multiple datasets. Average clean accuracy increases from 51.63% at (n=0) to 53.26% at (n=3), reaching 53.44% at (n=7). This improvement is not primarily driven by a single dataset.

  • The aggregate results show a better clean–robust trade-off. At (n=3), both average clean accuracy and average robust accuracy under all three attacks exceed Endpoint KL.

  • Additional results show gains beyond PCAM. We added experiments using 10-step PGD fine-tuning at (\epsilon=4/255) and evaluation at (\epsilon=1/255) . GATA improves accuracy on 12 of 16 datasets under both PGD, CW and AutoAttack. Performance is not dominated by any single dataset: | | Method | TinyImageNet | CIFAR10 | CIFAR100 | STL10 | SUN397 | Food101 | OxfordPets | Flowers102 | DTD | EuroSAT | FGVC Aircraft | ImageNet | Caltech101 | Caltech256 | StanfordCars | PCAM | Average | H | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Clean | Endpoint KL | 67.49 | 76.24 | 47.73 | 88.77 | 34.00 | 41.28 | 62.80 | 35.18 | 26.54 | 12.49 | 7.80 | 34.05 | 76.14 | 63.41 | 26.71 | 50.01 | 46.92 | — | | Clean | GATA | 66.52 | 74.79 | 47.33 | 88.56 | 33.53 | 43.67 | 64.51 | 38.05 | 28.24 | 17.54 | 9.21 | 35.42 | 77.84 | 65.56 | 30.06 | 50.01 | 48.18 | — | | PGD | Endpoint KL | 50.73 | 50.82 | 29.35 | 74.86 | 20.54 | 22.78 | 42.52 | 23.69 | 18.72 | 4.88 | 3.69 | 21.51 | 64.32 | 48.73 | 13.68 | 47.92 | 33.67 | 39.20 | | PGD | GATA | 48.75 | 48.93 | 28.57 | 75.09 | 20.08 | 23.41 | 43.88 | 25.08 | 19.31 | 7.91 | 4.26 | 21.87 | 65.43 | 50.19 | 14.67 | 49.62 | 34.19 | 40.00 | | AutoAttack | Endpoint KL | 48.74 | 49.33 | 27.13 | 74.33 | 18.27 | 20.65 | 41.21 | 21.97 | 16.86 | 4.30 | 2.91 | 19.48 | 62.76 | 46.83 | 11.48 | 47.41 | 32.10 | 38.12 | | AutoAttack | GATA | 47.18 | 47.21 | 26.42 | 74.50 | 18.15 | 21.00 | 42.27 | 22.75 | 17.23 | 6.21 | 3.60 | 20.02 | 63.74 | 48.51 | 12.70 | 49.47 | 32.56 | 38.86 | | CW | Endpoint KL | 49.08 | 49.80 | 27.26 | 74.25 | 18.33 | 20.69 | 41.95 | 22.08 | 16.33 | 4.24 | 2.82 | 19.73 | 62.82 | 46.95 | 12.19 | 47.32 | 32.24 | 38.22 | | CW | GATA | 47.54 | 47.67 | 26.76 | 74.52 | 18.33 | 21.26 | 42.93 | 22.75 | 16.97 | 6.25 | 3.78 | 20.25 | 63.90 | 48.74 | 13.87 | 49.46 | 32.81 | 39.04 |

  1. Why did PCAM enjoy the greatest benefit in Table 13?
  • Reduced semantic degradation is a possible explanation. We have added Figure 6 to the paper. Figures 6(a) and 6(b) illustrate that GATA mitigates the decline in classification accuracy and image–correct-label similarity during attacks compared with Endpoint KL. This supports the role of gradual alignment in preserving semantic information.
  1. Repeated training.

Following the same settings as Table 13, we fine-tune and evaluate the model using different seeds. The results are shown below. Seed 0 corresponds to the results reported in Table 13.


Endpoint KL n=0:

EvaluationSeedPCAMAverageH
Clean050.7751.63-
PGD039.6431.6439.47

GATA (Ours) n=3:

EvaluationSeedPCAMAverageH
Clean050.0353.26-
Clean150.0253.25-
Clean1050.0153.29-
Clean10050.0953.22-
PGD046.8932.3040.21
PGD145.8432.0540.02
PGD1046.5132.1940.14
PGD10045.5532.0139.98
  • Consistent improvements across different seeds. Across repeated runs with different seeds, our method consistently improved Endpoint KL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.