acceptodds
Withdrawn from ICLR 2027

Revisiting Robustness Gains of Vision-Language Models under Label-Guided Attack

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

It is generally expected that, even after applying a defense, models remain less accurate on adversarial inputs than on clean inputs. However, we observe that Directional Bias-guided Defense (DBD), a test-time defense for zero-shot CLIP, achieves high adversarial accuracy under ground-truth-guided (GT-guided) attacks, even surpassing clean zero-shot CLIP. We study this counterintuitive result and identify a critical evaluation caveat: when the attack objective explicitly uses the ground-truth label, the attack process leaves recoverable label information in the attacked feature that can be exploited by a subsequent recovery mechanism. Theoretically, we analyze this result from three aspects: (1) GT-guided attacks push the features away from the true class; (2) DBD can recover such attack direction from the transformed views; and (3) after reconstructing the attacked feature along this direction, the true class margin can exceed the margin on the clean feature, accounting for why DBD can surpass clean zero-shot CLIP. Empirically, our results across multiple datasets, attack settings, and CLIP backbones support this analysis. We vary the attack-label assignment through ground-truth, pseudo, loop, fixed, and controlled-corruption settings. DBD accuracy tracks the ground-truth label proportion and changes across these assignments. We further evaluate conditional subsets, perturbation budgets, other test-time defense pipelines, and complete-pipeline adaptive attacks. Finally, we introduce **D-Pred**, a simpler direction-only prediction method that serves as a diagnostic probe of attack-induced label information. It surpasses DBD in average accuracy under GT-guided PGD-100 across all three backbones, yet approaches guessing on clean inputs, exposing attack-dependent predictive information in the recovered direction.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Robustness, Vision-Language Models, CLIP

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5odmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes why DBD achieves high accuracy under attacks generated with ground truth labels. It provides a feature space analysis, varies attack labels, and introduces D-Pred to examine whether the recovered direction alone carries predictive information.

Critical strengths
  1. The analysis makes the attack and recovery process easier to follow. It provides explicit conditions for direction recovery and margin improvement.

  2. The label controls help examine how the recovered predictions depend on the attack label. These experiments provide useful evidence about the interpretation of DBD’s reported accuracy.

Critical weaknesses
  1. The novelty is overstated. DBD already attributes its unusually high accuracy to information in attacks generated with ground truth labels. Its public rebuttal explicitly discusses label leakage, dependence on pseudo-label quality, and correction of initially wrong clean predictions. These substantially overlap with the paper’s central explanation, but the contribution statements do not clearly separate established findings from new results.

  2. D-Pred adds limited insight beyond DBD. It uses DBD’s existing recovery direction dˉ\bar d for classification and discards the original feature z^\hat z. In DBD’s amplified reconstruction z^+λdˉ\hat z+\lambda\bar d, predictions become increasingly dominated by dˉ\bar d as λ\lambda grows. D-Pred therefore represents a limiting case of this branch. Its high accuracy under GT-guided attacks and very low clean accuracy support its use as a diagnostic. However, DBD already explains that the recovered direction carries information left by the attack, so the paper needs to establish what additional understanding D-Pred provides.

Questions for authors

Which findings in the paper go beyond DBD and its public rebuttal? Please distinguish the new theoretical and empirical results from the previously reported observations and explanations.

Reviewer_4wvCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper revisits Directional Bias-guided Defense (DBD), a test-time defense for zero-shot CLIP, and studies why DBD can achieve higher accuracy on ground-truth-guided (GT-guided) adversarial inputs than on clean inputs. It explains this result through GT-guided feature shifts away from the true class, recovery of the reverse attack direction from transformed views, and reconstruction that increases the true-class margin, and introduces D-Pred as a direction-only diagnostic probe. Experiments cover RN50, ViT-B/32, and ViT-B/16 on 15 datasets under FGSM, PGD-10/50/100, CW, and AutoAttack, including label interventions, transfer to other test-time defenses, and a complete-pipeline adaptive attack.

Critical strengths
  1. The work has broad evaluation coverage and is clearly organized. It tests three CLIP backbones, 15 datasets, six attack configurations, transfer to other test-time defenses, and complete-pipeline adaptive attacks.
  2. The paper is clearly written, and the label-assignment interventions are informative.
Critical weaknesses
  1. The paper's novelty is limited. DBD itself already reports the anomaly and offers a directional account. D-Pred and label interventions strengthen that account rather than establish a distinct one.
  2. D-Pred is not a faithfulness test, it bypasses view averaging, DB-score gating, and reconstruction, so it cannot validate DBD's actual mechanism.
  3. The paper links direction alignment, margin gain, and final defended accuracy, but does not separate each link's independent contribution to DBD's adversarial accuracy. 4.Section 3.3 shows only that reconstruction yields a larger margin than the clean feature, not a positive margin, but a larger margin does not imply correct classification.
Questions for authors

See the Weaknesses above.

Reviewer_ouEeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines how GT-guided attacks introduce label information that DBD can exploit. It provides conditional theoretical analyses, expands attack-label controls, and introduces D-Pred, which classifies using only the recovered direction. The main argument is that high accuracy on these attacked inputs reflects attack-dependent information rather than necessarily indicating general robustness.

Critical strengths
  1. Loop, fixed, and corrupted-label controls extend the GT-versus-pseudo comparison and help characterize dependence on the supplied attack label.
  2. D-Pred isolates predictive information in the recovered direction. Its contrast between attacked and clean accuracy strengthens the empirical diagnosis across multiple datasets and backbones.
Critical weaknesses
  1. The conceptual advance over DBD is limited. DBD already discusses label information in attacks. The additions are mainly expanded controls, conditional formalization, and D-Pred. The manuscript does not sufficiently establish what substantive new conclusion follows beyond DBD’s existing explanation.
  2. The scope of the evaluation criticism is overstated. Ground-truth access is standard and valid in robustness evaluation. Replacing the attack label changes the objective, demonstrating label dependence rather than invalidating GT-guided evaluation itself. A stronger contribution would identify previously unrecognized failure modes or establish a reproducible evaluation protocol beyond the controls already considered by DBD.
Questions for authors
  1. Which substantive conclusion is new relative to DBD’s existing label-dependence analysis, beyond additional controls and formalization?
  2. Given that GT-guided attacks are standard, what specific evaluation failure do the label-replacement experiments establish beyond label dependence, and what reproducible protocol would address it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.