acceptodds
Desk rejected at ICLR 2027

IMPROVING ADVERSARIAL TRANSFERABILITY VIA CLS–PATCH RELATIONSHIP-GUIDED TOKEN GRADIENT SCALING

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

The Transferability of adversarial examples pose a practical security threat to Vi￾sion Transformers (ViTs), as examples crafted on a surrogate model may also fool unseen black-box models. Existing methods improve transferability by regular￾izing token gradients or modifying attention and intermediate features. However, intervention targets are often selected using token gradients or attention information without examining how CLSpatch interactions persist or vary across layers. Since these interactions contribute to the class representation, their strength and crosslayer variation can inform a more selective intervention. We therefore propose a method that characterizes CLSpatch relationships using token representations before and after selfattention and assesses their variation across layers. Among tokens with strong relationships, we identify those exhibiting greater variation and scale their gradients at QKV and MLP outputs during backpropagation, without altering forward activations. We randomly select 1,000 images from the ImageNet validation set for evaluation. Preliminary results indicate improved black-box transferability over the MI-FGSM baseline.

Primary area:
causal reasoning
Keywords:
Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2pBuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method to improve adversarial transferability against Vision Transformers. It measures CLS–patch relationships using token representations and their variation across Transformer layers. The selected tokens receive gradient scaling at QKV and MLP outputs during adversarial example generation. Preliminary experiments on 100 ImageNet images show a modest improvement over MI-FGSM.

Critical strengths
  • The paper addresses a relevant problem in transferable adversarial attacks.
  • The proposed method explores CLS–patch relationships for selective gradient intervention.
  • The preliminary evaluation includes ablation studies and uncertainty analysis.
Critical weaknesses

Incomplete evaluation of the proposed method. The paper describes an evaluation involving 1,000 ImageNet images and four surrogate models. However, the reported results cover only 100 images using ViT-B/16 as the surrogate. The results for the remaining surrogate models are unavailable. More importantly, the conclusion explicitly states that the experiments use an earlier nonstable-token selection variant rather than the complete method described in Section 3. Therefore, the reported results do not establish the effectiveness of the proposed method. This is a fundamental limitation, as the central technical contribution has not been fully evaluated.
Lack of competitive results against stronger baselines. The proposed method improves the average black-box attack success rate from 45.7% to 47.1% over MI-FGSM. However, Table 1 reports substantially higher success rates for existing methods, including TGR (70.3%), GNS (68.7%), and FPR (70.1%). These baseline results come from earlier experiments and are not directly comparable with the current pilot evaluation. Nevertheless, the available evidence does not demonstrate an advantage over stronger existing attacks. The practical significance of the proposed method therefore remains uncertain.
Insufficient validation of the token selection strategy. The proposed method selects tokens based on CLS–patch relationship strength and cross-layer variation. However, the paper does not establish why tokens with greater relationship variation should produce more transferable adversarial examples. Table 2 compares the earlier nonstable-token variant with random token selection, but does not isolate the contributions of relationship strength and cross-layer variation. The results therefore provide limited evidence that the specific selection criteria in Section 3 are responsible for improved transferability.
Limited and inconsistent transferability gains. The pilot experiment reports an overall improvement of 1.42 percentage points, with a 95% bootstrap confidence interval of [0.21, 2.70]. This provides some evidence of a positive gain on the evaluated images. However, the improvement on Transformer targets is only 1.00 percentage point, with a confidence interval of [-0.67, 2.70]. Several target models show little or no improvement. The observed benefits therefore appear modest and dependent on the target architecture. Their generality remains uncertain.
Incomplete implementation details. The method depends on the selected scoring layers and gradient-intervention layers, denoted by LsL_s and LgL_g. However, the experimental setup does not specify these layer sets. The token budget is also adjusted across stages for PiT-Ti without reporting the corresponding values. These configuration choices directly affect token selection and gradient propagation, making the method difficult to reproduce from the current description.

Questions for authors
  • What experimental results establish the effectiveness of the complete method proposed in Section 3?
  • How does the proposed method compare with TGR, GNS, and FPR under matched experimental settings?
  • What evidence supports the use of both relationship strength and cross-layer variation for token selection?
  • Is the advantage over random token selection statistically reliable across different target models?
  • Which scoring and intervention layers are used for each surrogate architecture?
Reviewer_FVnROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The submission includes the "Acknowledgment" section, which is non-anonymous revealing the author information. Also, the submission does not include the "AI disclosure section", which is mandatory this year.

Critical strengths

Please see the "Critical Weaknesses" section below.

Critical weaknesses

The submission includes the "Acknowledgment" section, which is non-anonymous revealing the author information. Also, the submission does not include the "AI disclosure section", which is mandatory this year.

Questions for authors

N/A

Reviewer_s8MzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a transferable-attack method for Vision Transformers. It characterizes CLS–patch relationships via cosine alignment of token representations before and after self-attention, selects K=64 patch tokens with both strong mean relationship strength and high cross-layer variation, and scales their backward gradients at QKV and MLP outputs by 0.5 during MI-FGSM optimization, leaving forward activations unchanged. Validation consists of a 100-image pilot with ViT-B/16 as surrogate (average black-box ASR 47.1% vs 45.7% for MI-FGSM), branch ablations, a random-token control (46.1±0.3), and bootstrap confidence intervals. The rows for three of the four surrogates in the main table are marked "Pending".

Critical strengths

Unusual reporting honesty for a preliminary study: unmeasured entries are marked "Pending" rather than filled; the authors include an identity-hook sanity check, an ℓ∞ budget audit, a random-token control, and bootstrap confidence intervals. This transparency is commendable, though it also makes the preliminary status explicit.

A structurally sound ablation design: the pilot separates QKV-only, MLP-only, and joint interventions and compares against random token selection; the criterion itself — combining relationship strength with cross-layer variation of CLS–patch alignment — is a concrete alternative to gradient- or attention-based token selection.

Critical weaknesses

The submission is incomplete, and the reported numbers do not measure the proposed method. Table 1 contains "Pending" entries for three of four surrogates; all results come from a 100-image pilot; and Section 5 states the available experiments used "an earlier nonstable-token selection variant," i.e., not the method specified in Section 3. The abstract's claim that evaluation uses 1,000 images overstates what was run. An unfinished main table and a method/variant mismatch cannot be fixed within a rebuttal.

Even taken at face value, the pilot does not establish the contribution. The gain over MI-FGSM is +1.4 points (45.7→47.1), while random token selection reaches 46.1±0.3, and the bootstrap CI on Transformer targets includes zero ([-0.67, 2.70]). Under identical settings in the same table, TGR (70.3), FPR (70.1), and GNS (68.7) exceed the proposed method by more than 20 points, so the method is also not competitive with the methods it builds upon.

Questions for authors

(Clarification) Which method produced the numbers in Tables 1–2: the method specified in Section 3, or the "earlier nonstable-token selection variant" acknowledged in Section 5? What exactly differs between them?

(Clarification) Will the 1,000-image, four-surrogate evaluation be completed? Without it the central claim cannot be assessed.

Is CLS–patch selection better than random selection at matched scale (more images and seeds)? The current margin (47.1 vs 46.1±0.3) is within the reported uncertainty on Transformer targets.

How sensitive are the results to K, η, and the choices of scoring/intervention layers Ls, Lg? Were these tuned on the evaluation images?

(Minor) The submission keywords ("Large Language Models") and primary area ("causal reasoning") do not match the paper's content.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.