acceptodds
Withdrawn from ICLR 2027

TIGA: Improving Targeted Transferability on MLLMs through Global-Regional Alignment and Neighborhood Gradient Fusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Black-box targeted adversarial transfer attacks on Multimodal Large Language Models (MLLMs) aim to induce outputs of attacker-specified semantics through carefully crafted adversarial examples without accessing any internal information of the victim models. Existing methods typically generate adversarial examples by aligning adversarial and target features on accessible surrogate models, utilizing model ensembles and random cropping to further enhance adversarial transferability. However, feature alignment of previous work may overlook coarse regional information relevant to language generation, while gradients evaluated at individual inputs may provide insufficient optimization guidance under local input variations. To address these limitations, we propose the TIGA framework, which improves targeted transferability from both feature and optimization perspectives. Specifically, Global-Regional Collaborative Alignment combines global features with patch representations within each region of a grid, preserving spatial information without strict patch-wise correspondence. Moreover, Neighborhood-Aware Gradient Fusion combines normalized gradients from the current adversarial input and a noise-perturbed neighbor, providing complementary guidance for targeted alignment under local input variations. Experiments on six closed-source and six open-source MLLMs demonstrate that TIGA outperforms existing methods in both attack success rates and output semantic similarities, with particularly pronounced gains on the closed-source models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Attack, Transferable Attack, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Cdp7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TIGA, a framework designed to improve targeted adversarial transferability against Multimodal Large Language Models (MLLMs) from both feature representation and optimization perspectives. Specifically, it introduces Global-Regional Collaborative Alignment to integrate global and regional visual features, and Neighborhood-Aware Gradient Fusion to incorporate the gradient information from a noise-perturbed neighbor. Experiments on 6 closed-source and 6 open-source MLLMs demonstrate improved attack success rates and semantic similarity compared with existing transfer attacks.

Critical strengths
  • The paper investigates the challenging and practically relevant problem of black-box targeted transfer attacks against MLLMs, without requiring access to victim model parameters or gradients.
  • The proposed framework achieves substantial improvements over existing baselines across a diverse set of closed-source and open-source MLLMs, particularly on commercial models.
Critical weaknesses
  • Both proposed components primarily rely on straightforward feature augmentation and input-gradient augmentation rather than fundamentally new attack mechanisms. GRCA concatenates spatially pooled regional features with global representations, while NGF introduces a noise-perturbed input and combines the resulting gradients. These designs closely resemble established feature aggregation, input diversification, and gradient ensemble techniques in adversarial attacks [1,2].
  • The paper argues that global feature alignment overlooks regional information relevant to language generation. However, the proposed GRCA simply applies fixed 2×2 average pooling and feature concatenation, without explicitly modeling regional semantic correspondence or interactions. The experiments mainly demonstrate performance improvements but provide limited evidence that these gains actually result from better alignment of region-level information used by victim MLLMs.
  • The experimental evaluation is primarily limited to image captioning with a fixed prompt, restricting the scope of the findings. It remains unclear whether the improvements generalize to more challenging multimodal tasks, such as visual question answering and complex visual reasoning.

[1] Xie C, et al. Improving transferability of adversarial examples with input diversity. In CVPR 2019.

[2] Wang X, et al. Enhancing the transferability of adversarial attacks through variance tuning. In CVPR 2021.

Questions for authors
  • What fundamentally distinguishes GRCA and NGF from existing feature aggregation, input diversification, and gradient ensemble techniques? Could the authors further clarify the methodological novelty of the proposed components?
  • Could the authors provide more direct evidence that GRCA improves regional semantic alignment in victim MLLMs, beyond the performance gains? For example, would analyzing region-level feature similarities or comparing against alternative regional aggregation strategies help validate the proposed motivation?
  • How well does TIGA generalize beyond image captioning? Could the authors provide additional evaluations on more challenging multimodal tasks, such as visual question answering or complex visual reasoning?
  • The evaluated defenses are primarily conventional image preprocessing methods, which may not adequately reflect the robustness of modern MLLMs against adversarial attacks. Could the authors evaluate whether TIGA remains effective against more advanced adversarial defenses, such as DiffPure [1]?

[1] Nie W, et al. Diffusion models for adversarial purification. In ICML 2022.

Reviewer_tmwyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIGA, a targeted transfer attack method for MLLMs. TIGA builds on the multi paradigm model ensemble and contrastive optimization framework of MPC-Attack. It adds global and regional feature alignment and combines gradients from the current input with gradients from a perturbed neighborhood. Experiments show that TIGA outperforms the compared baselines on the evaluated open source and closed source models.

Critical strengths
  1. The evaluation across models has some value. The method shows performance gains across multiple target models overall, which suggests some potential for transfer.
  2. The comparison accounts for computation. TIGA is compared with MPC-Attack under the same backpropagation budget. This provides some support that the gain does not come entirely from extra computation.
Critical weaknesses
  1. The contribution of the added method is limited. TIGA retains the multi paradigm model ensemble and contrastive optimization objective from MPC-Attack. The added parts are fixed regional pooling with feature concatenation, and gradient fusion between the current point and a perturbed neighborhood. Although the experiments show performance gains, the paper does not sufficiently explain what is substantively new compared with existing spatial pooling and neighborhood gradient methods. The supplement mainly describes how NGF differs from Variance Tuning and related methods at the mechanism level. It does not provide direct comparisons in a common attack setup with the same computation budget. The extra gain from GRCA after adding NGF is relatively small, and the paper does not report uncertainty for this gain. The evidence therefore places the contribution mainly in combining existing attack ideas. It does not establish the independent contribution or methodological novelty of the added modules.
  2. The evaluation protocol does not provide enough support for the main claims. ASR and AvgSim both use the same automated semantic score. They therefore do not provide validation independent of that evaluator. The paper does not report how well the score agrees with human judgments. It also does not report results for clean inputs or uncertainty across repeated runs. These omissions leave limited evidence that the results reflect genuine targeted semantic transfer. They also leave open the possibility of evaluator bias or a permissive semantic matching criterion.
Questions for authors
  1. The main text describes the attack as loss minimization, but the algorithm updates in the gradient ascent direction. Please state the exact objective and update direction, and clarify whether this discrepancy is a sign error or a wording error.
  2. How well does the GPT-4o semantic score agree with human judgments? Please provide results for clean inputs and random targets, together with standard deviations or confidence intervals across repeated runs.
  3. Under the same attack setup and gradient computation budget, does NGF outperform existing neighborhood gradient methods such as Variance Tuning? Is the extra gain from GRCA after adding NGF stable?
Reviewer_rgqXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes TIGA to address two limitations of existing targeted transfer attacks on MLLMs: global-regional feature mismatch and incomplete gradient guidance under local input variations. To this end, TIGA introduces Global-Regional Collaborative Alignment (GRCA) to jointly align global and regional features, and Neighborhood-Aware Gradient Fusion (NGF) to integrate gradients from the current adversarial input and its perturbed neighbor. Experiments on multiple closed-source and open-source MLLMs demonstrate the effectiveness of TIGA in improving targeted transferability.

Critical strengths

The authors evaluate the individual contributions of GRCA and NGF, investigate regional granularity and gradient fusion hyperparameters, and report adversarial example generation time. Importantly, the paper also includes a comparison with MPC-Attack under matched backward-propagation budgets, strengthening the experimental evidence beyond comparisons using identical iteration counts.

Critical weaknesses
  1. Insufficient evidence supporting the observations. The two observations presented in the Motivation section lack convincing empirical evidence. In particular, Observation 1 frequently uses speculative expressions such as "may", suggesting that the claimed limitations are primarily based on assumptions rather than direct experimental findings. Although regional features have demonstrated their importance in previous studies, their effectiveness in targeted transfer attacks cannot be simply inferred from prior work. The authors should provide additional experimental evidence or visualizations to substantiate these observations and better justify the motivation behind the proposed method.

  2. Limited evaluation across datasets. The authors evaluate TIGA on twelve closed-source and open-source MLLMs, demonstrating consistent improvements in ASR and AvgSim across different model architectures. However, all experiments are conducted under a single source-target benchmark setting. In contrast, some recent studies cited by the authors have conducted evaluations on multiple datasets. Therefore, the current experiments are insufficient to establish the generalizability of TIGA across different datasets. Additional cross-dataset evaluations are necessary to provide stronger evidence of its robustness and applicability.

  3. Unclear effectiveness of NGF throughout the optimization process. Although the ablation results demonstrate improvements from NGF, its effectiveness throughout the optimization process remains unclear. Gradient characteristics may change considerably across iterations, particularly during the later stages when the optimization approaches convergence. In such cases, the additional neighborhood gradients may provide limited complementary information, potentially reducing the benefit of NGF while nearly doubling the computational cost. The authors should analyze the evolution of gradient magnitudes and directions throughout the optimization process to clarify whether NGF consistently provides meaningful improvements, especially during later iterations.

Questions for authors

Q1. Empirical evidence for the motivation. The two observations in Section 3.2 are primarily supported by theoretical reasoning and findings from previous studies. Could the authors provide additional quantitative analyses or visualizations to directly demonstrate the global-regional feature mismatch and insufficient gradient guidance in targeted transfer attacks?

Q2. Generalizability across datasets. The experiments are conducted using a single source-target benchmark setting. Could the authors evaluate TIGA on additional datasets to demonstrate whether its improvements generalize to different data distributions and attack scenarios?

Q3. Effectiveness of NGF during optimization. Since gradient characteristics may change throughout the optimization process, could the authors analyze the gradient magnitudes and directional similarities between the two branches across iterations? Additionally, would applying NGF only during the early stages achieve comparable performance with lower computational overhead?

Q4. Statistical reliability. Although the experiments are repeated three times, the reported results mainly contain averages without standard deviations. Could the authors provide measures of variability?

Q5. Clarity of figures. The presentation of Figures 1 and 2 could be improved for better clarity. In Figure 1, although the relationship between the images and the corresponding model outputs can be inferred, I suggest making their associations more explicit rather than requiring readers to distinguish them on their own. Additionally, Figure 2 is somewhat confusing. According to the mathematical formulation, the framework involves three types of inputs: clean, target, and adversarial images. However, the overview only illustrates a single input image (x), making it difficult to understand how these images are involved in the framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.