acceptodds
Under review as a conference paper at ICLR 2027

Granularity-aware Semantic Transfer Learning for Targeted MLLM Attacks

Abstract

Transfer-based targeted attacks provide an important means of evaluating the robustness of closed-source multimodal large language models (MLLMs). Recent advances have improved transferability by exploiting increasingly fine-grained and hierarchical visual representations. However, existing objectives are still largely designed around the representations being aligned. Local views, patch representations, and intermediate features are often optimized through separate objectives, without explicitly connecting them through shared target semantics. As a result, semantically related signals can become fragmented across visual granularities. To address this issue, we introduce **Granularity-Aware Semantic Transfer (GAST)**, which leverages target semantics to coordinate surrogate supervision across different visual granularities. Specifically, Semantic Scope Construction (SSC) establishes distinct spatial scopes for different semantics in the target image, providing localized guidance for subsequent alignment. Correspondence-Aware Full-Patch Alignment (CFPA) then preserves the complete set of patch representations within each semantic scope and establishes optimal one-to-one correspondences between adversarial and target patches. Furthermore, Hierarchy-Wide Semantic Alignment (HWSA) propagates the aligned semantic signals across the intermediate representation hierarchy, enabling representations at different depths to jointly contribute to the transfer objective. Black-box evaluations on ten open-source and closed-source MLLMs demonstrate that GAST outperforms the second-best method by 12%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
MLLM Attacks, Adversarial Attacks, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_d1UtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a granularity-aware black-box transfer adversarial attack. The author refines existing cropping, hidden state alignment, and hidden state selection, respectively, with semantic scope construction, correspondence-aware full-patch alignment, and hierarchy-wise semantic alignment. The SSA emphasizes cropping within reasonable regions to control the abrupt perturbation brought by random cropping, while the CFPA provides a more robust alignment through computing the optimal transport in the hidden state. The experiment shows that this method achieves state-of-the-art results in almost all cases.

Critical strengths
  1. This paper provides a method with clear motivation and explanation. The method itself is easy to follow and performs reasonably well. The semantic scope construction is novel, providing alignment in a restricted region with strong semantic meanings.

  2. The authors provide extensive experiments across different victim model series, including both commercial models like GPT-5.4, GPT-5.6o, Gemini, and Claude Opus 5, and open-source series like Kimi, Qwen, and GLM, covering a wide range of victim models.

Critical weaknesses
  1. There is no ablation study on the hyperparameters introduced in this paper, although many are presented such as λc,λp,μ\lambda_c, \lambda_p, \mu as well as cropping parmeters.
  2. Appendix A.5 admits that the cost of CFPA is extensive when trying to align a 1024 tokens, but they also did not provide a time comparison.
Questions for authors

I am concerned about potential model bias (leak) in the evaluation, as the SSC used GPT-6 Astra to label the semantic areas and GPT-4o to act as the judge, since they are both GPT series. It would be better to include a small additional setting using either a different judge or a different labeling model.

4 comments
AuthorsOpenReview ↗

Response to Reviewer d1Ut (part1)

We thank the reviewer for the positive review. Below we respond to the comments in Questions (Q) and Weaknesses (W). The new results on the concept extractor, the judge, hyperparameter sensitivity, and computational cost have been added to the revised paper (Sections 4.4 and 4.5, Appendices A.5, A.7, A.8, and A.10), with all changes marked in red.

Q: GPT family annotator and judge.

We followed this suggestion and replaced each of the two components separately.

Different annotator. We replaced GPT-6-Astra with RAM++, an open-source image tagger, and passed its tags directly to Grounding DINO without any verification by GPT, keeping the number of scopes close to the original. On the 100-pair benchmark (averaged over GPT-5.4, Gemini-2.5-Flash, and Qwen3.8-Max), [email protected] changes from 75.0% to 72.7% and AvgSim from 0.688 to 0.685, which remains well above PRAF (60.3% and 0.610). On the OpenAI victim GPT-5.4, [email protected] stays at 69% with RAM++, the same as with GPT-6-Astra. This shows that the gain of GAST comes mainly from the semantic scopes of SSC rather than from a specific extractor. We have added these results to Section 4.5 (Table 4) and Appendix A.7 (Table 9) of the revised paper.

Different judge. We rescored the same victim outputs used in the paper (GPT-5.4, Gemini-3-Flash, Qwen3-VL-32B-Instruct and Kimi K3) with three judges outside the OpenAI family, namely Claude Sonnet 5, Qwen3.7 Flash and DeepSeek 4.1 Flash. Under all four judges, GAST ranks first among the five methods on all four metrics and leads the second-best method, PRAF, by 13.0 to 15.7 points in [email protected]. In addition, the scoring reference for each victim is the caption that the victim itself produces for the clean target image (Section 4.1), so no GPT model supplies the reference for victims outside the GPT family. The gains of GAST therefore reflect a genuine transfer of target semantics rather than a preference of the GPT-4o judge. We have added these results to Section 4.5 (Table 4) and Appendix A.8 (Table 10) of the revised paper.

Changing either the annotator or the judge leaves the conclusions unchanged, so the gains do not appear to be tied to GPT family annotation or judging.

W1: Hyperparameter sensitivity.

For a fair comparison, GAST adopts the same hyperparameter values as prior work (Appendix A.4). The values of the loss weights (λc,λp)=(0.5,1.5)(\lambda_c, \lambda_p) = (0.5, 1.5) follow PRAF, the momentum coefficient μ=0.9\mu = 0.9 takes the same value as in M-Attack-v2, and the crop ranges are taken from M-Attack-v2 and PRAF. Since GAST changes what the patch term aligns, from the top 60% of patches in PRAF to all patches under exact correspondence, the inherited loss weights are not guaranteed to remain suitable. We therefore ran a sensitivity study on λc\lambda_c and λp\lambda_p with 12 settings around the default, which is shown in bold. Results are on the 100-pair benchmark and averaged over GPT-5.4, Gemini-2.5-Flash, and Qwen3.8-Max.

λc\lambda_cλp\lambda_pAvgSim[email protected] (%)[email protected] (%)KMR (%)
0.00.50.66884.071.772.3
0.50.00.44952.330.055.0
0.50.50.65785.765.072.2
0.51.00.66686.768.372.9
0.51.50.68887.775.072.9
0.52.00.67285.771.071.3
1.00.50.64583.065.072.1
1.01.50.66284.769.073.7
1.50.50.62781.060.368.9
1.51.00.65883.371.071.6
1.52.00.66286.368.072.1
2.00.50.60375.757.768.4
2.01.50.64883.068.371.6

Removing the patch alignment term (λp=0\lambda_p = 0) causes a sharp drop to 30.0% [email protected], showing that correspondence-aware patch alignment is indispensable. When λp\lambda_p is small, increasing λc\lambda_c lowers performance, since the global CLS term then dominates the patch term. The inherited default remains the best setting in AvgSim, [email protected] and [email protected], and the other settings with λp≥1.0\lambda_p \geq 1.0 keep [email protected] between 68.0% and 71.0%, still 7.7 to 10.7 points above PRAF (60.3%). GAST therefore does not rely on a narrowly tuned choice of loss weights. We have added this sensitivity study to Appendix A.10 (Table 13) of the revised paper.

Unlike the loss weights, μ\mu and the crop ranges govern the optimizer and view sampling rather than the alignment objective that GAST modifies, so we keep them at these inherited values in all experiments. The only crop that GAST changes is the target crop of each semantic branch, which is placed according to its semantic scope, and this choice is evaluated by the Random Scope ablation in Table 3(a). Placing this crop on the semantic scope raises average [email protected] by 8.2 points over random placement. We have added the value of μ\mu to Appendix A.4 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer d1Ut (part2)

W2: Running time and memory cost.

As suggested, we compare the running time and peak GPU memory for generating an adversarial example on a single NVIDIA RTX 4090, including model loading.

MethodIterationsTime (s)Peak GPU memory (GB)
AnyAttack1 forward pass28.00.75
M-Attack30048.48.38
FOA30096.38.41
M-Attack-v2300462.48.30
MPCAttack30048.610.43
PRAF30067.412.09
GAST300746.79.79

Most of GAST's extra runtime comes from exact Hungarian matching solved sequentially on the CPU via SciPy. This matching only pairs adversarial and target patches outside the computation graph, adding no extra loss terms or surrogate passes. Given the pairs, the alignment loss costs the same as aligning patches in a fixed order. The remaining overhead comes from the per-scope branches and the alignment of all intermediate layers. Even so, GAST's peak GPU memory (9.79 GB) stays below that of PRAF (12.09 GB) and MPCAttack (10.43 GB).

PRAF and MPCAttack use the same five surrogate encoders as GAST. To check that this extra cost does not explain our gains, we further ran PRAF and MPCAttack for more iterations to match GAST's total FLOPs. Since GAST's cost scales with the number of scopes, we match FLOPs per image pair and list the average iterations below. Each cell reports AvgSim / [email protected] (%) / [email protected] (%) / KMR (%) on the 100-pair benchmark.

MethodIterationsGPT-5.4Gemini-2.5Qwen3.8-MaxAverage
MPCAttack3000.415 / 45 / 25 / 47.90.558 / 69 / 51 / 65.80.593 / 76 / 58 / 74.60.522 / 63.3 / 44.7 / 62.8
MPCAttack (FLOPs-matched)2,2130.431 / 53 / 29 / 53.30.625 / 79 / 62 / 68.30.628 / 84 / 61 / 76.30.561 / 72.0 / 50.7 / 66.0
PRAF3000.553 / 69 / 49 / 63.20.610 / 79 / 61 / 65.70.668 / 87 / 71 / 75.00.610 / 78.3 / 60.3 / 68.0
PRAF (FLOPs-matched)1,5000.611 / 76 / 58 / 64.20.664 / 87 / 68 / 69.40.705 / 90 / 76 / 78.50.660 / 84.3 / 67.3 / 70.7
GAST3000.659 / 86 / 69 / 69.50.677 / 85 / 73 / 72.80.727 / 92 / 83 / 76.60.688 / 87.7 / 75.0 / 72.9

More iterations boost PRAF and MPCAttack by 7.0 and 6.0 points in average [email protected]. However, GAST still leads both FLOPs-matched baselines by at least 5 points in [email protected] on every victim, showing that compute budget alone does not explain the performance gap. We have added both analyses to Section 4.4 (Table 4) and Appendix A.5 (Tables 5 and 6) of the revised paper.

Summary. We have made the following revisions in response to the reviewer's comments.

  1. Section 4.5 and Appendices A.7 and A.8: results with two alternative concept extractors, including the non-GPT tagger RAM++, and with three non-OpenAI judges.
  2. Appendices A.4 and A.10: the momentum coefficient μ=0.9\mu = 0.9 and the sensitivity study on the loss weights.
  3. Section 4.4 and Appendix A.5: running time, peak GPU memory, and comparison with FLOPs-matched baselines.

Across all of these analyses, GAST keeps a clear lead over the strongest baseline. Its gains therefore come from semantic-scope supervision rather than from GPT-family components, tuned hyperparameters, or extra computation.

We hope these results address the reviewer's concerns, and we would be glad to provide further details if needed.

Thank you for the detailed response and the additional experiments. The rebuttal has addressed most of my concerns, and I will maintain my score.

Thank you for your positive score and continued support for our work. We are glad that the additional experiments and clarifications have resolved your concerns.

Reviewer_n5HNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes GAST for targeted transfer attacks against black box MLLMs. Given a source image and a target image, GAST uses a vision language model and Grounding DINO to locate semantic regions in the target image. It then optimizes three parts. SSC constructs semantic scopes. CFPA establishes correspondences between source and target images using patch feature similarity and Hungarian matching. HWSA aligns CLS features and patch features across several intermediate layers. The evaluation covers 100 manually verified image pairs and 1000 expanded image pairs. It includes multiple closed source and open source MLLMs, several transfer attack baselines, and ablation settings. The reported metrics include ASR and AvgSim. GAST reports better attack performance in most settings.

Critical strengths
  1. The evaluation covers several closed source and open source MLLMs, including different types of target models. This gives the claim about transfer across models some support.
  2. The method is divided into SSC, CFPA, and HWSA, with ablation studies for these parts. The CFPA objective is also defined clearly enough to support understanding and reproduction.
Critical weaknesses
  1. The contributions of the semantic region and patch matching components are not sufficiently isolated. Once SSC produces RqR_q, the semantic description cqc_q does not directly enter the attack loss. The random region ablation therefore combines the effect of semantic understanding with the effect of better crop quality. Hungarian matching minimizes the stated feature cost, but that objective alone does not show that the matched patches refer to the same semantic entity. Compared with M-Attack and PRAF-Attack, GAST follows an existing route based on local and intermediate layer alignment. It therefore appears to be an incremental extension of existing methods, and the paper does not yet establish a new transfer attack principle.
  2. The source of the reported gains and the reliability of the evaluation remain unclear. GAST uses several surrogate encoders, layers, and alignment branches, but the paper does not compare the methods at a similar computational cost. This leaves open whether some of the gain comes from a larger optimization setup rather than from the proposed design. The main evaluation also depends heavily on a single GPT-4o judge. The paper reports no variation across repeated runs, no human evaluation of attack outcomes, and no independent image and text similarity measure. On Claude Opus 5, GAST does not consistently outperform M-Attack-v2, which limits the claim of broad superiority.
Questions for authors
  1. After SSC generates RqR_q, does the textual semantic description cqc_q enter the attack optimization directly? If cqc_q only determines the region, please compare the resulting regions with regions from a standard object detector, regions marked by humans, and local crops selected without semantic labels. These controls would separate the effect of semantic understanding from the effect of crop quality.
  2. Please report the total computation used by GAST and each baseline. Also clarify whether the comparisons use comparable surrogate models and optimization budgets. If GAST uses more encoders, layers, or separate loss terms, does its gain remain after computation is controlled?
  3. How stable are the GPT-4o judgments across prompt variants and repeated runs? Do human ratings or an independent image and text similarity measure support the main claims? Why does GAST fail to consistently outperform M-Attack-v2 on Claude Opus 5?
  4. For the semantic scopes generated for the expanded set of 1000 image pairs, what filtering and quality checks were applied? What are the rates of incorrect regions, empty regions, and semantically ambiguous scopes?
7 comments
AuthorsOpenReview ↗

Response to Reviewer n5HN (part1)

We thank the reviewer for the careful review and the specific questions. Below we respond to the comments in Questions (Q) and Weaknesses (W). The new results on alternative region sources, computational cost, and alternative evaluators have been added to the revised paper (Sections 4.4 and 4.5, Appendices A.5, A.7, and A.8), with all changes marked in red.

Q1: Role of cqc_q and comparison with other region sources.

As defined in Section 3.3, cqc_q does not enter the loss. It only locates RqR_q, and semantics acts in GAST through where the alignment is enforced. Below we compare the SSC regions with detector regions and with crops without semantic labels.

Detector regions. We replaced GPT-6-Astra with the open-source tagger RAM++ and fed its tags directly into Grounding DINO without GPT verification, keeping the scope count close to the original. Results on the 100-pair benchmark (averaged over GPT-5.4, Gemini-2.5-Flash, and Qwen3.8-Max) are shown below.

Region sourceAvgSim[email protected] (%)[email protected] (%)KMR (%)
SSC (paper)0.68887.775.072.9
RAM++ tags with Grounding DINO0.68588.072.773.5

Even though RAM++ only outputs short object tags, detector-derived regions nearly match SSC and far exceed PRAF (0.610 AvgSim, 60.3% [email protected]). This confirms that gains stem from localizing coherent semantic units rather than rich textual descriptions, which is consistent with cqc_q not entering the loss.

Human regions. Human region annotations are difficult to obtain, so we use the detector regions above as a substitute, which already perform on par with SSC.

Crops without semantic labels. This is the Random Scope control discussed in W1 below, which lowers average [email protected] on the four victims of Table 3 from 76.0% to 67.8%.

Regions that match semantic units therefore transfer far better than random regions, regardless of how they are obtained. We have added the detector results to Section 4.5 (Table 4) and Appendix A.7 (Table 9) of the revised paper.

Q2 & W2: Computational cost, surrogate parity, and FLOPs-matched evaluation.

All baselines are evaluated using their official surrogates and default settings under the same perturbation budget (ϵ=16/255\epsilon = 16/255). GAST adds no extra encoders: it uses the exact same five-encoder ensemble as PRAF and MPCAttack. Its additional compute comes strictly from aligning all intermediate layers and running one local branch per semantic scope. The FLOPs-matched experiment below directly controls for both factors.

For the total compute asked in Q2, we report the running time and peak GPU memory for generating an adversarial example on a single NVIDIA RTX 4090, including model loading.

MethodIterationsTime (s)Peak GPU memory (GB)
AnyAttack1 forward pass28.00.75
M-Attack30048.48.38
FOA30096.38.41
M-Attack-v2300462.48.30
MPCAttack30048.610.43
PRAF30067.412.09
GAST300746.79.79

Most of GAST's extra runtime comes from exact Hungarian matching solved sequentially on the CPU via SciPy. This matching only pairs adversarial and target patches outside the computation graph, adding no extra loss terms or surrogate passes. Given the pairs, the alignment loss costs the same as aligning patches in a fixed order. GAST's peak GPU memory (9.79 GB) still stays below that of PRAF (12.09 GB) and MPCAttack (10.43 GB).

For the equal-compute comparison in Q2, we ran PRAF and MPCAttack for more iterations to match GAST's total FLOPs. Since GAST's cost scales with the number of scopes, we match FLOPs per image pair and list the average iterations below. Each cell reports AvgSim / [email protected] (%) / [email protected] (%) / KMR (%) on the 100-pair benchmark.

MethodIterationsGPT-5.4Gemini-2.5Qwen3.8-MaxAverage
MPCAttack3000.415 / 45 / 25 / 47.90.558 / 69 / 51 / 65.80.593 / 76 / 58 / 74.60.522 / 63.3 / 44.7 / 62.8
MPCAttack (FLOPs-matched)2,2130.431 / 53 / 29 / 53.30.625 / 79 / 62 / 68.30.628 / 84 / 61 / 76.30.561 / 72.0 / 50.7 / 66.0
PRAF3000.553 / 69 / 49 / 63.20.610 / 79 / 61 / 65.70.668 / 87 / 71 / 75.00.610 / 78.3 / 60.3 / 68.0
PRAF (FLOPs-matched)1,5000.611 / 76 / 58 / 64.20.664 / 87 / 68 / 69.40.705 / 90 / 76 / 78.50.660 / 84.3 / 67.3 / 70.7
GAST3000.659 / 86 / 69 / 69.50.677 / 85 / 73 / 72.80.727 / 92 / 83 / 76.60.688 / 87.7 / 75.0 / 72.9

More iterations boost PRAF and MPCAttack by 7.0 and 6.0 points in average [email protected]. However, GAST still leads both FLOPs-matched baselines by at least 5 points in [email protected] on every victim, showing that compute budget alone does not explain the performance gap. We have added both analyses to Section 4.4 (Table 4) and Appendix A.5 (Tables 5 and 6) of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer n5HN (part2)

Q3 & W2: Evaluator stability across model families, LLM-free metrics, and Claude Opus 5 analysis.

Rather than varying the prompt of GPT-4o or resampling it, we tested a stronger form of evaluator variation. We rescored the same victim outputs and target captions used in the paper (the 100-pair benchmark on GPT-5.4, Gemini-3-Flash, Qwen3-VL-32B-Instruct, and Kimi K3) with three judges from other model families, namely Claude Sonnet 5, Qwen3.7 Flash and DeepSeek 4.1 Flash. Each cell reports AvgSim / [email protected] (%) / [email protected] (%) / KMR (%).

MethodGPT-4o (paper)Claude Sonnet 5Qwen3.7 FlashDeepSeek 4.1 Flash
PRAF0.608 / 76.3 / 60.3 / 67.90.590 / 71.3 / 39.8 / 78.50.678 / 84.0 / 70.3 / 77.80.688 / 81.8 / 66.3 / 77.8
GAST0.694 / 88.3 / 76.0 / 74.00.670 / 85.5 / 54.3 / 84.10.736 / 90.8 / 83.3 / 82.30.748 / 89.5 / 79.5 / 81.8

Under all four judges, GAST ranks first among the five methods on all four metrics and leads the second-best method, PRAF, by 13.0 to 15.7 points in [email protected]. The gains of GAST therefore reflect a genuine transfer of target semantics rather than a preference of the GPT-4o judge.

To evaluate outputs without any LLM judge, we also computed the cosine similarity between the Sentence-BERT embeddings (all-mpnet-base-v2 [4]) of each victim output and its target caption. This measure is deterministic and involves no prompt. We report it on five victims (GPT-5.4, Gemini-3-Flash, Qwen3-VL-32B-Instruct, Qwen3.8-Max, and Kimi K3).

MethodMean cosine similaritySimilarity ≥ 0.7 (%)
FOA0.49117.9
M-Attack-v20.57829.0
MPCAttack0.58029.2
PRAF0.62839.2
GAST0.68553.9

GAST achieves the highest score under both metrics on every single victim, closely matching the LLM-judge rankings. Furthermore, KMR, which directly measures the recall of human-verified target keywords in victim captions, is highest for GAST across all ten victims in the paper (Tables 1 and 2). Sentence-BERT thus provides a deterministic, LLM-free validation of our main claims, which KMR further supports at the level of human-verified keywords. We have added the judge and Sentence-BERT results to Section 4.5 (Table 4) and Appendix A.8 (Table 10) of the revised paper.

Regarding the concern that GAST does not consistently lead on Claude Opus 5, this model is substantially more robust than other victims across all tested transfer attacks. Every method stays near failure on it ([email protected] ≤14%\le 14\% and AvgSim ≤0.185\le 0.185), whereas GAST achieves ≥76%\ge 76\% [email protected] on all other nine victims. Prior work reports the same pattern, with far lower attack success rates on Claude than on other commercial models [1, 2]. Hu et al. [3] also find Claude models significantly more robust than other victims. They suggest that the visual training of Claude may differ substantially from that of public vision-language models, from which the surrogate encoders are drawn. The small differences among methods on Claude Opus 5 therefore reflect a floor effect shared by all methods rather than a limitation of semantic scope supervision. Even on this model, GAST obtains the highest KMR among all methods (Table 1).

[1] X. Jia et al. Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment. NeurIPS 2025.

[2] Y. Li et al. Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models. CVPR 2026.

[3] K. Hu et al. Transferable Adversarial Attacks on Black-Box Vision-Language Models. arXiv:2505.01050, 2025.

[4] N. Reimers and I. Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP 2019.

AuthorsOpenReview ↗

Response to Reviewer n5HN (part3)

Q4: Automated quality control and error sensitivity on the 1,000-pair benchmark.

Both the 100-pair and 1,000-pair benchmarks use the exact same automated pipeline (Appendix A.1) with zero manual curation, ensuring that the 100-pair results do not rely on hand-tuned boxes. Scopes are extracted offline once per target image prior to optimization. Specifically, GPT-6-Astra extracts candidate concepts, Grounding DINO localizes them, and GPT-6-Astra verifies the resulting boxes against the target image. This verification step directly addresses all three failure modes raised:

  1. Incorrect regions: Boxes that fail to capture their intended concept are discarded.
  2. Empty regions (0% rate): Missed salient concepts are re-queried, and every target image is guaranteed to retain at least one valid scope, which eliminates empty regions.
  3. Ambiguous scopes: Labels and box boundaries are refined, and densely overlapping instances of the same concept are merged into a single scope. Some scopes intentionally span interacting objects or scene context when the target semantics extends beyond a single object.

Across the 1,000-pair benchmark, this pipeline produces 3,834 scopes in total (3.83 per image).

While computing exact localization error rates across 3,834 boxes would require human ground-truth bounding boxes, our ablations tightly bound the impact of any residual pipeline noise:

  1. Semantic alignment is essential: Replacing SSC with Random Scopes (Table 3(a)) drops average [email protected] from 76.0% to 67.8%, confirming that gains do not come from arbitrary multi-cropping.
  2. Residual detector noise has minimal impact: Even when removing GPT verification entirely and feeding raw RAM++ tags into Grounding DINO (see Q1), GAST still achieves 72.7% [email protected] (vs. 75.0% with full SSC and 60.3% for PRAF).

At scale on the 1,000-pair benchmark (Appendix A.6, Tables 7 and 8), using these fully automated scopes, GAST ranks first across all seven victims at every threshold from 0.3 to 0.8.

Reviewer_NxZMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies targeted transfer attacks against closed-source MLLMs: adversarial examples are optimized on accessible surrogate vision encoders and are expected to induce unseen victim models to produce the semantics of a target image after cross-model transfer. The authors argue that recent work (M-Attack, FOA, PRAF, MPCAttack, etc.) incorporates local views, patch representations, and intermediate features, but optimizes these supervision signals independently per representation, so the same target semantic is fragmented across visual granularities. To address this, the paper proposes GAST, which uses target semantics to coordinate supervision across granularities.

Experiments cover ten victim MLLMs spanning six families (GPT, Gemini, Claude, Kimi, Qwen, GLM). The main benchmark uses 100 source-target pairs (NIPS 2017 + MSCOCO), extended to 1,000 pairs in the appendix. Metrics are AvgSim, [email protected]/0.7, and keyword matching rate (KMR) under an LLM-as-a-judge protocol (GPT-4o). GAST achieves the best results on most victims; the abstract claims a 12% improvement over the second-best method, with the largest advantage under the stricter [email protected] criterion.

Critical strengths
  1. The paper cleanly contrasts "designing separate objectives per representation" with coordinating supervision around shared target semantics, and maps its contributions one-to-one against M-Attack (local views), FOA (patches), and PRAF (intermediate-layer selection), making the incremental contribution boundary clear.
  2. Under equal cardinality and uniform mass, the Birkhoff-von Neumann theorem shows the optimal balanced transport plan can be realized as a one-to-one assignment (the Hungarian solution is exactly the OT solution), and the CFPA loss is upper-bounded by the Index loss. The authors also explicitly state in the appendix that "this is an optimality statement about the objective for fixed representations; it does not by itself guarantee black-box transferability," honestly separating theoretical claims from empirical effects.
  3. The 2×2 (Index/Hungarian × Top-60%/Full) design reveals that full-patch supervision becomes reliably beneficial only after patch correspondence is resolved—an interaction effect that most convincingly justifies match first, then use all patches. The SSC ablation controls the number of crops (Random Scope), separating the benefit of semantic organization from the benefit of simply using more crops.
  4. Ten victims across six families, both closed- and open-source; a 1,000-pair validation plus a complete threshold sweep ([email protected]–0.9). The largest gains under the strictest thresholds are consistent with the interpretation of stronger target alignment rather than more borderline successes.
Critical weaknesses
  1. The offline semantic pipeline depends on a closed-source, heavyweight model, and its robustness is never analyzed. SSC relies on GPT-6-Astra for concept extraction + Grounding DINO for localization + a second GPT-6-Astra review pass, an unexamined external dependency introduced into a "black-box attack" setting. There is no sensitivity analysis of scope quality whatsoever: how much does the attack success rate drop when scopes miss salient objects, have inaccurate boundaries, or come from a different concept-extraction model? The average number of scopes per image (4.41) is an upstream variable of attack performance, yet there is neither failure-case analysis nor quantification.
  2. Fairness of the baseline comparison is questionable. Section 4.1 only states that baselines follow official implementations and recommended configurations, without stating whether they were re-run under the same surrogate ensemble and the same iteration/compute budget as GAST. Each GAST iteration runs 5 encoders × all intermediate layers × one 1024×1024 Hungarian matching per branch, its per-iteration gradient signal and compute far exceed typical baselines, so part of the gain may come from more supervision/more compute rather than the design itself. Appendix A.5 acknowledges the high cost but provides no wall-clock, memory, or per-sample cost comparison against baselines.
  3. Evaluation relies entirely on a single LLM judge. All AvgSim/ASR numbers are produced by GPT-4o, while the victim set includes two GPT-family models, potential same-family judge bias is never controlled (no alternative judge, no human spot-checks). On 100 samples, a few percentage points correspond to 1-3 samples, yet no confidence intervals or repeated-run variance are reported. The abstract's 12% over second-best also does not specify the metric or the averaging protocol.
  4. The most directly related baselines are missing. SGHA-Attack (semantic-guided hierarchical alignment over intermediate global, pooled-patch, and visual-textual objectives, almost identical problem framing) and FRA-Attack (also OT-based patch-feature alignment) are cited but not compared; Omni-Attack and InstructTA are also not compared. The second-best +12% claim against six author-chosen baselines is therefore weakened.
  5. Single-task, permissive setting. All experiments use image captioning limited to 20 words; there is no targeted VQA or instruction-transfer evaluation. The claim of motivating stronger defenses is unsupported by any defense experiments (adversarial training, purification, input transforms).
Questions for authors

Were the six baselines evaluated with their official surrogate models, or re-run under the same 5-encoder ensemble and the same 300-iteration budget as GAST? Please report per-adversarial-example wall-clock time and peak GPU memory for each method.

SSC quality sensitivity: if GPT-6-Astra is replaced by a weaker/different-family concept extractor, or if 10%/25% scope-miss and boundary noise is injected, how much do [email protected] and AvgSim degrade? Is there a correlation analysis between the number of scopes Q and success rate? What do failure cases look like when scopes miss salient objects?

Which model generates the target captions? Has a non-OpenAI judge (open-source MLLM or human annotators) re-scored a subset to rule out same-family judge bias toward GPT victims? What is the resampling variance of GPT-4o scores?

Are confidence intervals or significance tests provided for differences on the 100-pair benchmark (e.g., Gemini-3 97% vs 86%)? Why does the 1,000-pair evaluation cover only seven victims?

Why does semantic-scope supervision underperform M-Attack-v2's multi-source crops + auxiliary target references on Claude Opus 5? Is it a scope-granularity issue, or does this model respond differently to local semantic perturbations? Can you provide a diagnostic experiment?

Why are SGHA-Attack and FRA-Attack not compared? Does the second-best +12% conclusion still hold without them?

4 comments
AuthorsOpenReview ↗

Response to Reviewer NxZM (part1)

We thank the reviewer for the detailed review. Below we respond to the comments in Questions (Q) and Weaknesses (W). The new results on computational cost, concept extractors, and judges have been added to the revised paper (Sections 4.4 and 4.5, Appendices A.5, A.7, and A.8), with all changes marked in red.

Q1 & W2: Surrogate alignment, computational cost, and FLOPs-matched comparisons.

All six baselines are run with their official surrogate models and default settings under the same perturbation budget (ϵ=16/255\epsilon = 16/255). GAST does not use a new ensemble of its own. GAST uses the same five-encoder ensemble as PRAF and MPCAttack, ensuring a direct comparison under identical surrogates.

As requested, we report the running time and peak GPU memory for generating an adversarial example on a single NVIDIA RTX 4090 (including model loading).

MethodIterationsTime (s)Peak GPU memory (GB)
AnyAttack1 forward pass28.00.75
M-Attack30048.48.38
FOA30096.38.41
M-Attack-v2300462.48.30
MPCAttack30048.610.43
PRAF30067.412.09
GAST300746.79.79

Most of GAST's extra runtime comes from exact Hungarian matching solved sequentially on the CPU via SciPy. This matching only pairs adversarial and target patches outside the computation graph, adding no extra loss terms or surrogate passes. Given the pairs, the alignment loss costs the same as aligning patches in a fixed order. The remaining overhead comes from the per-scope branches and the alignment of all intermediate layers. Even so, GAST's peak GPU memory (9.79 GB) stays below that of PRAF (12.09 GB) and MPCAttack (10.43 GB).

Regarding the compute budget, we ran PRAF and MPCAttack for more iterations to match GAST's total FLOPs. Since GAST's cost scales with the number of scopes, we match FLOPs per image pair and list the average iterations below. Each cell reports AvgSim / [email protected] (%) / [email protected] (%) / KMR (%) on the 100-pair benchmark.

MethodIterationsGPT-5.4Gemini-2.5Qwen3.8-MaxAverage
MPCAttack3000.415 / 45 / 25 / 47.90.558 / 69 / 51 / 65.80.593 / 76 / 58 / 74.60.522 / 63.3 / 44.7 / 62.8
MPCAttack (FLOPs-matched)2,2130.431 / 53 / 29 / 53.30.625 / 79 / 62 / 68.30.628 / 84 / 61 / 76.30.561 / 72.0 / 50.7 / 66.0
PRAF3000.553 / 69 / 49 / 63.20.610 / 79 / 61 / 65.70.668 / 87 / 71 / 75.00.610 / 78.3 / 60.3 / 68.0
PRAF (FLOPs-matched)1,5000.611 / 76 / 58 / 64.20.664 / 87 / 68 / 69.40.705 / 90 / 76 / 78.50.660 / 84.3 / 67.3 / 70.7
GAST3000.659 / 86 / 69 / 69.50.677 / 85 / 73 / 72.80.727 / 92 / 83 / 76.60.688 / 87.7 / 75.0 / 72.9

More iterations boost PRAF and MPCAttack by 7.0 and 6.0 points in average [email protected]. However, GAST still outperforms both FLOPs-matched baselines across nearly all victims and metrics, and leads them by at least 5 points in [email protected] on every victim, showing that compute budget alone does not explain the performance gap. We have added both analyses to Section 4.4 (Table 4) and Appendix A.5 (Tables 5 and 6) of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer NxZM (part2)

Q2 & W1: Robustness to alternative concept extractors and role of scope count Q.

We tested two alternative extractors: a weaker same-family model (GPT-6-Luna) and an open-source image tagger from a different family (RAM++). In both cases, extracted concepts are fed directly into Grounding DINO without GPT verification, keeping the average scope count close to the original pipeline (4.50 for RAM++ and 4.49 for GPT-6-Luna, vs. 4.41 originally). All other settings on the 100-pair benchmark remain identical. Each cell reports AvgSim / [email protected] (%) / [email protected] (%) / KMR (%).

MethodGPT-5.4Gemini-2.5Qwen3.8-MaxAverage
PRAF0.553 / 69 / 49 / 63.20.610 / 79 / 61 / 65.70.668 / 87 / 71 / 75.00.610 / 78.3 / 60.3 / 68.0
GAST w/ GPT-6-Astra (paper)0.659 / 86 / 69 / 69.50.677 / 85 / 73 / 72.80.727 / 92 / 83 / 76.60.688 / 87.7 / 75.0 / 72.9
GAST w/ RAM++0.658 / 84 / 69 / 71.10.670 / 87 / 71 / 71.30.727 / 93 / 78 / 78.30.685 / 88.0 / 72.7 / 73.5
GAST w/ GPT-6-Luna0.635 / 83 / 65 / 72.50.690 / 91 / 77 / 73.60.720 / 91 / 81 / 81.70.682 / 88.3 / 74.3 / 75.9

The performance drop is minor: average [email protected] decreases by only 2.3 points with RAM++ and 0.7 points with GPT-6-Luna (and AvgSim by 0.003 and 0.006). Notably, on GPT-5.4, RAM++ achieves the exact same 69% [email protected] as GPT-6-Astra, confirming that GAST does not rely on a specific proprietary extractor. Instead, its gain comes mainly from the semantic scopes of SSC, which stay effective with extractors from different model families. Both variants still beat all baselines on every victim and metric. We have added these results to Section 4.5 (Table 4) and Appendix A.7 (Table 9) of the revised paper.

Regarding the scope count QQ, each scope acts as a local alignment branch, which plays a role similar to the number of local views per iteration in M-Attack-v2 (Fig. 10 of the M-Attack-v2 paper). In GAST, however, QQ is not a free hyperparameter. It reflects the salient semantic units in each target image. Adding extra scopes would mostly introduce redundant overlapping boxes while increasing compute linearly. We therefore keep a compact set of scopes to balance target coverage against efficiency, which is also why we matched QQ in the extractor ablation above. With only 4.41 scopes per image on average (Appendix A.1), GAST already achieves the gains reported in Tables 1 and 2.

Q3 & W3: Reference caption generation and evaluation with non-OpenAI judges.

The target captions are not produced by a single model. For each victim, we use the caption that the victim itself generates for the clean target image as the scoring reference (Section 4.1 and Appendix A.8). This caption is generated once and shared across all attack methods. Thus, all methods are evaluated against the exact same reference, and any stylistic bias toward a victim's own phrasing affects all methods equally.

To rule out GPT-4o judge bias, we rescored the 100-pair benchmark outputs on GPT-5.4, Gemini-3-Flash, Qwen3-VL-32B-Instruct, and Kimi K3 using three non-OpenAI judges: Claude Sonnet 5, Qwen3.7 Flash, and DeepSeek 4.1 Flash. Each cell reports AvgSim / [email protected] (%) / [email protected] (%) / KMR (%).

MethodGPT-4o (paper)Claude Sonnet 5Qwen3.7 FlashDeepSeek 4.1 Flash
PRAF0.608 / 76.3 / 60.3 / 67.90.590 / 71.3 / 39.8 / 78.50.678 / 84.0 / 70.3 / 77.80.688 / 81.8 / 66.3 / 77.8
GAST0.694 / 88.3 / 76.0 / 74.00.670 / 85.5 / 54.3 / 84.10.736 / 90.8 / 83.3 / 82.30.748 / 89.5 / 79.5 / 81.8

Under all four judges, GAST ranks first among the five methods on all four metrics and leads the second-best method, PRAF, by 13.0 to 15.7 points in [email protected]. The gains of GAST therefore reflect a genuine transfer of target semantics rather than a preference of the GPT-4o judge. We have added these results to Section 4.5 (Table 4) and Appendix A.8 (Table 10) of the revised paper.

Q4 & W3: Statistical significance and victim selection.

The gains of GAST are statistically significant. On the 100-pair benchmark, GAST reaches 97% [email protected] on Gemini-3-Flash and PRAF reaches 86% (p = 0.009, Fisher’s exact test). On the 1,000-pair benchmark, GAST exceeds PRAF by 10.5 ± 0.8 points in [email protected] (66.2% vs. 55.7%, mean ± standard error).

The 1,000-pair benchmark evaluates seven victims due to API budget constraints: each additional victim requires ~7,000 vision queries and ~7,000 judge calls. Under four model families (GPT, Gemini, Qwen, and GLM), GAST ranks first at every ASR threshold from 0.3 to 0.8 (Appendix A.6, Tables 7 and 8).

AuthorsOpenReview ↗

Response to Reviewer NxZM (part3)

Q5: Performance analysis and floor effect on Claude Opus 5.

Regarding the comparison with M-Attack-v2 on Claude Opus 5, this model is substantially more robust than other victims across all tested transfer attacks. Every method stays near failure on it ([email protected] ≤14%\le 14\% and AvgSim ≤0.185\le 0.185), whereas GAST achieves ≥76%\ge 76\% [email protected] on all other nine victims. Prior work reports the same pattern, with far lower attack success rates on Claude than on other commercial models [1, 2]. Hu et al. [3] also find Claude models significantly more robust than other victims. They suggest that the visual training of Claude may differ substantially from that of public vision-language models, from which the surrogate encoders are drawn. The small differences among methods on Claude Opus 5 therefore reflect a floor effect shared by all methods rather than a limitation of semantic scope supervision. Even on this model, GAST obtains the highest KMR among all methods (Table 1).

[1] X. Jia et al. Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment. NeurIPS 2025.

[2] Y. Li et al. Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models. CVPR 2026.

[3] K. Hu et al. Transferable Adversarial Attacks on Black-Box Vision-Language Models. arXiv:2505.01050, 2025.

Q6 & W4: Comparisons with other recent attack methods.

Both SGHA-Attack and FRA-Attack were only preprints on arXiv at the time of submission. Among the four methods in W4, only FRA-Attack shares our exact task setting (and had no public code). SGHA-Attack uses text targets, Omni-Attack targets VQA/OCR, and InstructTA assumes grey-box access to the victim's visual encoder.

As suggested, we adapted all four methods to image-guided targeted captioning and evaluated them on the 100-pair benchmark under GAST's exact resolution, iteration count, and perturbation budget (ϵ=16/255\epsilon = 16/255). Each cell reports AvgSim / [email protected] (%) / [email protected] (%) / KMR (%).

MethodGPT-5.4Gemini-2.5Qwen3.8-MaxAverage
InstructTA0.016 / 0 / 0 / 3.80.020 / 0 / 0 / 3.90.023 / 0 / 0 / 4.70.020 / 0.0 / 0.0 / 4.1
SGHA-Attack0.037 / 0 / 0 / 7.30.055 / 0 / 0 / 8.60.060 / 3 / 1 / 9.80.051 / 1.0 / 0.3 / 8.6
Omni-Attack0.368 / 37 / 20 / 43.10.381 / 36 / 23 / 43.30.373 / 40 / 20 / 48.80.374 / 37.7 / 21.0 / 45.1
FRA-Attack0.288 / 27 / 13 / 33.90.345 / 35 / 19 / 39.90.525 / 63 / 43 / 62.20.386 / 41.7 / 25.0 / 45.3
PRAF0.553 / 69 / 49 / 63.20.610 / 79 / 61 / 65.70.668 / 87 / 71 / 75.00.610 / 78.3 / 60.3 / 68.0
GAST0.659 / 86 / 69 / 69.50.677 / 85 / 73 / 72.80.727 / 92 / 83 / 76.60.688 / 87.7 / 75.0 / 72.9

GAST outperforms all four methods by a large margin on every victim. The strongest of them, FRA-Attack, reaches 25.0% average [email protected], compared with 75.0% for GAST. InstructTA scores close to zero because it relies on access to the victim’s visual encoder, which our victims do not expose. SGHA-Attack also scores low, although its adversarial images do move closer to the targets in the feature space of all three surrogates. The results reported by its authors on commercial models come from SGHA combined with the M-Attack framework, whereas we evaluate SGHA on its own. Since none of the four methods surpasses the strongest baseline in the paper, the +12% gain in the abstract still holds. This gain refers to the average [email protected] over the ten victims on the 100-pair benchmark (68.7% vs. 56.7%).

W5: Task scope.

We follow the evaluation protocol of M-Attack and FOA, which also evaluate targeted image captioning, so that the comparison with these baselines is direct. The attack itself is not tied to captioning. GAST aligns the image features with the target image without using any task prompt, so the perturbation changes what the victim perceives in the image rather than how it answers a particular question. The same adversarial image can therefore be queried with any prompt.

Summary. We have made the following revisions in response to the reviewer's comments.

  1. Section 4.4 and Appendix A.5: running time, peak GPU memory, and comparison with FLOPs-matched baselines.
  2. Section 4.5 and Appendix A.7: GAST with two alternative concept extractors.
  3. Section 4.5 and Appendix A.8: rescoring with three judges from other model families.

Across these analyses and the four additional methods in Q6 & W4, GAST keeps a clear lead over every competing method. Its gains therefore come from semantic-scope supervision rather than from extra computation, a specific extractor, or the choice of judge.

We hope these responses address the reviewer’s concerns, and we would be glad to provide further details if needed.

AuthorsOpenReview ↗

Looking Forward to Further Discussion

Dear Reviewer NxZM,

We hope this message finds you well. As the discussion period ends soon, we wanted to kindly check whether our responses have addressed your concerns, or if there is anything else you would like us to clarify. We would be glad to discuss further. Thank you again for your feedback and time.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Current Rebuttal Status

Dear Reviewers, ACs, SACs, and PCs,

We thank all reviewers for their feedback and discussion. We have replied to each question and added new experiments. All new results are included in the revised paper and marked in red.

Below is a brief summary for each reviewer.

Reviewer d1Ut (most concerns addressed, positive score maintained)

Reviewer d1Ut said that our response addressed most of the concerns and kept the positive score. We thank the reviewer for the support.

  • GPT-family bias: We replaced the annotator and the judge with non-GPT models. GAST still ranks first. (response to Q)
  • Hyperparameters: We added a sensitivity study. The default setting works best, and GAST is not sensitive to the loss weights. (response to W1)
  • Runtime: We added time and memory results. GAST still leads when baselines are given the same compute. (response to W2)

Reviewer n5HN (soundness raised from 2 to 3)

Reviewer n5HN agreed that our results resolved the concerns on compute fairness and judge bias, and raised the soundness score. We believe the remaining points come from some misunderstandings.

  • RAM++ result: The RAM++ variant still uses SSC. It only changes the tagger, and each box is still a semantic scope. When we remove the semantics and keep the same boxes (Random Scope), the performance drops much more. So the gain comes from semantics, not from detector boxes.
  • Novelty: Aligning all patches helps only after patches are matched within the same semantic scope. This shows that the key is to organize supervision around target semantics, which is not a simple combination of existing modules.
  • Runtime: The slow speed was caused by running Hungarian matching on the CPU one by one. After parallelizing it, GAST takes 212 s per image and is now faster than M-Attack-v2.
  • Claude Opus 5: All methods almost fail on this model, and the small gap with M-Attack-v2 is not significant. M-Attack-v2 also uses three times as many target images.

Reviewer NxZM (no reply yet)

We replied to all of Reviewer NxZM's questions and sent a follow-up, but the reviewer has not joined the discussion. Most of these concerns are the same as those raised by n5HN and d1Ut, who have confirmed that they are resolved.

  • Compute fairness: GAST uses the same surrogates as PRAF and MPCAttack, and still leads when they run with the same FLOPs. (response to Q1 & W2)
  • SSC dependency: Using RAM++ or a weaker GPT model only causes a small drop. (response to Q2 & W1)
  • Judge bias: Three non-OpenAI judges give the same ranking. (response to Q3 & W3)
  • Significance: The gains are statistically significant and hold on the 1,000-pair benchmark. (response to Q4)
  • Claude Opus 5: All methods perform poorly on this model. (response to Q5)
  • More baselines: We added four more methods, and GAST outperforms all of them by a large margin. (response to Q6 & W4)
  • Task scope: We follow the protocol of M-Attack and FOA, and GAST does not depend on a specific prompt. (response to W5)

Thank you again for your time and effort in reviewing our work.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.