acceptodds
Under review as a conference paper at ICLR 2027

Enhancing Transferable Targeted Attacks on Closed-Source MLLMs with Progressive Semantic Chain

Abstract

Transfer-based targeted attacks on commercial black-box multimodal large language models (MLLMs) remain challenging because attackers can access only the model’s inputs and outputs. Existing methods primarily rely on visual semantic alignment in the embedding space to guide adversarial example generation, but the resulting perturbations often lack sufficiently explicit semantic cues, which limits their transferability. In contrast, textual descriptions can provide more explicit and fine-grained semantic cues for adversarial optimization. Motivated by this insight, we propose Progressive Semantic Chain Attack (PSC-Attack), which introduces a semantic chain to provide explicit semantic guidance for adversarial example generation. Specifically, PSC-Attack decomposes the target semantics into a sequence of concise semantic nodes that capture explicit semantic cues and progressively guide the optimization in the shared vision-language embedding space. Furthermore, we introduce a Minimum-Redundancy Semantic Chain to adaptively select more explicit and fine-grained target semantic nodes while suppressing redundant guidance, thereby enhancing attack transferability. Extensive experiments on closed-source commercial MLLMs demonstrate that PSC-Attack consistently outperforms existing targeted transferable attack methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
MLLMs, Transferable Targeted Attacks, Semantic Chain

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1ngWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PSC-Attack for targeted transfer attacks against black-box multimodal large language models. It augments target-image feature alignment with textual guidance organized into a progressively weighted semantic chain. The authors first introduce a predefined chain and then propose a Minimum-Redundancy Semantic Chain (MRSC), which greedily selects semantic units using target relevance and redundancy. Experiments report improved attack success rates and semantic similarity across open-source, closed-source, and reasoning-oriented models.

Critical strengths
  1. Organizing target descriptions into explicit semantic components is a reasonable direction for investigating how textual supervision affects adversarial transfer.

  2. The evaluation covers multiple model families and includes an additional 1,000-image experiment. The ablations on the image/text trade-off, redundancy penalty, and perturbation budget provide useful information about the method’s behavior.

Critical weaknesses
  1. Some details are missing. Section 3.3 does not explain how the core object unit in Eq. (11) is identified, or how Compose in Eq. (12) converts selected units into a description. ClipCap is identified as the caption generator, but this does not specify these subsequent operations. The reference image encoder in Eq. (14) is also not identified. These are central algorithmic choices because they determine the selected units, their scores, and the resulting chain.
    In addition, Eqs. (10)–(19) construct MRSC directly from the target description without using the predefined chain from Section 3.2. The paper should make clear that these are alternative constructions, if that is the intended design, and identify which variant is evaluated in each experiment.

  2. Table 1 does not precisely define the baseline and holistic textual guidance conditions or their shared optimization settings. The predefined chain in Table 1 obtains ASR = 0.89 and AvgSim = 0.648 on GPT-4o, exactly matching the final PSC-Attack results in Table 2. This requires clarification: under the same setup, these results do not show an additional benefit from MRSC; under different setups, the differences must be specified.

  3. The redundancy-penalty ablation is helpful but does not replace a direct comparison of predefined and minimum-redundancy chains. Moreover, there is no controlled comparison between progressive weighting and fixed uniform weighting over the same nodes. The predefined and MRSC constructions also differ in whether descriptions accumulate selected units. Consequently, the results do not establish whether the gains come from progressive scheduling, adaptive selection, cumulative descriptions, or additional textual supervision.

  4. The experiments evaluate transferability across different target models but do not assess attack effectiveness under explicitly applied defenses or defensive preprocessing, such as JPEG compression, denoising, or image purification. Evaluating models with lower attack success rates does not substitute for a controlled defense evaluation. Consequently, it remains unclear whether PSC-Attack’s improvements over existing methods persist when the adversarial perturbations are processed by a defense.

  5. Section 4.3 describes Figure 2 as showing adversarial examples and corresponding perturbation maps, but the figure contains two different source-image examples and their attacked versions, without separate perturbation maps. It therefore does not directly support the comparison of perturbation intensity. Besides, the claims that AdvDiffVLM produces more visible color distortions and that PSC-Attack produces smoother perturbations than M-Attack are also not clearly established by these examples.

Questions for authors
  1. Appendix A.5 acknowledges additional computation, but the paper does not quantify chain-construction time, per-image attack runtime, or overhead relative to the baselines. Such measurements would help assess the performance-cost trade-off.
  2. Under matched settings, how does MRSC compare with the predefined chain, and how does progressive weighting compare with fixed uniform weighting over the same nodes? A node-order control would further clarify whether the proposed ordering is important.
Reviewer_KuW3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PSC-Attack, which uses structured textual guidance to improve targeted transfer attacks on MLLMs. It decomposes a target-image description into semantic units covering objects, attributes, relations, and scene context. A Minimum-Redundancy Semantic Chain greedily selects units based on image–text relevance gains and semantic redundancy, forming progressively expanded textual nodes. The attack jointly optimizes image-level and textual alignment across three CLIP surrogates, using time-dependent weights to shift emphasis from earlier to later nodes. Experiments cover open-source, commercial closed-source, and reasoning-oriented MLLMs, with an additional 1,000-image comparison, and report improvements in attack success and target-semantic similarity.

Critical strengths
  1. The comparative evaluation has broad coverage. The paper evaluates twelve victim models spanning multiple open-source families, commercial closed-source models, and reasoning-oriented models, supplemented by a 1,000-image comparison with M-Attack. Keyword coverage, parameter sensitivity, and perturbation-budget analyses provide additional evidence beyond attack success and semantic similarity.

  2. The motivation is clear, and the semantic-guidance design is intuitive. Starting from the concern that holistic visual or textual representations may obscure fine-grained target semantics, the method decomposes target descriptions into explicit semantic units and constructs guidance chains using relevance and redundancy criteria. Comparisons with holistic descriptions and attention visualizations provide some empirical support for structured textual guidance.

Critical weaknesses
  1. The main quantitative evaluation concentrates on earlier victim versions, with insufficient evidence on newer models. Both the main tables and the 1,000-image evaluation primarily use models such as GPT-4o, GPT-4.1, and Gemini-2.0. Although the appendix presents successful examples on GPT-5.0, Opus-4.6/4.7, and Sonnet-4.6, it lacks aggregate results and baseline comparisons under matched samples and attack budgets. Whether the reported advantages consistently transfer to these newer models therefore remains unclear.

  2. The ablations do not adequately separate progressive scheduling from structured textual guidance. Table 1 compares holistic descriptions with semantic chains, while Tables 4 and 5 examine loss weighting and redundancy penalties. However, none compares the proposed schedule with fixed uniform weights, shuffled order, or reversed order while preserving the semantic nodes. The improvements therefore cannot be clearly attributed to progressive ordering and temporal scheduling rather than more explicit textual representations.

Questions for authors
  1. Could the authors report aggregate ASR and AvgSim for PSC-Attack and the main baselines on the newer models shown in the appendix, using the same source–target pairs and attack budgets, beyond individual successful examples?

  2. Could the authors hold the semantic nodes, surrogate models, and optimization budget fixed, and compare the current schedule with fixed uniform weighting, shuffled order, and reversed order to isolate the contribution of progressive ordering?

Reviewer_vaLLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Progressive Semantic Chain Attack (PSC-Attack), a targeted transferable adversarial attack against black-box multimodal large language models (MLLMs). The method aims to improve cross-model transferability by introducing explicit semantic guidance into adversarial perturbation optimization. Rather than relying solely on holistic image-text alignment, PSC-Attack decomposes the target description into a sequence of semantic nodes and progressively adjusts their contributions during optimization. The authors further introduce a Minimum-Redundancy Semantic Chain (MRSC) mechanism that adaptively selects informative and complementary semantic units to construct target-specific guidance.

Critical strengths
  1. Progressive Semantic-Chain Guidance. The paper introduces a progressive semantic-chain-based guidance strategy that dynamically shifts the optimization focus across different target-semantic nodes, rather than relying solely on holistic semantic alignment. This provides more explicit and structured target-semantic guidance during adversarial optimization, offering a meaningful approach to improving the transferability of targeted attacks.

  2. Minimum-Redundancy Semantic Chain Construction. The proposed MRSC adaptively constructs fine-grained semantic chains by selecting informative and semantically distinct semantic units, rather than relying on a fixed, predefined semantic decomposition. This target-adaptive strategy is designed to reduce redundant semantic guidance and provide a more flexible representation of target semantics, offering a potentially useful extension to progressive semantic-chain guidance.

Critical weaknesses
  1. Insufficient Ablation Studies to Validate the Key Contributions. Although the proposed method achieves consistent improvements over existing baselines, the experiments do not adequately isolate the effectiveness of its two main contributions. First, progressive semantic weighting is not compared against fixed or uniform weighting. Table 1 does not disentangle the effects of semantic decomposition and progressive weighting, making it unclear whether progressive scheduling provides additional benefits beyond decomposition alone. Second, MRSC is not directly compared with the predefined semantic chain. While the β ablation provides some evidence for the effectiveness of redundancy penalization, it does not establish the advantage of adaptive semantic chain construction over predefined decomposition. Notably, the predefined-chain result in Table 1 (0.89/0.648) is identical to the main PSC-Attack result on GPT-4o, but the relationship between these configurations is not clarified. Consequently, although the overall performance improvements are promising, the individual contributions of the proposed components remain insufficiently validated.

  2. Incremental Methodological Contributions. The proposed approach builds upon established techniques, including text-guided adversarial optimization and feature alignment for targeted attacks. Its main methodological contributions lie in combining progressive weighting of decomposed semantic guidance with redundancy-aware semantic unit selection. While this integration provides a meaningful extension to existing approaches, the individual components appear relatively straightforward, and the degree of methodological novelty remains somewhat limited. Stronger experimental evidence isolating the benefits of these components would help better establish the significance of the proposed contributions.

Questions for authors

Q1. Effectiveness of Progressive Semantic Weighting. The current experiments do not clearly isolate the contribution of progressive semantic weighting from that of semantic decomposition. Could the authors provide further evidence demonstrating whether progressive weighting offers additional benefits over non-progressive semantic guidance? It would also be helpful to clarify the sensitivity of the method to the transition parameter tau.

Q2. Independent Contribution of MRSC and Experimental Consistency. Could the authors directly compare MRSC with the predefined semantic chain under identical optimization settings? Although Table 5 examines the effect of the redundancy penalty coefficient beta, it does not isolate the benefit of adaptive semantic chain construction. Furthermore, the predefined-chain result in Table 1 (0.89/0.648) is identical to the full PSC-Attack result on GPT-4o in Table 2. Were these experiments conducted using the same 100-image subset, random seed, and evaluation settings? If so, could the authors clarify the relationship between these two configurations and whether MRSC yields additional gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.