- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RefBind, a target-reference-guided object-level semantic replacement method for vision-language models. Given a source image and a target reference image, RefBind redirects the model's understanding of the designated source object toward the target concept under a perturbation constraint. Methodologically, it first uses CVR to localize the semantic support regions of the source and target objects, then adopts spatially decoupled optimization, and complements final-level visual alignment through task-compatible intermediate layer selection. Experimental results show that the proposed method outperforms baseline methods.
Critical strengths
-
The paper is well written and easy to follow;
-
The baseline methods are reasonably selected and are all relatively strong baselines;
-
The attacked models are relatively advanced and possess a certain exploratory value.
Critical weaknesses
-
The real-world effectiveness of this paper has not been sufficiently demonstrated;
-
The metrics in the paper are too messy. The authors should carefully read the adversarial attack literature and select conventional, reasonable metrics to evaluate the effectiveness of the proposed method. In general adversarial attack papers, ASR and imperceptibility are chosen as metrics, so what is the significance of metrics such as "S/P/F" in your paper?
-
The adversarial examples generated by the authors' method have perturbations that are almost visible to the naked eye. Why does the paper not include an imperceptibility metric?
-
The second paragraph of the Introduction needs to add corresponding citations to support the authors' claims.
Questions for authors
See Weaknesses.
7 comments
W3 Perturbation Magnitude and Perceptual Quality & W4 Citations in the Introduction
W3 Perturbation Magnitude and Perceptual Quality
(1) Evaluation Protocol.
We adopt four complementary metrics: normalized and distances to measure pixel-level perturbation magnitude, MS-SSIM to assess multi-scale structural similarity, and LPIPS to evaluate perceptual similarity. Lower values are better for , , and LPIPS, while higher values are better for MS-SSIM. We compare RefBind with the closely related object-level attack V-Attack on COCO300 (300 cases), evaluating adversarial images against their corresponding clean inputs under both single- and ensemble-surrogate settings.
(2) Quantitative Results.
The results are summarized in Table 1.
| Method | L1 ↓ | L2 ↓ | MS-SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| V-Attack | 0.02676 | 0.03276 | 0.94839 | 0.19685 |
| RefBind | 0.02938 | 0.03534 | 0.95018 | 0.16959 |
| V-Attack-Ens | 0.03317 | 0.03889 | 0.93896 | 0.22968 |
| RefBind-Ens | 0.03330 | 0.03901 | 0.94131 | 0.22962 |
Table 1. Comparison of perturbation magnitude and perceptual similarity between V-Attack and RefBind on COCO300 (300 cases) under single- and ensemble-surrogate settings. Arrows indicate the preferred direction for each metric.
Under the single-surrogate setting, RefBind achieves higher MS-SSIM (0.95018 vs. 0.94839) and lower LPIPS (0.16959 vs. 0.19685) than V-Attack, while maintaining comparable pixel-level perturbation magnitudes. Under the ensemble setting, the two methods exhibit nearly identical , , and LPIPS values, with RefBind achieving slightly higher MS-SSIM (0.94131 vs. 0.93896). Overall, these results indicate that RefBind maintains visual quality comparable to V-Attack under both attack settings.
W4: Citations in the Introduction
The second paragraph of the Introduction needs to add corresponding citations to support the authors' claims.
Response Summary
We have added relevant citations to the second paragraph of the Introduction to better contextualize the technical challenges addressed in this work and clarify their connections to prior research. The revised paragraph is as follows:
Extending target references from image-level semantic transfer (Li et al., 2026c; Jia et al., 2026) to object-level replacement is non-trivial: target visual evidence must first be associated with the designated source object, and then induce the target semantics while suppressing the source object and limiting interference with unrelated scene content (Wang et al., 2026b; Cao et al., 2026; Nie et al., 2026). Prior studies have also highlighted the importance of visual representation choices in targeted attacks, including Value features and intermediate-projector representations (Nie et al., 2026; Cao et al., 2026). This raises two related questions: how textual and reference-visual supervision should be realized over pixel space, and which representation levels should incorporate reference-visual evidence. The central challenge is therefore to coordinate the two forms of supervision across spatial support and representation hierarchy.
REFERENCES
[1] Zhaoyi Li, Xiaohan Zhao, Dong-Dong Wu, Jiacheng Cui, and Zhiqiang Shen. A frustratingly simple yet highly effective attack baseline: Over 90% success rate against the strong black-box models of GPT-4.5/4o/o1. Advances in Neural Information Processing Systems, 38:45018–45051, 2026.
[2] Xiaojun Jia, Sensen Gao, Simeng Qin, Tianyu Pang, Chao Du, Yihao Huang, Xinfeng Li, Yiming Li, Bo Li, and Yang Liu. Adversarial attacks against closed-source MLLMs via feature optimal alignment. Advances in Neural Information Processing Systems, 38:57993–58028, 2026.
[3] Yichen Wang, Hangtao Zhang, Hewen Pan, Ziqi Zhou, Xianlong Wang, Peijin Guo, Lulu Xue, Shengshan Hu, Minghui Li, and Leo Yu Zhang. ADVEDM: Fine-grained adversarial attack against VLM-based embodied agents. Advances in Neural Information Processing Systems, 38:136551–136575, 2026.
[4] Yiming Cao, Yanjie Li, Kaisheng Liang, and Bin Xiao. Enhancing targeted adversarial attacks on large vision-language models via intermediate projector. IEEE Transactions on Information Forensics and Security, 21:5946–5959, 2026. doi: 10.1109/TIFS.2026.3702560.
[5] Sen Nie, Jie Zhang, Jianxin Yan, et al. V-Attack: Targeting disentangled value features for controllable adversarial attacks on LVLMs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 42257–42267, 2026.
Thank you for the authors' careful response. The last issue regarding citations has been resolved, but the other issues may not have been addressed systematically:
-
The stealthiness evaluation should compare against all baseline methods in the paper, rather than only one of them;
-
I believe the physical realizability of the attack is an important aspect, but it has not been well addressed.
Based on these issues, I will tentatively maintain my current score.
W1 Real-world Attack Effectiveness & W2 Evaluation Metrics and Their Significance
We thank the reviewer for the thoughtful comments and constructive suggestions.
W1: Real-world Attack Effectiveness
(1) Real-world Security Motivation. Existing image-level targeted attacks primarily aim to shift the overall semantics of adversarial images toward predefined targets. As highlighted by ADVEDM, excessive changes to scene semantics may create inconsistencies between visual inputs and task instructions, leading to invalid rather than actionable but incorrect decisions. Fine-grained attacks therefore aim to manipulate designated objects while preserving non-target scene information, potentially influencing subsequent reasoning and decision-making.
(2) Distinction from ADVEDM. ADVEDM-R investigates the semantic removal of designated objects, whereas ADVEDM-A manually selects contiguous patches in background or object-sparse regions to inject new object semantics using target reference images. In contrast, RefBind targets strict source-to-target replacement of existing objects by jointly leveraging textual and reference-visual supervision while minimizing interference with unrelated scene content. This objective is directly relevant to applications relying on object identity for reasoning and task planning.
W2: Evaluation Metrics and Their Significance
Response
RefBind adopts the original ASR evaluation protocol of V-Attack for CAP/VQA tasks. We additionally report S/P/F outcomes to distinguish complete replacement from incomplete semantic manipulation. For robotic manipulation scenarios, VLM-PTR further evaluates strict object replacement while accounting for scene validity.
Details
(1) Consistency with the established ASR protocol.
To ensure comparability with existing attack methods, we follow the original CAP/VQA evaluation protocol of V-Attack in Tables 1–2, which aggregates three-level semantic scores (1, 0.5, and 0) into ASR.
(2) Significance of S/P/F for strict object replacement.
The aggregated ASR alone cannot fully characterize strict source-to-target replacement. Specifically, S/P/F correspond to full success (score 1), partial success (score 0.5), and failure (score 0), respectively. For example, in Table 1, SSA-CWA achieves an ASR of 39.50% on InternVL2 without a single full-success case among 300 cases. In contrast, RefBind-Sgl achieves an ASR of 56.17%, with 115 cases achieving complete replacement. We therefore additionally report the S/P/F outcome distribution to distinguish complete replacement from intermediate-scored outcomes and failures. These are the underlying outcomes of the original scoring protocol, rather than additional independent metrics, and make strict replacement performance more transparent.
(3) Task-oriented evaluation with scene validity.
CAP/VQA evaluates semantic manipulation through model-generated text but does not independently assess task-scene validity. We therefore introduce VLM-PTR on RoboCasa365 to jointly evaluate basic robotic-scene validity and strict object replacement.
These results demonstrate that RefBind achieves higher strict object replacement rates even when basic robotic-scene validity is incorporated into the success criterion.