acceptodds
Under review as a conference paper at ICLR 2027

RefBind: Reference-Guided Object-Level Semantic Replacement in VLMs

Abstract

Targeted attacks on vision–language models can induce a target concept without fully replacing the source object. We study reference-guided source-to-target semantic replacement, in which an attacker perturbs a source image while limiting interference with unrelated scene content. Building on disentangled Value features, RefBind extracts source- and target-object semantic supports and combines textual redirection with reference-visual alignment. Textual supervision operates over a broader spatial scope, whereas reference-visual supervision is restricted to the source-object support. A task-compatible intermediate representation complements final-level visual alignment. Controlled spatial ablations show that this asymmetric allocation improves replacement relative to the tested alternatives. On COCO300, RefBind increases the macro-average judge-assessed full-success rate over V-Attack from 5.58% to 37.33% for captioning and from 8.11% to 38.28% for VQA under the single-surrogate setting. On the 150-case robotic-scene benchmark, the macro strict source-to-target replacement rate increases from 5.47% to 28.40%. These results characterize a transferable vulnerability in multimodal semantic interpretation and planning-related VLM outputs, while the effectiveness of such attacks in physical-world robotic execution remains to be validated.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Controllable Adversarial Attacks, Object-Level Semantic Replacement, Reference-Guided Attacks, Adversarial Transferability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QGkAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RefBind, a target-reference-guided object-level semantic replacement method for vision-language models. Given a source image and a target reference image, RefBind redirects the model's understanding of the designated source object toward the target concept under a perturbation constraint. Methodologically, it first uses CVR to localize the semantic support regions of the source and target objects, then adopts spatially decoupled optimization, and complements final-level visual alignment through task-compatible intermediate layer selection. Experimental results show that the proposed method outperforms baseline methods.

Critical strengths
  1. The paper is well written and easy to follow;

  2. The baseline methods are reasonably selected and are all relatively strong baselines;

  3. The attacked models are relatively advanced and possess a certain exploratory value.

Critical weaknesses
  1. The real-world effectiveness of this paper has not been sufficiently demonstrated;

  2. The metrics in the paper are too messy. The authors should carefully read the adversarial attack literature and select conventional, reasonable metrics to evaluate the effectiveness of the proposed method. In general adversarial attack papers, ASR and imperceptibility are chosen as metrics, so what is the significance of metrics such as "S/P/F" in your paper?

  3. The adversarial examples generated by the authors' method have perturbations that are almost visible to the naked eye. Why does the paper not include an imperceptibility metric?

  4. The second paragraph of the Introduction needs to add corresponding citations to support the authors' claims.

Questions for authors

See Weaknesses.

7 comments
AuthorsOpenReview ↗

W1 Real-world Attack Effectiveness & W2 Evaluation Metrics and Their Significance

We thank the reviewer for the thoughtful comments and constructive suggestions.

W1: Real-world Attack Effectiveness

(1) Real-world Security Motivation. Existing image-level targeted attacks primarily aim to shift the overall semantics of adversarial images toward predefined targets. As highlighted by ADVEDM, excessive changes to scene semantics may create inconsistencies between visual inputs and task instructions, leading to invalid rather than actionable but incorrect decisions. Fine-grained attacks therefore aim to manipulate designated objects while preserving non-target scene information, potentially influencing subsequent reasoning and decision-making.

(2) Distinction from ADVEDM. ADVEDM-R investigates the semantic removal of designated objects, whereas ADVEDM-A manually selects contiguous patches in background or object-sparse regions to inject new object semantics using target reference images. In contrast, RefBind targets strict source-to-target replacement of existing objects by jointly leveraging textual and reference-visual supervision while minimizing interference with unrelated scene content. This objective is directly relevant to applications relying on object identity for reasoning and task planning.

W2: Evaluation Metrics and Their Significance

Response

RefBind adopts the original ASR evaluation protocol of V-Attack for CAP/VQA tasks. We additionally report S/P/F outcomes to distinguish complete replacement from incomplete semantic manipulation. For robotic manipulation scenarios, VLM-PTR further evaluates strict object replacement while accounting for scene validity.

Details

(1) Consistency with the established ASR protocol.

To ensure comparability with existing attack methods, we follow the original CAP/VQA evaluation protocol of V-Attack in Tables 1–2, which aggregates three-level semantic scores (1, 0.5, and 0) into ASR.

(2) Significance of S/P/F for strict object replacement.

The aggregated ASR alone cannot fully characterize strict source-to-target replacement. Specifically, S/P/F correspond to full success (score 1), partial success (score 0.5), and failure (score 0), respectively. For example, in Table 1, SSA-CWA achieves an ASR of 39.50% on InternVL2 without a single full-success case among 300 cases. In contrast, RefBind-Sgl achieves an ASR of 56.17%, with 115 cases achieving complete replacement. We therefore additionally report the S/P/F outcome distribution to distinguish complete replacement from intermediate-scored outcomes and failures. These are the underlying outcomes of the original scoring protocol, rather than additional independent metrics, and make strict replacement performance more transparent.

(3) Task-oriented evaluation with scene validity.

CAP/VQA evaluates semantic manipulation through model-generated text but does not independently assess task-scene validity. We therefore introduce VLM-PTR on RoboCasa365 to jointly evaluate basic robotic-scene validity and strict object replacement.

  • Primary metric: Strict-All measures the proportion of all cases achieving strict replacement under the scene-validity requirement.
  • Complementary metrics: E-VORS measures graded replacement quality, while SVR, TER, SER, and STRR characterize scene validity, target emergence, source retention, and conditional strict replacement, respectively.
  • Experimental evidence: As shown in Table 3, RefBind achieves Strict-All scores of 28.40% and 36.40% under Single and Ensemble settings, respectively, compared with 5.47% and 22.00% for V-Attack.

These results demonstrate that RefBind achieves higher strict object replacement rates even when basic robotic-scene validity is incorporated into the success criterion.

AuthorsOpenReview ↗

W3 Perturbation Magnitude and Perceptual Quality & W4 Citations in the Introduction

W3 Perturbation Magnitude and Perceptual Quality

(1) Evaluation Protocol.

We adopt four complementary metrics: normalized L1L_1 and L2L_2 distances to measure pixel-level perturbation magnitude, MS-SSIM to assess multi-scale structural similarity, and LPIPS to evaluate perceptual similarity. Lower values are better for L1L_1, L2L_2, and LPIPS, while higher values are better for MS-SSIM. We compare RefBind with the closely related object-level attack V-Attack on COCO300 (300 cases), evaluating adversarial images against their corresponding clean inputs under both single- and ensemble-surrogate settings.

(2) Quantitative Results.

The results are summarized in Table 1.

MethodL1 ↓L2 ↓MS-SSIM ↑LPIPS ↓
V-Attack0.026760.032760.948390.19685
RefBind0.029380.035340.950180.16959
V-Attack-Ens0.033170.038890.938960.22968
RefBind-Ens0.033300.039010.941310.22962

Table 1. Comparison of perturbation magnitude and perceptual similarity between V-Attack and RefBind on COCO300 (300 cases) under single- and ensemble-surrogate settings. Arrows indicate the preferred direction for each metric.

Under the single-surrogate setting, RefBind achieves higher MS-SSIM (0.95018 vs. 0.94839) and lower LPIPS (0.16959 vs. 0.19685) than V-Attack, while maintaining comparable pixel-level perturbation magnitudes. Under the ensemble setting, the two methods exhibit nearly identical L1L_1, L2L_2, and LPIPS values, with RefBind achieving slightly higher MS-SSIM (0.94131 vs. 0.93896). Overall, these results indicate that RefBind maintains visual quality comparable to V-Attack under both attack settings.

W4: Citations in the Introduction

The second paragraph of the Introduction needs to add corresponding citations to support the authors' claims.

Response Summary

We have added relevant citations to the second paragraph of the Introduction to better contextualize the technical challenges addressed in this work and clarify their connections to prior research. The revised paragraph is as follows:

Extending target references from image-level semantic transfer (Li et al., 2026c; Jia et al., 2026) to object-level replacement is non-trivial: target visual evidence must first be associated with the designated source object, and then induce the target semantics while suppressing the source object and limiting interference with unrelated scene content (Wang et al., 2026b; Cao et al., 2026; Nie et al., 2026). Prior studies have also highlighted the importance of visual representation choices in targeted attacks, including Value features and intermediate-projector representations (Nie et al., 2026; Cao et al., 2026). This raises two related questions: how textual and reference-visual supervision should be realized over pixel space, and which representation levels should incorporate reference-visual evidence. The central challenge is therefore to coordinate the two forms of supervision across spatial support and representation hierarchy.

REFERENCES

[1] Zhaoyi Li, Xiaohan Zhao, Dong-Dong Wu, Jiacheng Cui, and Zhiqiang Shen. A frustratingly simple yet highly effective attack baseline: Over 90% success rate against the strong black-box models of GPT-4.5/4o/o1. Advances in Neural Information Processing Systems, 38:45018–45051, 2026.

[2] Xiaojun Jia, Sensen Gao, Simeng Qin, Tianyu Pang, Chao Du, Yihao Huang, Xinfeng Li, Yiming Li, Bo Li, and Yang Liu. Adversarial attacks against closed-source MLLMs via feature optimal alignment. Advances in Neural Information Processing Systems, 38:57993–58028, 2026.

[3] Yichen Wang, Hangtao Zhang, Hewen Pan, Ziqi Zhou, Xianlong Wang, Peijin Guo, Lulu Xue, Shengshan Hu, Minghui Li, and Leo Yu Zhang. ADVEDM: Fine-grained adversarial attack against VLM-based embodied agents. Advances in Neural Information Processing Systems, 38:136551–136575, 2026.

[4] Yiming Cao, Yanjie Li, Kaisheng Liang, and Bin Xiao. Enhancing targeted adversarial attacks on large vision-language models via intermediate projector. IEEE Transactions on Information Forensics and Security, 21:5946–5959, 2026. doi: 10.1109/TIFS.2026.3702560.

[5] Sen Nie, Jie Zhang, Jianxin Yan, et al. V-Attack: Targeting disentangled value features for controllable adversarial attacks on LVLMs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 42257–42267, 2026.

Thank you for the authors' careful response. The last issue regarding citations has been resolved, but the other issues may not have been addressed systematically:

  1. The stealthiness evaluation should compare against all baseline methods in the paper, rather than only one of them;

  2. I believe the physical realizability of the attack is an important aspect, but it has not been well addressed.

Based on these issues, I will tentatively maintain my current score.

Reviewer_HDG4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RefBind, a transfer-based targeted attack on VLMs that makes the model perceive a designated object in an image as a different target object (e.g., a carrot as a bottle) while keeping the rest of the scene intact. Compared with V-Attack, which only uses text guidance, RefBind additionally uses a reference image of the target object. The source and target objects are localized with connected Value regions; the text gradient is applied over the whole image, while the reference-visual gradients are restricted to the source object region; and an intermediate layer is selected for additional visual alignment. Experiments on COCO300 (captioning and VQA) and a new RoboCasa-based planning benchmark show clear gains over V-Attack, including on GPT-4o, Gemini and Claude.

Critical strengths
  1. The evaluation is thorough. Reporting full/partial/failure counts instead of only ASR makes it clear that the improvement comes from complete replacements. The VLM-PTR protocol, which tracks scene validity, target emergence and source retention separately, is also useful in its own right.
  2. The main design choices are supported by careful ablations (area-matched controls, background restoration, paired statistical tests) rather than simply asserted.
Critical weaknesses
  1. The motivation for using a reference image is not clear to me. The references are generated by GPT-Image-2 from the target text alone, so it is not obvious what information they add beyond the text, or in what realistic scenario an attacker would need one. The ablation shows that adding the visual branch helps, but this may simply be the effect of an extra prototype-based loss rather than of the reference itself. Controls such as mismatched or random references, or prototypes built without a reference, are needed.
  2. Important baselines are missing, and one result is left unexplained. M-Attack and FOA-Attack are exactly the reference-guided attacks the paper positions itself against, yet they are not compared. Also, the ensemble version is consistently weaker than the single-surrogate version. This goes against the usual trend and against what V-Attack shows in the same table, and it is not discussed.
Questions for authors
  1. Why is adding a reference image reasonable and meaningful? What does the reference image add beyond its text description? Results with mismatched/random references, or with prototypes built without a reference, would help answer this.
  2. How does RefBind compare with M-Attack and FOA-Attack under the same setting?
  3. Why is RefBind-Ens weaker than RefBind-Sgl? Were the same hyperparameters used for both?
  4. How does the method perform under some defenses? If questions are addressed convincingly, I am willing to raise my score.
7 comments
AuthorsOpenReview ↗

[W1/Q1]: Motivation and Role of Target Reference Images

We thank the reviewer for the thoughtful comments and constructive suggestions.

Response Summary

Reference images instantiate the target semantics as object-level visual prototypes, providing a feature-space alignment signal distinct from textual supervision. RefBind extracts these prototypes from the Target-CVR and aligns the designated Source-CVR features through source-prototype repulsion and target-prototype attraction.

Details

(1) Motivation, Role, and Acquisition of Reference-Visual Supervision.

Motivation for dual-modal supervision. AttackVLM (Zhao et al., 2023) reports different transfer behaviors for image–text and image–image feature alignment, while AdvEDM (Wang et al., 2025) identifies a modality gap that may limit the injection of target semantics into local visual features. These findings suggest that textual and visual supervision provide non-equivalent optimization signals, even when representing the same target semantics. Accordingly, RefBind investigates how to coordinate textual and reference-visual supervision for designated source-to-target object-level semantic replacement.

Role and acquisition of reference images. Reference images provide object-level representations of the target semantics in visual feature space, enabling RefBind to establish an additional object-level alignment constraint through source-prototype repulsion and target-prototype attraction. Since the original COCO300 benchmark introduced by V-Attack (Nie et al., 2026) does not provide target reference images, we follow the reference-generation practice of AttackVLM and use GPT-Image-2 to generate images containing the designated target objects (Appendix A.1). Reference images are used only during offline adversarial optimization and are not provided to victim models. The formulation accommodates either generated or retrieved images, provided they contain semantically matched, recognizable, and localizable target objects, without requiring a particular predefined reference instance.

(2) Suggested Controls and Existing Evidence.

Regarding the suggested control experiments. Mismatched references can assess the importance of target-semantic consistency, randomly selected semantically matched references can examine reference-instance sensitivity, and reference-free prototypes can evaluate alternative forms of prototype supervision. These controls would offer complementary insights into reference selection and prototype construction. Our study focuses on the incremental contribution of semantically matched reference-visual supervision and its coordination with textual guidance for object-level semantic replacement.

Our ablations show that adding reference-visual supervision improves STRR by 15.89 percentage points (Table 9; Holm-adjusted p = 0.0096), while object-supported Final-only alignment achieves higher E-VORS than both V-Attack and GlobalRef (39.67% vs. 36.33% and 27.33%, respectively; Table 17). These findings support the effectiveness of object-level reference-visual supervision, and we will further investigate reference selection and alternative prototype construction in future work.

REFERENCES

[1] Zhao, Y., Pang, T., Du, C., Yang, X., Li, C., Cheung, N.-M., and Lin, M. (2023). On Evaluating Adversarial Robustness of Large Vision-Language Models. Advances in Neural Information Processing Systems (NeurIPS 2023).

[2] Wang, Y., Zhang, H., Pan, H., Zhou, Z., Wang, X., Guo, P., Xue, L., Hu, S., Li, M., and Zhang, L. Y. (2025). AdvEDM: Fine-grained Adversarial Attack against VLM-based Embodied Agents. Advances in Neural Information Processing Systems (NeurIPS 2025).

[3] Nie, S., Zhang, J., Yan, J., Shan, S., and Chen, X. (2026). V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2026).

AuthorsOpenReview ↗

[W2/Q2]: Comparison with M-Attack and FOA-Attack

Response Summary

Following the reviewer's suggestion, we conducted additional experiments with M-Attack and FOA-Attack. All four methods were evaluated on the RoboCasa365-derived Full150 benchmark under the same (L_\infty=16/255) perturbation budget, five victim models, and VLM-PTR evaluation protocol. RefBind achieves the highest macro Strict-All (28.40%) and E-VORS (45.20%) while maintaining high scene validity (SVR: 98.00%), demonstrating a favorable balance between robotic-scene validity and strict object-level semantic replacement. Detailed results and discussion are provided below.

Details

Table R2-Q2. VLM-PTR comparison of four attack methods on RoboCasa365-derived Full150 (%). Each method is evaluated on 150 cases per victim. Macro denotes the unweighted average across five victims. SVR measures scene validity; TER and SER measure target emergence and source retention among scene-valid cases. E-VORS and Strict-All are computed over all cases.

VictimMethodSVR ↑E-VORS ↑TER ↑SER ↓Strict-All ↑
DeepSeek-VLM-Attack48.0038.00100.0041.6728.00
FOA-Attack66.6736.0090.0076.0012.00
V-Attack100.0037.6768.0078.677.33
RefBind100.0051.3372.6752.6730.00
InternVL2-8BM-Attack30.6720.3397.8365.2210.67
FOA-Attack82.6746.3399.1986.2910.67
V-Attack100.0045.3386.6795.334.00
RefBind100.0062.6796.0068.0029.33
GPT-4oM-Attack4.003.67100.0016.673.33
FOA-Attack63.3334.0070.5349.4723.33
V-Attack100.009.6714.6756.004.67
RefBind92.6739.3346.0418.7136.00
Gemini-3.6M-Attack45.3330.0086.7642.6520.67
FOA-Attack84.6738.3366.1468.5020.67
V-Attack100.0013.0022.0085.334.00
RefBind98.0044.6757.8239.4632.67
Claude-4.6M-Attack26.6718.6792.5050.0012.67
FOA-Attack89.3338.6767.9174.6316.67
V-Attack100.0015.0022.6755.337.33
RefBind99.3328.0042.2855.0314.00
MacroM-Attack30.9322.1395.4243.2415.07
FOA-Attack77.3338.6778.7570.9816.67
V-Attack100.0024.1342.8074.135.47
RefBind98.0045.2062.9646.7728.40

Results and discussion.

  • M-Attack and FOA-Attack. M-Attack achieves high target emergence among scene-valid cases (macro TER: 95.42%) but exhibits low scene validity (SVR: 30.93%). FOA-Attack achieves higher scene validity (77.33%) while retaining substantial source-object semantics (SER: 70.98%). These observations are consistent with their image-level target alignment objectives.
  • V-Attack. V-Attack achieves 100.00% SVR but only 5.47% Strict-All, indicating that high scene validity alone does not ensure strict source-to-target semantic replacement.
  • RefBind. RefBind achieves the highest Strict-All on four of the five victim models. Its coordinated textual and object-level reference-visual supervision supports effective strict semantic replacement while maintaining high scene validity.
AuthorsOpenReview ↗

[W2/Q3]: Why is RefBind-Ens weaker than RefBind-Sgl? Were the same hyperparameters used for both?

Response Summary

We thank the reviewer for highlighting the unexpected Single–Ensemble performance gap.

Our analysis suggests that Majority-based Source-CVR fusion may exclude useful surrogate-specific visual support. Replacing Majority with Union raises COCO300 CAP macro ASR from 44.12% to 68.00% (+23.88 pp), surpassing RefBind-Sgl (53.75%). On RoboCasa365, Union improves strict replacement on common scene-valid samples but decreases SVR and E-VORS, revealing a task-dependent trade-off between visual support and scene compatibility.

The hyperparameters are not identical: RefBind-Sgl uses CLIP ViT-L/14@336 (200 steps), while RefBind-Ens uses B16+B32+LAION (300 steps). Each follows its corresponding V-Attack setting; both use ϵ=16/255\epsilon=16/255.

Details

(1) Source-CVR Fusion and Experimental Design.

Fusion mechanism. CVRs identify object-semantic support from CLIP Value features. V-Attack selects source-aligned Value features independently for each surrogate; RefBind-Ens instead fuses Source-CVRs by Majority Voting, requiring spatial agreement across surrogates. This may discard useful surrogate-specific regions.

For the three surrogate masks Ms(m)M_s^{(m)}, Majority and Union are

\qquad M_s^{\mathrm{Union}}(p)=\mathbb{1}\left[\sum_{m=1}^{3}M_s^{(m)}(p)\geq1\right].$$ We introduce **RefBind-Ens-U**, replacing Source-CVR Majority fusion with Union while keeping Target-CVR unchanged. Union restores single-vote regions excluded by Majority. On a common $24\times24$ grid, Source-CVR coverage increases from approximately 186 to 289 positions (**1.55×**) on COCO300, versus 32 to 121 (**3.76×**) on RoboCasa365. Because Majority support is initially more compact on RoboCasa365, its relative expansion is larger and may affect scene compatibility. **(2) COCO300: Source-Union Attack Results.** **Table Q3-1. Source-Union ablation on COCO300 (CAP ASR, %).** | Method | LLaVA-NeXT | InternVL2 | DeepSeek-VL | GPT-4o | Macro | | ------------- | ---------- | --------- | ----------- | --------- | --------- | | RefBind-Sgl | **69.00** | 56.17 | 40.83 | 49.00 | 53.75 | | RefBind-Ens | 39.83 | 51.00 | 39.33 | 46.33 | 44.12 | | RefBind-Ens-U | 61.33 | **72.00** | **66.33** | **72.33** | **68.00** | Union improves CAP ASR by **21.00–27.00 pp across all four victims**, reversing the Ensemble–Single gap in the macro average. This supports Source-CVR consensus filtering as a contributing factor to the original Ensemble performance gap. **(3) RoboCasa365: Replacement–Scene Validity Trade-off.** **Table Q3-2. Source-Union ablation on RoboCasa365 (VLA-PTR, macro %, five victims).** | Method | SVR ↑ | E-VORS ↑ | Strict-All ↑ | | ------------- | --------- | --------- | ------------ | | RefBind-Ens | **94.93** | **51.20** | **36.40** | | RefBind-Ens-U | 84.00 | 46.73 | 36.27 | | Δ (U − Ens) | −10.93 | −4.47 | −0.13 | Among **616 case–victim pairs valid under both configurations**, Union increases strict replacement from **37.50% to 43.18% (+5.68 pp)**. However, **96** pairs shift from Majority Valid to Union Invalid/Error, versus **14** in the reverse direction. SVR falls by **10.93 pp** and E-VORS by **4.47 pp**; Strict-All remains nearly unchanged. Some transitions are evaluation errors, so these losses cannot all be attributed to scene degradation. Consistently, the GC→GG ablation in Appendix A.6 (Table 14) also reduces SVR (84.67%→58.67%) and E-VORS (57.33%→41.33%).
Reviewer_JFzGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies targeted attacks on vision-language models that replace a designated source object with a target object, using a reference image of the target. The proposed RefBind finds object supports in the source and reference images from Value-feature similarity, applies text supervision over the whole image and reference-visual supervision only inside the source-object support, and adds an intermediate layer chosen by gradient compatibility with the final layer. It is evaluated by transfer from CLIP surrogates to open-source and commercial VLMs on captioning, VQA, and a robotic planning benchmark. The paper concludes that the attack replaces objects more often than V-Attack and that multimodal understanding and planning-related outputs are vulnerable to fine-grained semantic manipulation.

Critical strengths
  1. The ablations isolate design factors with controls: matched-area interventions (Table 12) show that the benefit of broader textual realization is not explained by area alone, and Table 9 uses case-clustered paired tests with Holm correction. This gives evidence for the spatial-allocation claim beyond a single comparison.
  2. The evaluation reports the full, partial, and failure distribution besides ASR, which separates replacement from partial induction (for example SSA-CWA on InternVL2 has 39.50% ASR with no full success). This makes the comparison with V-Attack more informative than ASR alone.
Critical weaknesses
  1. In Section 1 and Section 5.1, the baselines (MF-ii, AnyAttack, SSA-CWA, V-Attack) do not use a target reference image, and M-Attack and FOA-Attack, which the paper describes as target-image-guided, are not evaluated. The gain over V-Attack may come from the extra reference image and not from the proposed binding, since only a 30-case ablation (Table 17) gives V-Attack a reference. The authors should compare with M-Attack and FOA-Attack, and with V-Attack given the same reference image, on the full benchmarks.
  2. In Section 4.3 and Appendix A.7, the intermediate layer is selected once by gradient cosine at the clean input, and Table 17 shows gains of 2.00 E-VORS and 3.33 Strict-All over Final-only on 30 cases, with one random layer per surrogate as the control. No confidence interval or repeated random draw is reported, so the benefit of the selection rule is not established. The authors should report multiple random layers and seeds with intervals.
  3. In Appendix A.1.2 and Appendix A.6, the 30-case subset used for all ablations and diagnostics is the first 30 cases of the 150-case benchmark, and the spatial allocation of the final method is chosen on it. Table 3 therefore includes cases used to select the method, so the authors should report the remaining 120 cases separately.
  4. In Section 5.1 and Appendix A.4.2, VLM-PTR scores the victim's own answers to "pick up the target" and "pick up the source" with no clean-image control. In Table 3, V-Attack reaches 86.67% TER and 95.33% SER on InternVL2 while its CAP ASR is 12.00% (Table 1), which indicates the queries are often answered affirmatively without replacement. The authors should report the same rates on clean images and with unrelated objects.
  5. In Section 5.2 and Table 6, LLaVA-NeXT uses the same CLIP-ViT-L/14@336 encoder as the single surrogate, and RefBind-Ens (44.12) is below RefBind-Sgl (53.75) in Table 1. The headline single-surrogate gain therefore includes a victim that shares the surrogate encoder, so the authors should report results without LLaVA-NeXT and explain why the ensemble does not help RefBind.
Questions for authors
  1. How many candidate source and target cases were removed by the LLaVA-NeXT and DeepSeek-VL recognition filter (Appendix A.1.2), given that both are evaluated victims?
  2. What are TER and SER of the VLM-PTR queries on the clean source images and with objects unrelated to the scene?
6 comments
AuthorsOpenReview ↗

[Q1] Recognition Filtering and Evaluation Validity

[Q1] Recognition Filtering and Evaluation Validity

Response Summary

We thank the reviewer for the thoughtful comments and constructive suggestions.

The recognition filtering is designed to establish valid pre-attack semantic conditions for targeted object replacement, rather than to select cases based on attack outcomes. The resulting benchmark contains 150 cases, and all attack methods are evaluated on the same set.

Details

(1) Dataset Construction and Quality Control.

Source-image selection. RoboCasa365 provides robot manipulation videos, from which we construct a static-image benchmark for object-level semantic replacement evaluation. We first extract candidate source images and manually verify that the designated source objects are clearly visible and not substantially occluded. Candidates that fail this visibility check are excluded.

Recognition verification. We then use LLaVA-NeXT and DeepSeek-VL to verify three conditions: (i) the designated source object is correctly recognized in the clean source image, (ii) the corresponding target object is absent from the clean source image, and (iii) the designated target object is correctly recognized in its reference image.

(2) Filtering Rationale and Evaluation Validity.

Pre-attack semantic conditions. These checks establish the necessary initial conditions for evaluating targeted semantic replacement. The source object must be recognizable before the attack, and the target object must not already be present in the source image. Otherwise, it would be difficult to distinguish the model's initial recognition state from semantic changes induced by adversarial perturbations.

Independence from attack outcomes. Although LLaVA-NeXT and DeepSeek-VL are also evaluated victims, their role in filtering is limited to verifying pre-attack object recognition conditions. The filtering process does not use adversarial examples or attack success rates from RefBind or V-Attack. All methods are subsequently evaluated on the same 150 retained cases.

[Q2/W4] Clean-Image and Unrelated-Object Controls for VLM-PTR

Response

On clean source images, VLM-PTR yields TER = 0% and SER = 100% for both the predefined target-object queries and the unrelated-object control using a teddy bear. These results indicate that the evaluated queries do not produce affirmative responses for absent objects under the tested clean-image conditions.

Clean-image control protocol. Building on the benchmark construction described in Q1, we evaluate clean source images using the same VLM-PTR queries and scoring rules as in the adversarial evaluation. The Clean-source control queries the designated source object and its corresponding target object, while the Clean-unrelated control replaces the target-object query with a teddy bear and retains the original source-object query.

Evaluation findings. Across the evaluated valid scenes, the source-object queries receive affirmative responses, whereas the corresponding target-object and teddy-bear queries receive negative responses. This provides direct evidence that the tested VLM-PTR queries do not exhibit false affirmative responses on these clean-image controls.

AuthorsOpenReview ↗

Weaknesses 1 & Weaknesses 2

[W1] Reference Visual Supervision and Baseline Comparisons

Response Summary

Several baselines use visual target information, but RefBind focuses on object-level binding and coordinated text–visual supervision. On RoboCasa365 Full150, RefBind achieves the highest macro E-VORS (45.20%) and Strict-All (28.40%).

Details

(1) Baseline supervision. MF-ii aligns adversarial features with a text-synthesized target image (Eq. 2); AnyAttack learns image-conditioned perturbations via self-supervision; SSA-CWA emphasizes transferability; and V-Attack uses text-guided Value features. Thus, the baselines are not all text-only.

(2) Binding and ablation. RefBind organizes target-reference semantics through source–target object binding and coordinated text–visual supervision; binding is not a separate information source. In Table 17, GlobalRef lowers V-Attack's E-VORS from 36.33% to 27.33%, whereas RefBind reaches 41.67%. Together with Table 9, this supports structured reference supervision over global alignment.

(3) Additional baselines. We compare M-Attack and FOA-Attack with V-Attack and RefBind on the same RoboCasa365 Full150 cases, five victims, ε=16/255, and VLM-PTR protocol. V-Attack targets object replacement, whereas M-Attack and FOA-Attack primarily target image-level transfer.

Table W1. RoboCasa365 Full150 VLM-PTR, five-victim macro average (%).

MethodSVR ↑E-VORS ↑Strict-All ↑
M-Attack30.9322.1315.07
FOA-Attack77.3338.6716.67
V-Attack100.0024.135.47
RefBind98.0045.2028.40

Findings. M-Attack and FOA-Attack yield lower SVR (30.93%/77.33%) and Strict-All (15.07%/16.67%). V-Attack attains 100% SVR but only 5.47% Strict-All. RefBind maintains 98.00% SVR and achieves the highest E-VORS and Strict-All, leading strict replacement on four of five victims (except Claude-4.6: 14.00% vs. FOA-Attack's 16.67%).

[W2] Hierarchical Visual Alignment and Intermediate-Layer Selection

Response Summary

On RoboCasa365 Full150, three random-layer seeds confirm gains from intermediate supervision. Compatible selection outperforms each random run, exceeding the random mean in E-VORS and Strict-All by 0.91 and 1.20 points.

Details

Under Single-L336, we compare Final-only (F), F + Random Intermediate (F+IR), and F + Compatible Intermediate (F+IC/RefBind-Sgl). Compatible layers are selected once using clean-input gradient cosine similarity; random layers use the same candidate set with seeds 2024/2025/2026. Selected layers remain fixed, and attack settings and the five-victim VLA-PTR protocol are identical.

Table W2. RoboCasa365 Full150, five-victim macro average (%); random mean and 95% CI across three seeds.

ConfigurationE-VORS ↑Strict-All ↑
Final-only40.2722.00
Random (2024)43.8727.73
Random (2025)44.3326.53
Random (2026)44.6727.33
Random mean (95% CI)44.29 [43.29, 45.29]27.20 [25.68, 28.72]
Compatible (RefBind-Sgl)45.2028.40

Random supervision improves E-VORS and Strict-All over Final-only by 4.02 and 5.20 points on average. Compatible selection adds 0.91 and 1.20 points over the random mean, outperforming all three draws. These results support intermediate-layer supervision and the task-aware selection rule without claiming global optimality.

AuthorsOpenReview ↗

Weaknesses 3 Evaluation on the Remaining 120 Cases

Response

We additionally evaluate the remaining 120 cases (case_030–case_149), excluding the first 30 cases used for ablation studies and configuration selection. All attack configurations, victim models, and evaluation protocols remain unchanged.

Table W3. VLM-PTR results (%) on the remaining 120 cases of the RoboCasa365-derived benchmark under single- and ensemble-surrogate settings. Macro denotes the equally weighted average across five victim models.

MethodVictimSVR ↑E-VORS ↑Strict-All ↑
V-Attack-SglDeepSeek-VL100.0037.086.67
InternVL2100.0045.834.17
GPT-4o100.008.334.17
Gemini-3.6100.0014.585.00
Claude-4.6100.0016.258.33
Macro100.0024.425.67
RefBind-SglDeepSeek-VL100.0051.6730.00
InternVL2100.0061.2526.67
GPT-4o93.3338.3335.00
Gemini-3.698.3345.4231.67
Claude-4.699.1729.5815.83
Macro98.1745.2527.83
V-Attack-EnsDeepSeek-VL100.0032.5019.17
InternVL2100.0054.5820.00
GPT-4o95.0023.7522.50
Gemini-3.699.1727.0815.83
Claude-4.696.6739.5835.00
Macro98.1735.5022.50
RefBind-EnsDeepSeek-VL96.6750.8336.67
InternVL2100.0061.6730.00
GPT-4o85.8347.0843.33
Gemini-3.699.1747.5035.83
Claude-4.698.3346.2532.50
Macro96.0050.6735.67

Results and Discussion. On the remaining 120 cases, RefBind-Sgl improves E-VORS and Strict-All over V-Attack-Sgl by 20.83 and 22.16 percentage points, respectively. RefBind-Ens achieves corresponding gains of 15.17 and 13.17 percentage points. These improvements are comparable to those on the full 150-case benchmark, indicating that RefBind's performance advantage persists on cases not used for spatial configuration selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.