acceptodds
Withdrawn from ICLR 2027

TARGET-VLA: Mask-Guided Visual Retrieval for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) policies benefit from pretrained vision-language representations, but precise manipulation also requires fine-grained object-level and geometric information. However, our controlled comparison indicates that incorporating complementary representations from specialized vision foundation models without explicit spatial guidance yields only modest gains. Motivated by this observation, we introduce TARGET-VLA, a method for mask-guided retrieval of visual representations from frozen segmentation and depth models. It separates the roles of spatial guidance and visual content: prompt-conditioned segmentation masks guide where to retrieve information, while pretrained representations supply the features for action generation. Experiments in simulation and on a physical robot demonstrate improved manipulation performance over retrieval without mask guidance. TARGET-VLA achieves competitive results across simulation benchmarks, including zero-shot transfer to LIBERO-Plus, with real-world gains extending to shifted object positions and object categories absent from the robot demonstrations. These findings demonstrate the value of spatial guidance in integrating semantic and spatial representations to improve robotic manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-language-action

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rnjVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TARGET-VLA augments a vision-language-action policy with compact representations retrieved from frozen SAM3 and Depth Anything 3 models. Instruction-derived object masks and an end-effector mask provide additive spatial biases to query-to-feature cross-attention, while the pretrained features supply the retrieved content. The segmentation branch jointly summarizes the prompted regions, whereas the depth branch separately retrieves object and end-effector information before aggregation. These tokens condition an action expert together with VLM and robot-state tokens. Evaluations cover multiple simulation benchmarks, zero-shot transfer to LIBERO-Plus, and three real-world tasks. The reported comparisons show improvements over unguided retrieval and over supplying masks as additional feature inputs.

Critical strengths
  1. A useful comparison of how task-conditioned masks are used. Table 3 compares TARGET-VLA with both unguided query retrieval and a variant that supplies the same masks as feature inputs, using the same frozen visual sources and 104 final conditioning tokens. The improvement over the mask-input variant from 4.45 to 4.68 in CALVIN average task-chain length, and from 77.0% to 84.3% in Success@5, is meaningful evidence that the retrieval interface matters. This is more informative than comparing only against a VLM-only policy.

  2. Encouraging improvements across evaluation settings. Relative to unguided retrieval, success increases from 81.9% to 88.8% on zero-shot LIBERO-Plus, and the equal-weight average across the three real-world in-domain tasks increases from 62.8% to 83.3%. The breadth of these results supports the practical value of task-conditioned visual retrieval beyond one simulation benchmark, even though the individual real-world sample sizes are modest.

Critical weaknesses
  1. The design does not directly address the precision motivation. The mask bias favors task-relevant regions; it does not specify contact-sensitive structure or alignment requirements within them. Fine geometry may remain available in DA3 features, so coarse masks do not invalidate the method. However, whole-task success cannot distinguish improved geometric precision from better target selection/background suppression. I regard this as useful visual retrieval, but the distinctive benefit of soft-mask guidance over simpler region guidance remains unclear. Matched bounding-box or binary-mask attention baselines would better establish that contribution.

  2. The central mechanism is not fully isolated or validated across training runs. The unguided variant removes DA3 validity gating and embeddings together with the attention bias (Section 4.4; Appendix C.4). Thus, its comparison measures a package of changes. The mask-input control is valuable, but whether it preserves validity conditioning is unclear. Moreover, CALVIN ablations use one training seed; sequence-level bootstrap intervals do not capture training variability. This limits confidence that the specific attention design consistently explains the advantage over the strongest controlled alternative.

Questions for authors
  1. Contribution and evidence: What evidence connects the retrieval improvements to finer manipulation accuracy, rather than better target selection/background suppression? A failure analysis separating selection errors from alignment/contact errors would help. Separately, how does soft-mask guidance compare with bounding-box or binary-mask guidance under matched visual sources, validity conditioning, and token budgets? Either targeted analysis would materially inform my assessment; I am not requesting a new benchmark suite.

  2. Clarification: Does “Query retrieval + mask input” retain the same DA3 validity gating and embeddings as TARGET-VLA? With these components fixed, what improvement comes from enabling only the attention bias?

  3. Reliability: Does the advantage over unguided and mask-input retrieval persist across independent training seeds? Means and dispersion for CALVIN average task-chain length and Success@5 would address uncertainty not captured by the sequence-level bootstrap.

Reviewer_1WxxOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

TARGET-VLA is motivated by the observation that incorporating complementary representations (segmentation & depth features) from specialized vision foundation models without explicit spatial guidance yields only modest gains. To solve this, TARGET-VLA introduced a mask-guided visual retrieval framework in which instruction-conditioned SAM3 masks identify task objects and the end effector, then add soft spatial biases to cross-attention over frozen SAM3 and Depth Anything 3 features. Then the object- and geometry-aware tokens are aggregated with VLM and proprioception tokens for action prediction. Experiments are conducted on CALVIN, LIBERO & LIBERO-PLUS, and three real-world manipulation tasks and demonstrate improved performance. Ablations also show that using masks to guide attention is more effective than simply concatenating masks with visual features.

Critical strengths
  • SOTA(99.1 on LIBERO, 99.0 on SIMPLER, 88.8 on LIBERO-PLUS, 4.36 on CALVIN in-domain D$\to$D) or near SOTA performance (CALVIN ABC$\to$D) on the simulation benchmarks. However, on RoboDojo, the results are a little bit outdated. TARGET-VLA's current 12.51 score and 8.72 SR cannot even rank in the top 20 on the leaderboard. At current review timestamps, SOTA performance has already reached a 53.18 score and 46.85% SR.
  • Ablations demonstrated the effectiveness of mask guidance, different visual feature strategies, and the choice of visual sources.
Critical weaknesses
  • On benchmarks where TARGET-VLA achieved SOTA performances, the leaderboard has already been saturated (LIBERO, CALVIN, etc.). However, on RoboDojo, current SOTA methods have already achieved a 53.18 score and 46.85% SR.
  • The total number of trials is not enough to demonstrate the significance of TARGET-VLA's performance: 20/30/30 trials for the three in-domain tasks and 15/24 for position shifts/unseen fruits. Although in the bar chart, TARGET-VLA shows significant performance improvement compared to π0.5\pi_{0.5} or w/o Guidance, the real difference in terms of actual successful times could be only 1 or 2.
  • The idea of applying object masks and EE masks on visual/depth features is intuitive and similar to previous works' ideas that guide the model's attention to the manipulation object and end-effector, which is nothing new. Below are some references.

[1] Haifeng Huang, Xinyi Chen, Yilun Chen, Hao Li, Xiaoshen Han, Zehan Wang, Tai Wang, Jiangmiao Pang, and Zhou Zhao. RoboGround: Robotic Manipulation with Grounded Vision-Language Priors. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

[2] Jesse Zhang, Marius Memmel, Kevin Kim, Dieter Fox, Jesse Thomason, Fabio Ramos, Erdem Bıyık, Abhishek Gupta, and Anqi Li. Peek: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies. In arXiv preprint arXiv:2509.18282, 2025.

[3] Zonghe Liu, Shanyuan Jie, Xiaoquan Sun, Chen Cao, Zetian Xu, Zongsheng Liu, and Jiayu Chen. SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models. In arXiv preprint arXiv:2607.25912, 2026.

[4] Xiaosong Jia, Bowen Yang, Zuhao Ge, Xian Nie, Yuchen Zhou, Cunxin Fan, Yufeng Li, Yilin Chai, Chao Jing, Zijian Liang, et al. GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization. In arXiv preprint arXiv:2605.12369, 2026.

[5] Yulin Luo, Hao Chen, Zhuangzhe Wu, Bowen Sui, Jiaming Liu, Chenyang Gu, Zhuoyang Liu, Qiuxuan Feng, Jiale Yu, Shuo Gu, et al. Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models. In arXiv preprint arXiv:2603.15618, 2026.

Questions for authors
  • Could you please add more real-world trials to demonstrate the significance of your model's performance instead of just some random shift?
  • Can you show some stronger performance on the RoboDojo benchmark or elaborate on how TARGET-VLA could be further improved to match current SOTA performance?
Reviewer_No1HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TARGET-VLA uses mask-guided attention to retrieve visual tokens from frozen SAM3 and DA3 encoders for a flow-matching VLA policy. Soft masks provide an additive bias to the retrieval attention: the SAM3 branch uses a merged object and end-effector mask, while the DA3 branch retrieves features per entity with validity conditioning. These tokens are combined with VLM features and robot state to predict actions. The paper evaluates the method across several simulation benchmarks and three real-world dual-arm manipulation tasks.

Critical strengths
  1. The paper addresses a useful question: how to incorporate external visual features into a VLA beyond simply concatenating them. Using masks to guide where the policy retrieves information is a reasonable design, while feature-based attention still determines what information is selected.
  2. The evaluation is broad, covering standard benchmarks, zero-shot transfer, and real-world tasks with position shifts and unseen object categories. The strong results across these settings support the usefulness of the overall approach.
Critical weaknesses
  1. The main ablation does not isolate the mask attention bias. According to the Table 3 notes and Appendix C.4, “w/o Guidance” removes both the mask bias and the DA3 validity gating and embeddings. Thus, the improvement from 4.35 to 4.68 supports the guidance components as a whole, but does not establish how much comes from the proposed attention bias. It is also unclear whether the “Query retrieval + mask input” control retains validity conditioning. This matters because mask-biased retrieval is presented as the central contribution.
  2. The guidance design leaves some questions about multi-object tasks. The SAM3 branch merges object and end-effector masks using a pixel-wise maximum, so the mask bias no longer explicitly distinguishes object identities or roles. Other features may still retain this information, but the experiments do not establish whether the merged guidance is sufficient for tasks such as “put A on B.” In addition, object prompts are extracted offline from the instruction without reference to the image. Relevant objects omitted from the instruction therefore receive no explicit object-mask guidance, and the effect of this limitation is unclear.
  3. The inference overhead needs more context. Running both frozen encoders adds approximately 25 ms of latency and 2.1 GB of memory. These costs are reported, but it is difficult to judge their practical significance without a clearer discussion of the resulting control rate and the trade-off between performance and resource use. A comparison with a cheaper configuration would help establish whether both branches are necessary at deployment.
Questions for authors
  1. How robust is the policy to degraded guidance in simulation, e.g. missing detections, inaccurate masks, or incorrect object prompts?

  2. Can you separately ablate the mask attention bias and the DA3 validity gating/embeddings? Please also clarify whether the “Query retrieval + mask input” baseline retains validity conditioning.

  3. TAM shows that attention shifts toward the guided regions. Can you test whether this shift helps action prediction, for example by perturbing the masks while keeping the image and instruction fixed and measuring the effect on task success?

  4. Can the inference overhead be reduced by running the encoders less frequently or using only one retrieval branch? What performance trade-off would this introduce?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.