acceptodds
Withdrawn from ICLR 2027

Visual Instruction Channels for Generalist Robot Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action (VLA) models offer a capable foundation for robot control, but their usefulness depends on the ability to accurately steer them, that is to specify what a robot should do. Natural language provides a flexible, yet sometimes ambiguous and inaccurate, interface, further limited by the reliability of language-steering. We perform an in-depth analysis of visual instructions as conditioning, individual and complementary to language, supplying them through the policy's existing image inputs without architectural modification. In simulation, we vary the signal's shape (point, box, mask), its delivery format (overlay, blur, blackout, separate image), and its quality (view coverage and signal delay). We find that quality matters most: a signal updated at every observation substantially outperforms the static, first-frame signal often used in prior work. Shape and format effects depend on the backbone and task, with masks and boxes providing reliable overlay choices across the evaluated policies. On three real-robot tasks, visual instructions improve task performance over language alone in every case. These results highlight visual instructions as a practical channel for communicating task intent to generalist robot policies, one that can outperform language on its own and is strongest in combination with it.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Generalist Robot Policies, Robotics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xWTdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores fine-tuning a VLA to be conditioned on various visual cues to indicate the target object and evaluating the effectiveness of these augmentations in terms of task success, tested on four VLA backbones involving simulation and real-world tasks. The main contribution is the collection of results from doing these experiments, specifically that adding such augmentations improve task performance, and that live annotations of the target object is better than static annotations, with delayed annotations not having a great impact when the delay is short.

Critical strengths

I believe the paper's systematic evaluation of different visual guidance methods is thorough, and the claims that the authors draw are well-supported by results from their experiments.

Critical weaknesses

My main concern with this paper is on the novelty, as the paper serves as an evaluation of several rudimentary method of visually indicating objects and does not report significant results that will benefit the broader community. The results that the authors draw are not surprising: adding visual guidance improves performance over language-only, real-time annotations are better than stale ones, and wrist cameras matter more than exocentric camera. Other findings are also not informative. For example, the authors claim that static annotations hurt wrist cameras more than exocentric cameras, but for a fixed external camera, the annotation will always be coincident with the target object until the object is moved, but for a wrist camera, a static annotation is not accurate whenever there is motion on the robot arm, so other than the very beginning of the motion, the annotation serves as distraction rather than useful information. When the authors are acknowledging prior work like ControlVLA and VP-VLA that is already using real-time visual annotation, there does not seem to be major findings from this paper that could further inform future work along similar areas.

Questions for authors
  • For systems that already use visual cues to guide VLAs, which of your findings could change how a practitioner uses these systems?
  • For the case of static visual cues, if they are removed from the wrist cameras and only kept on the exocentric camera, does task success improve meaningfully?
Reviewer_4WmzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to supply visual instructions (point, box, mask) to pretrained Vision-Language-Action (VLA) robot policies through their existing image inputs, without any architectural modification. Through systematic simulation experiments on pi0.5 and three other backbones, the authors find that signal quality—especially temporal freshness—matters most. Delivery format and view coverage also impact performance. Real-robot experiments on three tasks confirm that visual instructions improve success rates over language alone in every case. The key takeaway is that a simple, architecture-free visual conditioning channel—when kept temporally aligned—can substantially boost VLA performance.

Critical strengths

1.Compared to prior research on robotic visual instruction, this work systematically evaluates freshness and view coverage as independent factors.

2.The core findings are validated not only on π0.5\pi_{0.5} but also on π0\pi_0, GR00T N1.7, and SmolVLA, demonstrating that the benefits of visual instructions are not tied to a specific architecture, which strengthens the generality and transferability of the approach.

3.Investigation into image input with scattered objects, cluttered objects and irregular objects has strong practical significance and could be helpful to future works.

Critical weaknesses

1.Lack of novelty.​ Compared to prior works like VP-VLA, the main addition is studying freshness and view coverage as factors, but the algorithmic contribution is marginal. Incorporating subtractive methods (blackout, blur) amounts to a typical "A+B" combination.

2.Figures fail to effectively visualize the factors influencing visual prompts.​ The paper does not provide clear or intuitive illustrations of how different design choices (shape, format, quality) affect the policy's behavior.

3.Missing real-robot setup details.​ There is no description or image of the hardware configuration, nor any explanation of the low-level policy or controller used.

Questions for authors

1.Can you please specify the exact language instructions used in each real-world task (Scattered Objects, ISO, Chess)? Was the same template used as in simulation (e.g., "Pick the target and place it into the box"), and were there any task-specific modifications?

  1. Could you also explain why combining language prompts with visual prompts is better than using visual prompts alone? When training a visual-prompt-only policy, do you provide any default language instructions? One possible explanation is that pretraining with language instructions enables the fine-tuned model to perform better when language input is included. Therefore, I suggest you train a policy from scratch for comparison, such as ACT.

3.Could you clarify the practical value of this work? In particular, what additional capabilities or practical benefits does the proposed chain of visual inputs offer compared with prior related approaches?

Reviewer_jVbZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper, based on VLA, introduces additional visual instruction information to improve robot operation performance. The paper compares different visual localization representations (points, bounding boxes, and segmentation masks), input formats, camera field of view coverage, and the temporal freshness of visual cues. Experiments covered four VLA models, three simulation environments, and three real-world robot operation tasks. Results show that providing additional visual localization information significantly improves task success rate compared to using only verbal instructions. The paper proposes that dynamically updated visual cues generally outperform static cues generated only at the initial moment, and the effectiveness of different visual localization representations depends on the specific model and task.

Critical strengths

Strengths: Experiments were conducted across multiple VLA models, providing a reasonable degree of sufficiency.

Critical weaknesses
  1. The core idea of introducing additional visual grounding information into VLA models has already been explored in previous works, such as MOO, RoboGround, and VP-VLA. The proposed method mainly compares existing visual prompting strategies without introducing a substantially new algorithm or architecture. Although the analysis of dynamic visual guidance is useful, the paper does not sufficiently demonstrate what fundamentally new insights it provides beyond prior studies.
  2. The proposed dynamic guidance requires repeatedly generating segmentation masks using an additional SAM3 model, which introduces considerable computational overhead. However, the paper does not provide a direct comparison of end-to-end inference latency, or control frequency between the original VLA and the dynamic-mask approach.
Questions for authors

What is the difference in speed between using a dynamic mask and the original model itself?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.