acceptodds
Under review as a conference paper at ICLR 2027

What Should a VLM Tell an Action Expert? Comparing Conditioning Interfaces for Long-Horizon Furniture Assembly

Abstract

Translating the semantic understanding, task decomposition, and visual grounding capabilities of vision–language models (VLMs) into continuous robot actions typically requires a conditioning interface that conveys upstream predictions to an action expert. However, systematic evidence remains limited on what information this interface should contain and whether downstream execution can tolerate errors in actual VLM predictions. We use multitask, long-horizon furniture assembly as a diagnostic setting. This domain requires switching across tasks and manipulation stages, localizing parts and their targets, and executing contact-rich skills such as insertion and screwing, thereby amplifying and exposing the downstream effects of interface design and upstream error at the complete-task level. We compare conditioning interfaces constructed from different combinations of three information types: spatial targets, gripper-action cues, and end-effector rotation. Based on this comparison, we recommend the Target–Action Guidance Point (TAGPoint) as an intermediate representation: it encodes a coarse gripper-action cue in the color of a target point, integrating spatial and action information within a unified, image-aligned interface. We further train the action expert with noisy guidance to mitigate performance degradation under inaccurate upstream predictions. Our evaluation covers clean FurnitureBench assembly, guidance-noise training and evaluation, end-to-end execution driven by actual VLM predictions, joint FurnitureBench–AutoMate training, and physical-robot execution. Together, these results provide an empirical basis for selecting measurable and intervenable intermediate representations in VLM–action expert systems.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Robot Manipulation, Imitation Learning, Long-Horizon Manipulation, Visuomotor Policy Learning, Vision-Language Models, Policy Conditioning, Multitask Imitation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VFtCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how different conditioning interfaces affect the performance of VLM-guided action experts in long-horizon robotic assembly. The authors compare several representations, including 2D guidance points, coarse gripper-action cues, and end-effector orientation, and propose TAGPoint, which encodes spatial targets and gripper-action cues through colored points in the image. They further study robustness to imperfect guidance through noise training and fixed-state interventions. Experiments on FurnitureBench and AutoMate, along with evaluations using actual VLM predictions and a qualitative real-world demonstration, suggest that compact spatial guidance is effective, while richer orientation information provides no consistent benefit.

Critical strengths

1. A meaningful research question supported by a systematic comparison of conditioning interfaces. Determining what information an upstream VLM should provide to a downstream action expert is an important problem for modular robotic systems. Rather than assuming that richer conditioning information necessarily leads to better control, this paper investigates the effects of different information types while keeping the action expert architecture largely fixed. The experiments cover spatial targets, action semantics, end-effector orientation, RGB versus RGB-D observations, scripted versus VLM-generated guidance, and expanded task distributions. This systematic evaluation provides useful insights into how high-level information affects downstream robotic control.

2. A useful experimental framework for analyzing robustness to imperfect guidance. Beyond evaluating complete-task success under different target perturbations, the authors introduce a Fixed-State Intervention Bank that compares policy behavior from matched physical states under different guidance conditions. This design helps isolate the effect of guidance errors from differences in preceding trajectories. The results suggest that training with noisy guidance can improve robustness to inaccurate upstream predictions and reduce noise-induced endpoint displacement. This analysis provides a more detailed understanding of policy behavior than task-level success rates alone.

Critical weaknesses

1. The advantages of TAGPoint over simpler spatial conditioning interfaces are limited, and the current experiments do not sufficiently establish the necessity of its specific design. The main methodological contribution of TAGPoint is to encode a binary gripper-action cue through the color of a 2D guidance point. However, keypoint-conditioned manipulation and skill-conditioned policies have already been explored in prior work, such as KITE. Consequently, the methodological novelty of this representation appears relatively limited. More importantly, the experimental results do not convincingly demonstrate a consistent advantage over standard guidance points. In the main RGB-D comparison, TAGPoint achieves an overall success rate of 58.64%, compared with 57.10% for GP, an improvement of only 1.54 percentage points. This difference is small relative to the reported variation across training seeds. In the joint FurnitureBench–AutoMate experiment, TAGPoint and GP achieve identical FurnitureBench success rates, while GP performs better on the AutoMate OOD10 evaluation. Furthermore, although the paper includes a GP + Skill baseline, this approach uses a different conditioning mechanism from TAGPoint. Therefore, the experiments do not fully distinguish the benefit of providing coarse action semantics from the benefit of encoding those semantics directly into the image. A controlled comparison using identical action information with different conditioning mechanisms would be helpful to establish the specific advantages of TAGPoint.

2. Several experimental comparisons contain potential confounding factors, and the limited physical evaluation restricts the scope of the conclusions. First, the RGB and RGB-D encoders use different initialization strategies. According to Appendix A, the RGB encoders are trained from scratch, whereas the RGB-D encoders are initialized with ImageNet-pretrained weights. As a result, the observed improvements cannot be attributed solely to the availability of depth information. This is particularly relevant because the paper draws conclusions about how depth helps the action expert interpret 2D spatial guidance. Second, although the paper evaluates the complete VLM–action expert pipeline using actual VLM predictions, these experiments are conducted in simulation. The physical evaluation consists of a single successful one-leg assembly demonstration using scripted guidance rather than predictions from the VLM. No quantitative real-world success rates, failure analysis, or comparisons between conditioning interfaces are provided.

While simulation is appropriate for controlled interface comparisons, the current physical results provide limited evidence that the reported advantages of TAGPoint transfer to real-world operation under actual VLM prediction errors.

Questions for authors

1. How significant and consistent are the improvements of TAGPoint over GP, and what is the specific benefit of its color-based encoding? In the main RGB-D experiment, TAGPoint improves overall success by only 1.54 percentage points over GP, while GP performs better on AutoMate OOD10. Could the authors provide additional statistical analysis to establish whether TAGPoint offers a consistent advantage? Additionally, could the authors compare TAGPoint against alternative ways of injecting the same binary gripper-action cue, such as an additional low-dimensional conditioning vector? This would help distinguish the contribution of action semantics from that of the particular visual encoding mechanism.

2. Can the authors isolate the contribution of depth from that of visual encoder pretraining? Appendix A indicates that the RGB encoders are trained from scratch, whereas the RGB-D encoders use ImageNet-pretrained weights. Could the authors clarify the motivation for this difference and provide a comparison under matched initialization settings? Without such a control, it is difficult to determine how much of the performance improvement comes specifically from depth information.

3. To what extent do the findings transfer to real-world execution with actual VLM predictions? The physical demonstration uses scripted guidance rather than VLM-generated targets. Could the authors clarify whether the system has been evaluated with actual VLM predictions on the physical robot, and, if so, what failure modes were observed? If additional real-world experiments are not available, it would be helpful to clarify the scope of the conclusions and discuss how well the simulated guidance perturbations reflect the structured errors produced by VLMs in real-world settings.

Reviewer_oLKWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks which visual prompts is most effective for conditioning a downstream visuomotor policy for multitask, long-horizon tasks such as furniture assembly. The authors compare four image-aligned representations as conditioning interfaces against an unconditioned baseline, each under RGB and RGB-D observations:

  • GP: a 2D guidance point
  • TAGPoint: GP but with color-coded gripper state (either the gripper is open or close)
  • 6D grasp: the end-effector's 6D pose with rotated box projected on the image plane.
  • Colored grasp: 6D grasp but with color-coded gripper state

The authors train DiT-based policies on each interface and concluded that TAGPoint as the most effective intermediate representation.

Critical strengths

The question of what's the most effective intermediate representation to interface a low-level action policy is relevant to hierarchical robot-learning system.

Critical weaknesses

Overall, the paper in its current form falls well short of the standard expected at this conference. My main concerns are as follows.


1. Poor presentation quality

The manuscript is not yet in a state suitable for publication. The central interfaces are underspecified, the choice of interfaces is not motivated, and key concepts are introduced late or not at all.

(a) Insufficient specification of the conditioning interfaces. Although the interfaces are the central object of study, Sec. 2.2 describes them only briefly and ambiguously, leaving the reader unable to reconstruct the actual design:

  • Target definition. What does the target point represent for each skill: the grasp point on the part, the insertion site, or an intermediate pre-contact pose? Appendix B states only that the point is the projection of "the corresponding 3D target," and the paper never precisely defines this target.
  • Representation details. Which colors, marker sizes, and marker shapes are used, and how are the markers rendered in the wrist view versus the front view?
  • Rotation encoding. How is the target end-effector orientation represented as a 2D rectangle, and which rotational degrees of freedom does the rectangle actually preserve?

(b) Unmotivated design space. The paper does not explain why these four interfaces were selected or how they were derived. Natural alternatives are neither evaluated nor discussed, such as:

  • multiple keypoints (e.g., a start point and an end point);
  • short trajectories or waypoints [3, 4, 5].

Without such a rationale, the comparison appears ad hoc, and the recommendation of TAGPoint amounts to a choice among four hand-picked variants rather than a principled answer to the question posed in the title.

(c) Undefined or prematurely used terminology.

  • Point VLM is used in Sec. 4.2 (L.308) to report a "position-equivalent error of 63.44 mm," but is not introduced until Sec. 4.4.
  • Grasp VLM is described in a single sentence (L.377–378), with no information on its training data or prediction accuracy.
  • GP + skill (Table 2) is never defined in the main text; the only indication of its meaning is a passing remark in Appendix D.3.

2. The main claims lack statistical support

(a) No interface yields a consistent improvement. Table 1 does not support the paper's central claims that "explicit 2D targets consistently improve action generation" and that TAGPoint is "the strongest interface":

  • With RGB observations, conditioning provides no meaningful benefit. GP exceeds the unconditioned baseline by only 0.46 points (48.61% vs. 48.15%), TAGPoint (47.92%) performs below it, and on lamp both TAGPoint (10.42%) and GP (14.58%) fall below the baseline (18.52%).
  • With RGB-D observations, the per-task results are mixed. On one-leg, TAGPoint underperforms GP (83.33% vs. 85.19%) and exceeds the baseline (82.41%) by only 0.92 points. On lamp, 6D grasp achieves the highest success rate (37.04%), contradicting the claim that rotation information provides "no consistent benefit."

(b) The reported differences are small relative to the variance.

  • In RGB-D, TAGPoint outperforms GP by only 1.54 points overall (58.64 ± 4.57 vs. 57.10 ± 5.10), averaged over just three seeds.
  • TAGPoint's margin over the unconditioned baseline is about 6 points, comparable to standard deviations of up to 5 points.
  • Per-task standard deviations are as large as ±15.47 (GP, round-table) and ±20.85 (6D grasp, round-table).

3. Limited novelty

Conditioning manipulation policies on image-space keypoints or visual prompts is well established. For example:

  • KITE [1] grounds language into 2D keypoints that condition learned skills.
  • CLIPort [2] combines semantic and spatial pathways for language-conditioned manipulation.
  • RT-Trajectory [3] conditions policies on 2D trajectory sketches drawn on the image.
  • ATM [4] guides policy learning with predicted point trajectories.
  • HAMSTER [5] uses a VLM to predict 2D paths that condition a low-level policy, which is the same hierarchical VLM-to-policy design studied here.

TAGPoint adds a single bit of gripper-action information encoded as a color, which is a minor variation on these approaches. Training with noisy conditions is likewise a standard augmentation technique. Moreover, the closely related hierarchical systems [3, 5] are neither cited nor discussed. Since the experiments do not distinguish TAGPoint from GP (see Weakness 2), the paper offers neither a novel method nor a reliable empirical finding.


References

[1] P. Sundaresan, S. Belkhale, D. Sadigh, and J. Bohg. KITE: Keypoint-Conditioned Policies for Semantic Manipulation. CoRL, 2023.

[2] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and Where Pathways for Robotic Manipulation. CoRL, 2021.

[3] J. Gu et al. RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches. ICLR, 2024.

[4] C. Wen et al. Any-point Trajectory Modeling for Policy Learning. RSS, 2024.

[5] Y. Li et al. HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation. ICLR, 2025.

Questions for authors
  • Definition of the target. What exactly does the target point represent for each of the five skills (Pick, Place, Push, Insert, Screw)? For example, is it the grasp point on the part, the insertion site on the receiving part, or an intermediate pre-contact pose? How is the target chosen when a stage involves both a part and a destination, and when does the target switch from one to the other within a stage?

  • Choice of interfaces. Why were these four interfaces chosen? Were alternatives such as multiple keypoints, waypoints, or short trajectories considered? Why is GP + skill evaluated only with VLM guidance (Table 2) and not under clean guidance in Table 1?

Reviewer_Bft8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates what information a vision-language model (VLM) should provide to a downstream action expert (Diffusion Transformer (DiT)). The authors use long-horizon, contact-rich furniture assembly as study case. The paper compares no condition, spatial guidance points (GP), target points augmented with a gripper-action cue (TAGPoint), and interfaces that additionally encode end-effector orientation, using RGB and RGB-D observations. Experiments cover clean scripted guidance, perturbed guidance, actual VLM predictions, joint FurnitureBench–AutoMate training, and a qualitative real-robot demonstration. The results suggest that explicit 2D targets improve performance, while coarse gripper-action cues can provide additional benefits; projected orientation does not consistently help.

Critical strengths

S1 - practical research question. The paper studies the interface between high-level VLM reasoning and low-level continuous control, an important design choice for embodied systems.

S2 - controlled comparisons. The authors largely hold the action-expert architecture and training setup fixed while varying the conditioning information, making the core comparisons informative.

S3 - broad evaluation within the chosen setting. The study examines multiple assembly tasks, RGB/RGB-D inputs, guidance perturbations, actual VLM predictions, and transfer to a larger task set. The refined assembly-success criteria also address potential false positives in the original simulator evaluation.

S4 - useful empirical findings. The results challenge the assumption that richer geometric conditioning is necessarily better and suggest that a compact, image-aligned target with a coarse action cue can be effective.

Critical weaknesses

W1 - limited and context-dependent gains. In the RGB-D comparison, TAGPoint achieves 58.64% success versus 57.10% for GP, a modest improvement whose statistical significance is unclear. TAGPoint is also not consistently best across modalities and task distributions.

W2 - limited evidence for generality. The study uses one relatively small DiT action expert and a narrow set of assembly tasks. The authors acknowledge that the conclusions may not transfer to other policy architectures, embodiments, or broader task distributions.

W3 - orientation comparison is difficult to interpret. Projecting 6D grasp orientation into an image-space marker may discard important rotational information or introduce ambiguities. Consequently, the poor results do not establish that explicit orientation information is inherently unhelpful.

W4 - real-world evidence is preliminary. The physical evaluation demonstrates a successful TAGPoint-guided rollout but provides no quantitative success rate or comparison against alternative interfaces. It therefore establishes feasibility, not real-world effectiveness or robustness.

Questions for authors

Q1 - are the performance differences between TAGPoint and GP statistically significant across seeds? Can per-task results clarify where the action cue helps?

Q2 - how much of the orientation interfaces' underperformance comes from the image-space projection rather than the orientation information itself?

Q3 - how does TAGPoint compare with providing the gripper action as a separate binary token or skill label?

Q4 - can the authors provide quantitative real-robot results and evaluate robustness to semantic errors, such as incorrect part or assembly-stage selection, beyond controlled spatial perturbations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.