- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how different conditioning interfaces affect the performance of VLM-guided action experts in long-horizon robotic assembly. The authors compare several representations, including 2D guidance points, coarse gripper-action cues, and end-effector orientation, and propose TAGPoint, which encodes spatial targets and gripper-action cues through colored points in the image. They further study robustness to imperfect guidance through noise training and fixed-state interventions. Experiments on FurnitureBench and AutoMate, along with evaluations using actual VLM predictions and a qualitative real-world demonstration, suggest that compact spatial guidance is effective, while richer orientation information provides no consistent benefit.
Critical strengths
1. A meaningful research question supported by a systematic comparison of conditioning interfaces. Determining what information an upstream VLM should provide to a downstream action expert is an important problem for modular robotic systems. Rather than assuming that richer conditioning information necessarily leads to better control, this paper investigates the effects of different information types while keeping the action expert architecture largely fixed. The experiments cover spatial targets, action semantics, end-effector orientation, RGB versus RGB-D observations, scripted versus VLM-generated guidance, and expanded task distributions. This systematic evaluation provides useful insights into how high-level information affects downstream robotic control.
2. A useful experimental framework for analyzing robustness to imperfect guidance. Beyond evaluating complete-task success under different target perturbations, the authors introduce a Fixed-State Intervention Bank that compares policy behavior from matched physical states under different guidance conditions. This design helps isolate the effect of guidance errors from differences in preceding trajectories. The results suggest that training with noisy guidance can improve robustness to inaccurate upstream predictions and reduce noise-induced endpoint displacement. This analysis provides a more detailed understanding of policy behavior than task-level success rates alone.
Critical weaknesses
1. The advantages of TAGPoint over simpler spatial conditioning interfaces are limited, and the current experiments do not sufficiently establish the necessity of its specific design. The main methodological contribution of TAGPoint is to encode a binary gripper-action cue through the color of a 2D guidance point. However, keypoint-conditioned manipulation and skill-conditioned policies have already been explored in prior work, such as KITE. Consequently, the methodological novelty of this representation appears relatively limited. More importantly, the experimental results do not convincingly demonstrate a consistent advantage over standard guidance points. In the main RGB-D comparison, TAGPoint achieves an overall success rate of 58.64%, compared with 57.10% for GP, an improvement of only 1.54 percentage points. This difference is small relative to the reported variation across training seeds. In the joint FurnitureBench–AutoMate experiment, TAGPoint and GP achieve identical FurnitureBench success rates, while GP performs better on the AutoMate OOD10 evaluation. Furthermore, although the paper includes a GP + Skill baseline, this approach uses a different conditioning mechanism from TAGPoint. Therefore, the experiments do not fully distinguish the benefit of providing coarse action semantics from the benefit of encoding those semantics directly into the image. A controlled comparison using identical action information with different conditioning mechanisms would be helpful to establish the specific advantages of TAGPoint.
2. Several experimental comparisons contain potential confounding factors, and the limited physical evaluation restricts the scope of the conclusions. First, the RGB and RGB-D encoders use different initialization strategies. According to Appendix A, the RGB encoders are trained from scratch, whereas the RGB-D encoders are initialized with ImageNet-pretrained weights. As a result, the observed improvements cannot be attributed solely to the availability of depth information. This is particularly relevant because the paper draws conclusions about how depth helps the action expert interpret 2D spatial guidance. Second, although the paper evaluates the complete VLM–action expert pipeline using actual VLM predictions, these experiments are conducted in simulation. The physical evaluation consists of a single successful one-leg assembly demonstration using scripted guidance rather than predictions from the VLM. No quantitative real-world success rates, failure analysis, or comparisons between conditioning interfaces are provided.
While simulation is appropriate for controlled interface comparisons, the current physical results provide limited evidence that the reported advantages of TAGPoint transfer to real-world operation under actual VLM prediction errors.
Questions for authors
1. How significant and consistent are the improvements of TAGPoint over GP, and what is the specific benefit of its color-based encoding? In the main RGB-D experiment, TAGPoint improves overall success by only 1.54 percentage points over GP, while GP performs better on AutoMate OOD10. Could the authors provide additional statistical analysis to establish whether TAGPoint offers a consistent advantage? Additionally, could the authors compare TAGPoint against alternative ways of injecting the same binary gripper-action cue, such as an additional low-dimensional conditioning vector? This would help distinguish the contribution of action semantics from that of the particular visual encoding mechanism.
2. Can the authors isolate the contribution of depth from that of visual encoder pretraining? Appendix A indicates that the RGB encoders are trained from scratch, whereas the RGB-D encoders use ImageNet-pretrained weights. Could the authors clarify the motivation for this difference and provide a comparison under matched initialization settings? Without such a control, it is difficult to determine how much of the performance improvement comes specifically from depth information.
3. To what extent do the findings transfer to real-world execution with actual VLM predictions? The physical demonstration uses scripted guidance rather than VLM-generated targets. Could the authors clarify whether the system has been evaluated with actual VLM predictions on the physical robot, and, if so, what failure modes were observed? If additional real-world experiments are not available, it would be helpful to clarify the scope of the conclusions and discuss how well the simulated guidance perturbations reflect the structured errors produced by VLMs in real-world settings.