- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether vision-language models can provide task-centric representations for latent action learning in the presence of distractors. The authors use VLM embeddings as prediction targets for the forward dynamics model and compare a wide range of VLMs and prompting strategies. Experiments on Distracting MetaWorld show substantial improvements in latent action quality and downstream success rates.
Critical strengths
The paper addresses an important limitation of latent action models, particularly their sensitivity to action-correlated distractors. Using VLM representations as prediction targets is a simple and practical approach.
The experimental evaluation is extensive, covering many VLMs, prompting strategies, and representation extraction settings. The analysis provides useful insights into which VLMs produce representations suitable for action learning.
The improvements are demonstrated through both linear probing and downstream success rates. The additional experiments with varying action dimensions and labeled data budgets further support the effectiveness of the approach.
Critical weaknesses
The benefit of language conditioning is uncertain. The comparisons with CLIP and DINOv2 are useful, but these models also differ from VLMs in architecture and pretraining. Therefore, their performance gaps cannot be attributed entirely to language conditioning. A comparison with simpler task-focused visual representations would help clarify the advantage of VLMs.
The methodological novelty is relatively limited. The main approach replaces the original FDM prediction target with pretrained VLM embeddings. Promptable representations have already been explored in reinforcement learning by prior works. Although applying this idea to latent action learning is useful, the paper mainly provides empirical validation rather than a new learning mechanism.
The evaluation is largely limited to controlled simulation. Most experiments use MetaWorld with artificially added background videos. The DROID experiment provides some real-world evidence, but it only reports action probing on 100 episodes without downstream robot control.
Questions for authors
Q1. Can the authors directly measure whether VLM embeddings remain stable when only the background changes, while remaining sensitive to actual robot actions?
Q2. How does the proposed approach compare with simpler task-focused visual representations, such as object cropping or segmentation with visual encoders?
Q3. Can the authors clarify whether the improvements also hold in real-world robot control, beyond the preliminary DROID probing results?
2 comments
Thanks for the response
Thank you for the additional experiments and detailed responses. The new results provide evidence for the benefits of prompt-based semantic filtering, addressing some of my concerns. However, the limited methodological novelty and lack of downstream real-world robot evaluation remain. Therefore, I have decided to maintain my original score.
We thank the reviewer for recognizing the importance of the problem and the value of our empirical evaluation. We conducted two additional experiments directly addressing Q1 and Q2, and clarify the scope of our real-world evaluation below.
Q1. Does language conditioning suppress distractors while preserving action-relevant changes?
We now provide direct counterfactual evidence of prompt-dependent semantic filtering. We sampled 576 small-motion expert transitions across Reach, Push, and Drawer Open (48 trajectories per task). For each transition, we independently varied the physical state and background using held-out DAVIS videos. Background-only and task-motion-only changes were matched within 15% in image-wide RGB RMSE.
We measured the difference between task-motion and background-motion sensitivity in Molmo embedding MSE. Positive changes indicate stronger relative sensitivity to task motion.
These results support a more precise conclusion: control-relevant semantic focus improves the representation's relative selectivity, whereas detailed task-goal descriptions are not consistently necessary.
This is consistent with our existing prompt ablations (Tables 1–2), which hold Molmo and the extraction configuration fixed. For example:
In an additional new prompt ablation, "Point to the robotic arm gripper." achieves 0.79 MSE versus 1.30 for "Describe the scene.", a 39% reduction under the same model configuration.
Together, these results demonstrate that prompt content affects both the representation's sensitivity to relevant versus irrelevant changes and the quality of subsequently learned latent actions. They do not establish that every fine-grained action variable is perfectly preserved, and we will clarify that distinction.
Q2. Comparison with simpler cropping or segmentation approaches
We implemented a SAM2-based masked reconstruction baseline, following the masking approach of MaskLAM, while retaining RGB observations for the IDM. We evaluated both methods across all 10 MT10 tasks and three random seeds, with identical training protocols and a single shared hyperparameter configuration.
At our primary evaluation budget of 16 labeled trajectories:
Thus, Molmo outperforms the practical masking baseline by 4.9 percentage points in our primary evaluation setting, without requiring explicit segmentation masks. This comparison demonstrates that VLM-based targets provide a measurable advantage over this spatial-filtering alternative in the evaluated setting. We do not claim universal superiority over segmentation, and will revise Appendix B to reflect this distinction.
Contribution and novelty
While our architectural intervention is deliberately simple, the contribution is not merely applying pretrained VLM features to LAPO. We establish the importance of prediction-target selection, systematically benchmark which VLM representations are suitable for latent-action learning, and now provide controlled evidence that language can steer the relative sensitivity of these targets toward action-relevant changes. We acknowledge that offline probing on DROID does not replace closed-loop real-robot evaluation, which remains as future work.
Importantly, our new experiments demonstrate both the mechanism of prompt-based semantic selection and its practical competitiveness with a strong foreground-masking baseline. We believe these findings constitute a meaningful empirical contribution beyond prior work on promptable representations for policy learning.