acceptodds
Under review as a conference paper at ICLR 2027

Vision-Language Models Provide Task-Centric Promptable Representations for Latent Action Learning

Abstract

Latent Action Models (LAMs) have rapidly emerged as a promising approach for addressing the scarcity of high-quality action-labeled data. However, they fail when observations contain action-correlated distractors, often encoding noise instead of meaningful latent actions. Humans, on the other hand, can effortlessly distinguish task-relevant motions from irrelevant details in any video given only a brief task description. In this work, we propose to utilize the common-sense reasoning abilities of Vision-Language Models (VLMs) to provide promptable representations, effectively separating controllable changes from the noise. We use these representations as targets during LAM training and benchmark a wide variety of popular VLMs, revealing substantial variation in the quality of promptable representations as well as their robustness to different prompts and hyperparameters. Interestingly, we find that newer VLMs do not necessarily outperform older ones; however, all evaluated VLMs consistently outperform CLIP and DINOv2, highlighting the vital importance of language conditioning. Finally, we show that simply asking VLMs to ignore distractors can substantially improve latent action quality and downstream performance on Distracting MetaWorld.

Primary area:
reinforcement learning
Keywords:
latent action learning, latent actions, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vAzBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether vision-language models can provide task-centric representations for latent action learning in the presence of distractors. The authors use VLM embeddings as prediction targets for the forward dynamics model and compare a wide range of VLMs and prompting strategies. Experiments on Distracting MetaWorld show substantial improvements in latent action quality and downstream success rates.

Critical strengths

The paper addresses an important limitation of latent action models, particularly their sensitivity to action-correlated distractors. Using VLM representations as prediction targets is a simple and practical approach.

The experimental evaluation is extensive, covering many VLMs, prompting strategies, and representation extraction settings. The analysis provides useful insights into which VLMs produce representations suitable for action learning.

The improvements are demonstrated through both linear probing and downstream success rates. The additional experiments with varying action dimensions and labeled data budgets further support the effectiveness of the approach.

Critical weaknesses

The benefit of language conditioning is uncertain. The comparisons with CLIP and DINOv2 are useful, but these models also differ from VLMs in architecture and pretraining. Therefore, their performance gaps cannot be attributed entirely to language conditioning. A comparison with simpler task-focused visual representations would help clarify the advantage of VLMs.

The methodological novelty is relatively limited. The main approach replaces the original FDM prediction target with pretrained VLM embeddings. Promptable representations have already been explored in reinforcement learning by prior works. Although applying this idea to latent action learning is useful, the paper mainly provides empirical validation rather than a new learning mechanism.

The evaluation is largely limited to controlled simulation. Most experiments use MetaWorld with artificially added background videos. The DROID experiment provides some real-world evidence, but it only reports action probing on 100 episodes without downstream robot control.

Questions for authors

Q1. Can the authors directly measure whether VLM embeddings remain stable when only the background changes, while remaining sensitive to actual robot actions?

Q2. How does the proposed approach compare with simpler task-focused visual representations, such as object cropping or segmentation with visual encoders?

Q3. Can the authors clarify whether the improvements also hold in real-world robot control, beyond the preliminary DROID probing results?

2 comments

We thank the reviewer for recognizing the importance of the problem and the value of our empirical evaluation. We conducted two additional experiments directly addressing Q1 and Q2, and clarify the scope of our real-world evaluation below.

Q1. Does language conditioning suppress distractors while preserving action-relevant changes?

We now provide direct counterfactual evidence of prompt-dependent semantic filtering. We sampled 576 small-motion expert transitions across Reach, Push, and Drawer Open (48 trajectories per task). For each transition, we independently varied the physical state and background using held-out DAVIS videos. Background-only and task-motion-only changes were matched within 15% in image-wide RGB RMSE.

We measured the difference between task-motion and background-motion sensitivity in Molmo embedding MSE. Positive changes indicate stronger relative sensitivity to task motion.

  • Directing visual focus: Changing from "What do you see in the background?" to "Point to the robotic arm gripper." increases the task-versus-background sensitivity gap by +0.210 [0.180, 0.240] (paired trajectory-bootstrap 95% CI), consistently across all three tasks. Gripper prompting reduces sensitivity to both types of motion, but suppresses background changes more strongly.
  • Adding task-specific instructions: Compared with the generic question "Briefly, what is the next logical action for the robotic arm?", prepending the task description changes the sensitivity gap by +0.021 [−0.020, 0.062] on average (paired trajectory-bootstrap 95% CI).

These results support a more precise conclusion: control-relevant semantic focus improves the representation's relative selectivity, whereas detailed task-goal descriptions are not consistently necessary.

This is consistent with our existing prompt ablations (Tables 1–2), which hold Molmo and the extraction configuration fixed. For example:

  • Foreground versus background prompting: 0.93 versus 1.20 distracted action-probe MSE.
  • Correct versus incorrect task-object prompting: 1.08 versus 1.31 MSE.
  • Generic gripper versus background-directed prompting: 0.64 versus 1.60 MSE.

In an additional new prompt ablation, "Point to the robotic arm gripper." achieves 0.79 MSE versus 1.30 for "Describe the scene.", a 39% reduction under the same model configuration.

Together, these results demonstrate that prompt content affects both the representation's sensitivity to relevant versus irrelevant changes and the quality of subsequently learned latent actions. They do not establish that every fine-grained action variable is perfectly preserved, and we will clarify that distinction.

Q2. Comparison with simpler cropping or segmentation approaches

We implemented a SAM2-based masked reconstruction baseline, following the masking approach of MaskLAM, while retaining RGB observations for the IDM. We evaluated both methods across all 10 MT10 tasks and three random seeds, with identical training protocols and a single shared hyperparameter configuration.

At our primary evaluation budget of 16 labeled trajectories:

  • Molmo: 65.9% IQM success [63.5%, 68.1%].
  • SAM2 masked reconstruction: 61.0% IQM success [59.3%, 62.8%].

Thus, Molmo outperforms the practical masking baseline by 4.9 percentage points in our primary evaluation setting, without requiring explicit segmentation masks. This comparison demonstrates that VLM-based targets provide a measurable advantage over this spatial-filtering alternative in the evaluated setting. We do not claim universal superiority over segmentation, and will revise Appendix B to reflect this distinction.

Contribution and novelty

While our architectural intervention is deliberately simple, the contribution is not merely applying pretrained VLM features to LAPO. We establish the importance of prediction-target selection, systematically benchmark which VLM representations are suitable for latent-action learning, and now provide controlled evidence that language can steer the relative sensitivity of these targets toward action-relevant changes. We acknowledge that offline probing on DROID does not replace closed-loop real-robot evaluation, which remains as future work.

Importantly, our new experiments demonstrate both the mechanism of prompt-based semantic selection and its practical competitiveness with a strong foreground-masking baseline. We believe these findings constitute a meaningful empirical contribution beyond prior work on promptable representations for policy learning.

Reviewer_vAzBOpenReview ↗

Thanks for the response

Thank you for the additional experiments and detailed responses. The new results provide evidence for the benefits of prompt-based semantic filtering, addressing some of my concerns. However, the limited methodological novelty and lack of downstream real-world robot evaluation remain. Therefore, I have decided to maintain my original score.

Reviewer_7eLSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how vision-language models (VLMs) can improve latent action learning in the presence of visual distractors. The authors argue that conventional latent action models struggle because their forward dynamics models attempt to predict irrelevant visual changes alongside task-relevant motion. To address this, they propose using task-conditioned VLM embeddings as prediction targets, encouraging the learned latent actions to focus on controllable changes.

The paper benchmarks a wide range of VLMs, prompts, and embedding extraction strategies, finding that Molmo provides particularly effective and robust representations. Experiments on Distracting MetaWorld MT10 demonstrate substantial improvements in latent action quality and downstream task success, with additional preliminary validation on real-world DROID data.

Critical strengths
  1. Clear motivation and effective empirical validation. The paper identifies an important limitation of latent action models and provides a convincing demonstration that changing the forward dynamics prediction target can substantially improve performance under visual distractions. The improvements in downstream task success, rather than just representation quality, make the findings practically meaningful.

  2. Extensive benchmarking and useful insights into VLM representations. The evaluation covers a broad range of VLMs, prompt configurations, and embedding extraction strategies. The finding that newer or embedding-specialized models do not necessarily produce better representations for latent action learning is interesting and potentially useful for future robotics research.

Critical weaknesses
  1. Limited methodological novelty. The proposed approach primarily combines an existing latent action learning framework (LAPO) with VLM-based promptable representations, which have already been explored in prior work (e.g., Chen et al., 2024). While the application to latent action learning is useful, the method itself is relatively simple, and the contribution appears to be mainly empirical rather than methodological.

  2. Insufficient analysis of why VLM representations improve performance. The paper attributes the improvements to task-centric semantic filtering, but it is unclear whether this is the primary reason for the gains. VLM embeddings may simply be more robust to background changes or provide an easier prediction target than raw pixels. Although the CLIP, DINOv2, and OTTER comparisons are helpful, they do not fully disentangle these explanations. Furthermore, the main evaluation relies on synthetic background distractors in MetaWorld, while the real-world DROID experiment only reports action-probe results without downstream policy evaluation.

Questions for authors
  1. How much of the improvement can be attributed specifically to prompt-based semantic filtering, rather than to the inherent robustness of pretrained VLM features? For example, would task-agnostic VLM embeddings provide similar improvements under the same prediction setup?

  2. How well would the proposed approach generalize to more realistic distractors, such as moving cameras, occlusions, or independently moving objects within the robot's workspace? The DROID results are encouraging, but downstream policy evaluation in real-world settings would help establish the practical significance of the method.

3 comments

We thank the reviewer for acknowledging the substantial downstream improvements and the value of our extensive VLM benchmarking. We address the concerns regarding novelty, the mechanism behind the improvements, and generalization below.

Novelty and scientific contribution. We respectfully disagree that our contribution is merely combining existing methods. While promptable representations were previously explored for policy learning (Chen et al., 2024), their suitability as prediction targets for unsupervised latent action learning is a distinct and nontrivial question. Representations suitable for policy inputs are not necessarily suitable for learning latent actions: the prediction target must suppress irrelevant dynamics while retaining fine-grained information about controllable changes.

Our contribution is primarily empirical, but provides several new findings:

  • The prediction target is critical: Our controlled experiment (Section 4) demonstrates that changing only the FDM target can restore latent-action quality and downstream success despite keeping distractors in the inputs.
  • VLMs differ substantially in suitability for latent-action learning: We benchmark over 20 VLMs and extensive prompt/extraction configurations, finding that newer and embedding-specialized models do not necessarily outperform older models such as Molmo.
  • The improvements translate to closed-loop control: With one shared Molmo configuration across all MT10 tasks, we achieve 55%+ IQM success with just 16 labeled trajectories, using fewer than 1% labeled trajectories relative to the 5,000 unlabeled demonstrations per task.

The simplicity of our intervention is deliberate: it isolates the role of representation targets without introducing architectural modifications.

Why do VLM representations improve performance?

Prompt conditioning versus pretrained features. We agree that comparing Molmo with CLIP/DINOv2 alone cannot isolate language conditioning. However, our existing within-Molmo ablations already fix the model and extraction configuration while varying the prompt.

For instance, Table 1 demonstrates that prompt focus interacts with distractors:

  • Foreground-focused prompt: MSE increases from 0.56 (clean) to 0.93 (distracted).
  • Background-focused prompt: MSE increases from 0.45 (clean) to 1.20 (distracted).

Thus, although background prompting performs better on clean observations, the ranking reverses under distractors. This is consistent with semantic focus affecting robustness, rather than one prompt simply producing universally better representations.

In Table 2, the generic gripper prompt achieves 0.64 MSE, compared with 1.08 for the correct task-object prompt, 1.31 for an incorrect task object, and 1.60 for background-directed prompting.

We additionally conducted a separate ablation with a generic scene-description baseline. "Point to the robotic arm gripper." achieves 0.79 MSE versus 1.30 for "Describe the scene." (39% lower error). Simply asking "Do you see a robot?" yields 1.27, indicating that mentioning the robot alone does not reproduce the improvement.

Direct evidence of semantic filtering. More importantly, we conducted a new controlled counterfactual experiment with 576 small-motion expert transitions across Reach, Push, and Drawer Open. We independently varied simulator state and background, matching background-only and task-motion-only changes within 15% in image-wide RGB RMSE.

We measured the gap between task-motion and background-motion sensitivity in Molmo embedding MSE. Positive changes indicate greater relative sensitivity to task motion.

  • Directing visual focus: Changing from "What do you see in the background?" to "Point to the robotic arm gripper." increases this gap by +0.210 [0.180, 0.240] (paired trajectory-bootstrap 95% CI), consistently across all three tasks. Gripper prompting reduces sensitivity to both changes, but suppresses background changes more strongly.
  • Adding task instructions: Prepending the explicit task goal to an otherwise identical generic question changes the gap by only +0.021 [−0.020, 0.062] on average.

These results provide direct evidence that control-relevant semantic focus changes the representation's sensitivity to task motion relative to distracting motion, whereas detailed task-goal descriptions are not consistently necessary. They support the mechanism behind our strong generic gripper prompt, although they do not establish lossless preservation of every fine-grained action variable.

Comparison with simpler foreground filtering. To further test whether the advantage can be explained by simple background removal, we implemented SAM2-based foreground-masked reconstruction, following the masking approach of MaskLAM.

Using identical training protocols, one shared hyperparameter configuration, all 10 MT10 tasks, and three seeds, Molmo achieves 65.9% IQM success [63.5%, 68.1%], compared with 61.0% [59.3%, 62.8%] for SAM2 masking at our primary budget of 16 labeled trajectories.

This demonstrates that VLM targets offer a measurable advantage over a practical spatial-filtering baseline in our primary setting.

Overall, our contribution goes beyond applying pretrained features to LAPO: we systematically identify which representations work as unsupervised prediction targets, demonstrate substantial downstream improvements, and now provide direct counterfactual evidence explaining how prompt focus changes the relative encoding of task-relevant and distracting motion. We believe these findings constitute a meaningful empirical contribution to latent-action learning.

Thank you for the detailed response and the additional experiments. The counterfactual analysis provides useful evidence that prompt focus affects sensitivity to task-relevant and distracting motion, and the SAM2 comparison strengthens cases for VLM prediction targets. However, my concerns about the limited methodological novelty and the scope of validation remain. The additional experiments do not address the more realistic distractors raised in my review, such as camera motion, occlusions, or independently moving objects within the workspace. The absence of downstream real-world policy evaluation also leaves the practical generalization of the approach uncertain. While I appreciate the stronger empirical analysis, these remaining concerns leave my overall assessment unchanged, and I will maintain my original rating.

Reviewer_HgwpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the sensitivity of latent action models to visual distractors by replacing pixel reconstruction targets with frozen, prompt-conditioned VLM representations. The authors benchmark multiple VLMs and representation extraction strategies, and demonstrate substantial improvements in action probing and downstream control on Distracting MetaWorld. Additional experiments examine shared configurations, latent dimensionality, lighting changes, and action probing on real-world DROID data.

Critical strengths
  1. A clear and well-motivated intervention. The clean-target experiment shows that changing the prediction target can substantially restore latent action learning while retaining distractors in the inputs. Using prompt-conditioned VLM features provides a straightforward approach that does not require paired clean observations during ordinary training.

  2. Substantial downstream improvements and useful empirical analysis. Figure 12 reports an increase in success-rate IQM from 0.09 to 0.60 under distractors. A shared prompt and extraction configuration still achieves 0.55 with 16 labeled trajectories, reducing reliance on task-specific choices. The broad comparison of VLMs, prompts, and extraction strategies also offers useful practical guidance.

Critical weaknesses
  1. The trade-off between distractor suppression and fine-grained information preservation remains unclear. Comparisons with CLIP and DINOv2 also change architecture and pretraining, limiting attribution to language conditioning. Tables 1–2 demonstrate prompt sensitivity, but the strong generic gripper-localization prompt leaves open whether the benefit primarily comes from robot-focused features. Meanwhile, pooled per-frame VLM representations may suppress small but control-critical differences alongside irrelevant variation. Although the downstream improvements are encouraging, the paper does not directly examine whether these targets preserve fine-grained displacement, gripper state, and contact transitions.

  2. A practical background-removal baseline is missing, particularly given the amplified synthetic distractors. The benchmark increases background exposure by moving the camera farther away and removing table borders. This is a valid stress test, but the gains under default camera settings or milder distractors remain unclear. For this relatively stable workspace, cropping or estimated foreground masks are relevant alternatives. Section 4 demonstrates effective oracle clean targets, but does not evaluate a deployable approximation. Appendix B’s discussion of segmentation limitations does not establish its inadequacy for the main background-video setting. These omissions leave the additional value of large VLM representations insufficiently established.

Questions for authors
  1. Could you quantify how the target representations respond to small, action-relevant changes versus background changes, and whether this sensitivity differs between task-specific and generic prompts? This would clarify the trade-off between distractor suppression and fine-grained information preservation.

  2. How does the method compare with workspace cropping or estimated foreground masks applied only to the prediction targets, while keeping the remaining training pipeline and label budget unchanged?

1 comment

We thank the reviewer for the constructive feedback. We conducted two additional experiments directly addressing Q1 and Q2, including a SAM2 masking baseline across all MT10 tasks and a controlled representation-sensitivity analysis.

Could you quantify how the target representations respond to small, action-relevant changes versus background changes, and whether this sensitivity differs between task-specific and generic prompts?

We measured the difference between task-motion and background-motion sensitivity in Molmo embedding MSE, comparing prompts with identical model and extraction settings. Positive changes indicate greater relative sensitivity to task motion.

The results distinguish explicit task-goal conditioning from directing the model's visual focus:

  • Adding explicit task goals: Compared with the generic question "Briefly, what is the next logical action for the robotic arm?", prepending the task description changes the sensitivity gap by +0.021 [−0.020, 0.062] on average (paired trajectory-bootstrap 95% CI).
  • Directing visual focus: Compared with "What do you see in the background?", the prompt "Point to the robotic arm gripper." increases the task-versus-background sensitivity gap by +0.210 [0.180, 0.240], consistently across all three tasks. Importantly, gripper prompting reduces sensitivity to both sources of change, but suppresses background changes more strongly.

These results support a more precise interpretation of our contribution: control-relevant semantic focus, rather than detailed task descriptions, is the primary useful form of prompting in our experiments. The results establish relative selectivity, not lossless preservation of every fine-grained state variable.

This is consistent with our existing within-Molmo prompt ablations (Tables 1–2). In an additional prompt ablation, gripper localization achieves 0.79 action-probe MSE versus 1.30 for generic scene description, a 39% reduction. Furthermore, Table 3 demonstrates that the same task-agnostic gripper prompt improves distracted MT10 success from 0.10 to 0.55 with 16 labeled trajectories. Thus, the selectivity measured in the embeddings is accompanied by substantial downstream control improvements.

How does the method compare with workspace cropping or estimated foreground masks applied only to the prediction targets, while keeping the remaining training pipeline and label budget unchanged?

We thank the reviewer for suggesting a practical foreground-removal baseline. We implemented SAM2-based masked reconstruction, following the masking approach of MaskLAM, while retaining RGB observations for the IDM.

We evaluated both approaches on all 10 MT10 tasks across three seeds, using identical training and evaluation protocols and a single shared hyperparameter configuration rather than task-specific tuning.

At our primary evaluation budget of 16 labeled trajectories, Molmo achieves 65.9% IQM success [63.5%, 68.1%], compared with 61.0% [59.3%, 62.8%] for SAM2-based masked reconstruction—a 4.9 percentage-point improvement.

These results demonstrate that prompt-conditioned VLM representations remain competitive with, and outperform, an explicit foreground-masking alternative in our primary evaluation setting, without requiring segmentation masks.

We also identified an IQM aggregation bug that underestimated previously reported success rates. We have corrected the computation and will update the corresponding results in the revised manuscript.

We agree that segmentation is a valuable alternative and will revise Appendix B to avoid suggesting that it is generally inadequate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.