acceptodds
Under review as a conference paper at ICLR 2027

Learning Action-Effect Latents for Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) policies are typically trained with embodiment-specific motor commands, creating a mismatch between rich multimodal representations and the low-level signals used for action supervision. We introduce AEL (Action-Effect Latents), a two-stage framework that augments action representation learning with the observed visual consequences of executed actions. In Stage I, AEL jointly encodes the current visual state and an executed action chunk into a latent representation that preserves the motor trajectory through action reconstruction while reconstructing the corresponding future visual representation through a residual effect formulation. In Stage II, this latent becomes the prediction target of a VLA policy conditioned on the current visual observations, language instruction, and robot state, and is decoded into continuous actions. Future observations are used only as supervision during Stage I and are not required at deployment. Separating representation learning from policy learning also allows suboptimal interactions to contribute action-effect supervision without being treated as imitation targets. Experiments on LIBERO, CALVIN, and real-world single-arm and bimanual manipulation show consistent improvements over matched direct-action baselines, supporting visual effects as a complementary supervision signal for learning robot action representations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Action Representation Learning, Latent Action Modeling, Future-Aware Representation Learning, Conditional Flow Matching, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_NCbyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AEL, a two-stage framework for learning latent representations that capture both robot actions and their visual effects. Stage I learns an action representation through action reconstruction and future visual-feature prediction. Stage II trains a VLA policy to predict this representation from current observations, language instructions, and robot state, then decode it into continuous actions. The method is evaluated on LIBERO, CALVIN, and real-world manipulation tasks. The paper also explores using suboptimal interactions for representation learning without treating them as imitation targets.

Critical strengths
  1. The method has a clear motivation. Using the observed effects of actions to shape their representations is a sensible approach, and the future-prediction branch is not required at deployment.

  2. The evaluation provides encouraging evidence across different settings. Experiments cover simulation and real-world single-arm and bimanual tasks, while controlled ablations support the benefit of visual-effect supervision. Using suboptimal interactions for representation learning without directly imitating them is also practically relevant.

Critical weaknesses
  1. The reuse scope of the Stage-I models could be clearer. Further details about the training data for each benchmark and whether encoders and decoders are shared across tasks would help assess the transferability of the representation and the practical value of the two-stage design.

  2. The comparison with direct auxiliary supervision could be strengthened. The ablations support the usefulness of visual-effect supervision, but do not compare against adding a future-feature prediction loss directly to the same VLA policy. This comparison would help clarify the independent benefit of two-stage representation learning.

  3. Generalization evidence could be broadened. CALVIN ABC→D provides useful cross-environment evidence. Additional controlled perturbation evaluations, such as LIBERO-Plus, would help establish whether the action-effect representation remains effective under changes in visual conditions and robot state.

Questions for authors
  1. Which datasets are used to train each Stage-I model? Are encoders and decoders shared across tasks or benchmarks? Can an existing Stage-I model be reused while training only Stage II for new tasks?

  2. Have you compared AEL against direct action prediction with an auxiliary future-feature prediction loss? Does the two-stage approach retain an advantage under matched data and total training budgets?

  3. Could you provide a LIBERO-Plus comparison of Direct Action, Recon. Latent, and AEL to further assess the OOD benefits of effect supervision?

Reviewer_hPZ6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a two-stage framework that augments action representation learning with supervision from observed visual changes. In the first stage, action reconstruction and residual visual-effect prediction jointly shape a latent representation of the executed action and current scene. At inference time, the VLA policy predicts this latent, which a frozen action decoder converts into continuous actions. Experimental results show improvements over the matched baselines and competitive performance against published latent-action-based methods.

Critical strengths

AEL achieves strong performance on the evaluated benchmarks and real-world tasks, with improvements over the matched baselines.

Critical weaknesses
  1. The discussion of closely related work is incomplete. The paper emphasizes learning action representations using visual dynamics, but several existing methods, including villa-X[1], XR-1[2], UniT[3], and METIS[4], already explore incorporating visual and motor information into latent representation learning, albeit in different ways. These works are not discussed or compared in the manuscript. A clearer explanation of the technical differences and the additional contribution of AEL, particularly its residual effect formulation, is needed.

[1]Xiaoyu Chen, Hangxing Wei, Pushi Zhang, Chuheng Zhang, Kaixin Wang, Yanjiang Guo, Rushuai Yang, Yucen Wang, Xinquan Xiao, Li Zhao, et al. Villa-x: enhancing latent action modeling in vision-languageaction models. arXiv preprint arXiv:2507.23682, 2025

[2]Shichao Fan, Kun Wu, Zhengping Che, Xinhua Wang, Di Wu, Fei Liao, Ning Liu, Yixue Zhang, Zhen Zhao, Zhiyuan Xu, et al. Xr-1: Towards versatile vision-language-action models via learning unified vision-motion representations. arXiv preprint arXiv:2511.02776, 2025

[3]Chen, Boyu, et al. "UniT: Toward a Unified Physical Language for Human-to-Humanoid Policy Learning and World Modeling." arXiv preprint arXiv:2604.19734 (2026).

[4]Yankai Fu, Ning Chen, Junkai Zhao, Shaozhe Shan, Guocai Yao, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Metis: Multi-source egocentric training for integrated dexterous vision-language-action model. arXiv preprint arXiv:2511.17366, 2025.

  1. The comparisons in Table 1 do not sufficiently isolate the contribution of the proposed latent representation. The methods differ in both how their latent actions are learned and how they are used for policy learning. For example, LA-Direct and LA-Tok use latent prediction as auxiliary supervision for VLM representations, whereas AEL predicts a latent that is directly decoded into actions. Although the paper acknowledges differences in experimental configurations, these comparisons cannot establish whether the gains come from the representation itself or its integration into the policy. Table 3 supports the proposed objectives within AEL’s framework, but does not compare alternative latent representations under a matched policy setup or different uses of the same latent representation.
Questions for authors
  1. Could you provide a controlled comparison with a joint vision–action latent method such as UniT under a matched policy-learning setup? Conversely, how does the same AEL latent perform when used for auxiliary VLM supervision compared with latent prediction followed by action decoding?

  2. Beyond the existing CALVIN and real-world evaluations, could you evaluate a more challenging benchmark or distribution-shift setting? The near-saturated success rates on LIBERO make small performance differences difficult to interpret.

  3. Have you evaluated learning a shared latent representation across embodiments with different action spaces, or transferring the learned representation to an unseen embodiment? How would AEL handle heterogeneous action spaces?

Reviewer_n572OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a new method, AEL, in latent action learning. AEL encodes the action chunk in the context of the current scene to get the latent action, which helps ground the latent action in scene-conditioned effect. The authors then use these latent actions as the prediction target in VLA policy training, and observe improvements in LIBERO, CALVIN and real-world experiments.

Critical strengths
  1. Future visual observations are used only as supervision during Stage I representation learning and are not required at deployment, which avoids the inference overhead and compounding errors of rolling out a world model at test time.

  2. Predicting residual updates in a frozen DINOv2 feature space is simple and effective, and the authors evaluate the framework across both standard simulation benchmarks (LIBERO, CALVIN) and real-world single-arm and bimanual tasks.

Critical weaknesses
  1. Some claims and figures are misleading. In Figure 1(b), the authors suggest that latent actions are derived via zt=f(at:t+H). But this is not what most latent action papers (e.g., LAPA, UniVLA) do. Previous latent action papers usually derive latent actions from zt=f(ot,ot+H). For more recent papers, they have already taken the action into consideration (e.g., JALA).

  2. The primary advantage of using latent actions is to leverage vast amounts of unlabeled video data. However, the proposed method relies directly on ground-truth action labels as input in Stage I, which makes this core advantage no longer valid. If the proposed method is restricted to action-labeled data, it is unclear why two-stage learning is necessary. For instance, one might doubt whether simply adding an auxiliary head to predict the delta visual embedding in single-stage training could achieve the same gain.

  3. The paper claims to capture semantic visual changes caused by actions, but the execution horizon is only H=8 with stride s=1 at 30 Hz (about 0.27 seconds). Over such a short horizon, high-level semantic states barely change. In fact, Table 6 shows that simply copying the current DINOv2 feature already yields a 0.9932 cosine similarity with the future feature, while AEL gets 0.9955 and the Shuffle Gap is only 0.000041. This suggests the effect branch mainly fits tiny frame-to-frame motion rather than meaningful semantic effects.

Questions for authors

See weakness.

There are several inconsistencies and missing results in the paper. First, Section 4.6 and Table 5 state that suboptimal data uses a 20% erroneous-action insertion rate, whereas Appendix C.2 states it is 30%, without explaining how erroneous actions are generated or executed. Second, the real-world bimanual baseline score of AEL is 61.1% in Figure 4, but drops to 57.8% (S/S) in Table 5, and the gain from adding suboptimal data (S+U/S, 61.1%) is only 3 trials out of 90. Third, Appendix D (lines 855–857) refers to "spatial-correlation, Top-k overlap, and latent-swap diagnostics reported for the final ablations", which do not exist anywhere in the paper.

Reviewer_jo2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AEL, a two-stage method for learning action representations for VLA policies. In Stage 1, the latent action encoder takes an action chunk and the current observation as input. The encoded latent is trained to reconstruct the action and to predict the future DINO features through a residual update. In Stage 2, a VLA policy predicts this latent from the current images, instruction, and robot state, and the predicted latent is then sent to the fixed decoder to generate the continuous action chunk. The paper reports gains on LIBERO, CALVIN, and six real-world tasks.

Critical strengths
  1. The method has a useful separation between representation learning and policy learning. It uses future observations as training targets and does not require the effect branch during deployment.

  2. The controlled ablation in Table 3 is helpful. It separates action reconstruction, future-feature supervision, and residual prediction while keeping the latent policy fixed. The improvements over the matched baseline appear on both LIBERO and CALVIN.

Critical weaknesses
  1. The novelty of this paper is insufficient. The latent action encoder has been proposed since 2024; previous works like LAPA take the current observation and action as input and reconstruct the action and future frame in the decoding phase. Meanwhile, there are many existing works that utlize DINO / Optical flow features as supervision in the VLA. Directly combining these two setups could not demonstrate the novelty of this framework. More ablation studies are required to illustrate which component is essential.

  2. The simulation and real-world experiments are weak. The simulation results show an obvious gap with the SOTA baselines. For the real-world experiments, the authors only introduce DP and Pi-0 as baselines, rather than more recent open-sourced VLAs like Pi-0.5 and LingBot-VLA. The evaluated real-world tasks are also trivial.

Questions for authors
  1. Which component or design contributes most to the performance?
  2. Could you include more recent baselines in the real world experiment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.