acceptodds
Withdrawn from ICLR 2027

Predicting Semantic Latents for Robust and Efficient World-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World-Action Models (WAMs) adapt pretrained video generative models for robot control by jointly predicting future pixel observations and actions. We argue that robot manipulation requires reasoning about high-level scene structure and object dynamics, but does not require pixel-level fidelity, which can be sensitive to nuisance factors like lighting. Rather than predicting future pixels and actions, we train our model to predict future semantic latents (e.g., frozen DINO, SigLIP, or V-JEPA encodings of future frames) and future actions. We refer to our approach as Semantic Latent Action Prediction (SLAP). Importantly, we find that video diffusion backbones can be efficiently post-trained to directly regress (in a single forward pass) future semantic latents, eliminating the need for expensive iterative video denoising. Semantic targets improves robustness to environmental perturbations (like lighting or even end-effector initial position) and requires far less robot data for post-training. We ablate several design choices and find that intermediate layers of the pretrained video transformer provide better features for control compared to the final layers, motivating an early-readout design that further improves efficiency and performance. On standard simulation benchmarks like LIBERO and RoboTwin, SLAP remains competitive with strong VLA and WAM baselines that train on 50× more robot-data. On the perturbation-heavy LIBERO-Plus benchmark, SLAP substantially outperforms pixel-level prediction. Finally, we show that SLAP can be deployed for real-world robot manipulation with as little as 30 demonstrations. SLAP is 1.9× faster than Fast-WAM, 2.2× faster than Flex-π Joint, and 19× faster than LingBot-VA, while remaining comparable in speed to .

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, World-Action Model, Video Generative Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Qm9COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents a semantic latent-based WAM. In other words, the vision branch of WAM is trained in DINO/SigLIP/V-JEPA space instead of VAE space. Besides, empirical experiments show that learning efficiency is improved and that the intermediate layer is more meaningful than the final layer. Experiments further demonstrate that this design achieves comparable results.

Critical strengths

The method is simple and easy to follow, and the paper’s presentation is clear.

Critical weaknesses

My main concern is the effectiveness. SLAP only conducts comparison on LIBERO and robotwin, but both of them are lay behind SOTA method. The reason for this is not pretrained on robot data, which does not make sense.
There are some inconsistency in the paper presentation, like hide the results in LIBERO comparison table but claim in your abs .

Questions for authors
  1. There are some baselines listed in the abstract, but why are only two listed in the teaser figure? Where is LingBot-VA in the figure? Is it also missing from Table 1?

  2. It does not make sense to attribute SLAP’s underperformance relative to previous SOTAs to the lack of pretraining on robot data. Why was the model in this work not pretrained to make it comparable to the other baselines? The performance gap widens further on RoboTwin. Could the real-world experimental results be presented in a new quantitative table using success rates?

  3. How should we understand the limitation mentioned in the introduction? As far as we know, the base model used in this work uses a VAE to encode pixels into high-level features and performs denoising and inference in latent space, not pixel space. Am I missing something? Even if you perform denoising in DINO space, do you still need to use flow-matching schedulers?

  4. How are the frame-wise DINO/SigLIP/V-JEPA features aligned with the original temporally compressed VAE space?

  5. Why not simply align with DINO/SigLIP/V-JEPA using a REPA-style approach [1]? In this scenario, the priors may be better exploited.

  6. Could the predicted DINO/SigLIP/V-JEPA features be decoded into pixel space, as in RAE?

  7. I think there may be a gap between the original Wan2.2 priors and the newly learned high-level feature space, which could lead to the degradation of the original Wan2.2 priors.

[1] REPRESENTATION ALIGNMENT FOR GENERATION: TRAINING DIFFUSION TRANSFORMERS IS EASIER THAN YOU THINK

Reviewer_oAXZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Semantic Latent Action Prediction (SLAP), a World-Action Model (WAM) that replaces pixel-level future video prediction with semantic latent prediction. Built on a pretrained Wan2.2 video diffusion backbone, SLAP jointly predicts future representations from frozen visual encoders (DINOv3, SigLIP, or V-JEPA) and robot actions. By directly regressing semantic latents in a single forward pass and reading out from intermediate transformer layers, the method reduces inference computation while retaining the benefits of predictive visual supervision.

Experiments on LIBERO, LIBERO-Plus, RoboTwin, and real-world manipulation demonstrate competitive policy performance, improved robustness to environmental perturbations, and substantially reduced inference latency. On LIBERO-Plus, SLAP achieves 80.8% success compared with 70.2% for Fast-WAM, while achieving 1.9× lower latency. The paper further reports successful real-world adaptation using only 30 demonstrations per task.

Critical strengths

A well-motivated direction for adapting video models to robot control.

The paper addresses an important question: whether pixel-level future prediction is necessary for learning effective robot policies. Replacing pixel reconstruction with semantic representation prediction is a reasonable approach to reducing sensitivity to irrelevant visual details while retaining useful predictive supervision.

Critical weaknesses
  1. Limited methodological novelty. The central idea of predicting future semantic representations instead of pixels is closely related to existing approaches in predictive representation learning, including JEPA-style objectives and representation-based robot policies. Similarly, intermediate-layer feature extraction and caching visual representations are established techniques.

The paper's contribution is primarily the integration of these ideas into a pretrained video diffusion backbone for robot control. While effective, the methodological novelty appears relatively incremental.

  1. The contribution of future semantic prediction is not sufficiently isolated. A central claim is that predicting future semantic latents enables more robust control. However, the experiments do not clearly establish whether the improvements arise from future prediction or simply from introducing semantic representation supervision.

In particular, the authors should compare against predicting current-frame semantic representations, as well as directly aligning intermediate features with frozen semantic encoders without predicting the future.

These comparisons would help establish whether learning future scene dynamics is actually necessary for the observed improvements.

Questions for authors
  1. Is future prediction necessary? How does SLAP compare with predicting current-frame semantic features or directly aligning intermediate features with frozen semantic encoders? This seems essential for establishing the contribution of predictive world modeling.

  2. What drives the efficiency improvements? Can the authors disentangle the contributions of single-pass prediction, visual feature caching, and intermediate-layer readout through controlled comparisons?

  3. How does the choice of semantic encoder affect manipulation precision? High-level semantic representations may discard fine-grained geometric information required for contact-rich manipulation. Have the authors evaluated tasks requiring precise spatial alignment, subtle object-state changes, or force-sensitive interactions?

  4. How robust is the method to different training-data scales? The appendix reports preliminary scaling experiments with 10–50 demonstrations per task. Does the advantage over pixel-level prediction persist with substantially larger robot datasets?

Reviewer_qrEqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Semantic Latent Action Prediction (SLAP), a World-Action Model that replaces future pixel-level video prediction with future semantic latent prediction. Built upon a pretrained video diffusion Transformer, SLAP uses frozen visual encoders to extract semantic representations from future video frames and directly regresses these representations while retaining a flow-matching objective for action prediction. To improve inference efficiency, the method processes the visual stream in a single forward pass, caches visual representations for subsequent action generation, and adopts an intermediate read-out design to reduce the computational cost of the video backbone.

Critical strengths
  1. Efficient integration of semantic prediction into a pretrained video-based WAM. The paper integrates semantic latent supervision into a pretrained video diffusion backbone while retaining the standard action flow-matching objective. The overall design is relatively straightforward and provides a practical approach to adapting video-pretrained models for robotic manipulation.
  2. Favorable performance–efficiency trade-off. By combining single-pass visual processing, intermediate read-out, and visual KV caching, SLAP substantially reduces inference latency while maintaining competitive manipulation performance. The results demonstrate practical value for deploying video-pretrained WAMs in robotic control.
Critical weaknesses
  1. The novelty relative to closely related work is not sufficiently established. Several core ideas in SLAP overlap with existing research. VLA-JEPA has already investigated predicting future latent states to improve robustness in robotic control. Flex-π has incorporated future DINO semantic feature prediction into World-Action Models. Fast-WAM has also demonstrated that joint video–action training can benefit action generation without requiring explicit future video generation at inference time. Although SLAP combines these ideas in a particular architecture, the paper does not sufficiently clarify its conceptual and methodological novelty relative to these closely related approaches. The distinction between the proposed method and existing efficient WAM designs needs to be better established.
  2. The contributions of the individual design choices are not sufficiently disentangled, and the necessity of future semantic prediction remains unclear. SLAP combines semantic latent supervision, direct regression, and intermediate read-out within a pretrained video diffusion model. However, the paper does not fully isolate the contributions of these components to robustness, action prediction performance, and inference efficiency. More importantly, the semantic latent prediction decoder is not used during inference. Although the existing ablations compare semantic supervision against video flow matching and action-only training, they do not establish whether predicting future semantic representations is essential. It remains unclear whether the observed improvements arise from learning predictive information about future scene dynamics or primarily from the regularization effect of semantic representation supervision.
Questions for authors
  1. Is predicting future semantic representations necessary for the observed performance improvements? Could the authors provide a controlled comparison between future semantic latent prediction and current-frame semantic representation supervision, using the same architecture, training data, and optimization budget? Such an experiment would help determine whether the improvements arise from learning predictive information about future scene dynamics or simply from the auxiliary semantic supervision.
  2. What are the individual contributions of the proposed design choices to inference efficiency? The reported inference speedup may result from several components, including single-pass visual processing, intermediate-layer truncation, and visual KV caching. Could the authors provide a more detailed latency breakdown and controlled comparisons to quantify the speedup attributable to each component? This would help clarify how much of the reported efficiency improvement can be attributed specifically to the proposed semantic latent prediction formulation.
  3. How comparable are the training conditions across the main baselines? Could the authors provide more details about the amount of downstream fine-tuning data, training budgets, and relevant model configurations used for each compared method? In particular, clarifying differences in robot-data pretraining and downstream adaptation would help assess the fairness of the main comparisons and the extent to which the reported performance can be attributed to the proposed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.