acceptodds
Under review as a conference paper at ICLR 2027

Rethinking the Representation Space for World-Action Models: Pixels, VAE Latents, or Semantic Representations?

31 views

Abstract

World-action models (WAMs) built on video generation models learn interaction dynamics that map observations to robot actions. Because video generators can produce implausible or hallucinated futures under perturbation, many recent WAMs add auxiliary supervision on other representations to improve robustness under distribution shift. These additions increase post-training cost and introduce trade-offs among loss weights, while leaving open where generalization comes from. We instead ask a more basic question: in which representation space should a WAM predict the future? Within a shared framework and without auxiliary objectives, we compare models that predict the future in pixels (Pixel-WAM), VAE latents (VAE-WAM), or semantic features from frozen pretrained encoders (RAE-WAM). Across six fusion designs, hidden-state fusion transfers video pretraining most effectively. Following this, the V-JEPA RAE-WAM reaches 55.2% success from scratch, compared with 45.7% for the DINOv2 RAE-WAM, 45.4% for VAE-WAM, and 37.2% for Pixel-WAM on LIBERO-Plus. The same representation ordering holds in clean simulated bimanual manipulation, and on a real dual-arm platform the V-JEPA RAE-WAM outperforms π0.5 and DreamZero. Our results point to a simple principle for building world-action models: rather than reconstructing pixels or relying on auxiliary supervision, predict the future in a pretrained semantic space whose structure already reflects how scenes evolve under action.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world-action model, robotic control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_M55NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how future-prediction representations and video-action fusion affect world-action models. It first compares six fusion designs within VAE-WAM, with random and LingBot-Video initialization, finding the highest final success with hidden-state fusion. A separate comparison fixes MoT-joint fusion and the action-expert initialization while predicting pixels, Wan2.1 VAE latents, DINOv2 features, or V-JEPA 2.1 features. On 1,204 LIBERO-Plus variants, V-JEPA achieves 55.15% success versus 45.43% for VAE, 45.68% for DINOv2, and 37.21% for pixels. ABC-130K pretraining raises V-JEPA success to 62.96%. Additional experiments examine clean/randomized RoboTwin and compare the selected model with π0.5 and DreamZero on three physical dual-arm tasks.

Critical strengths
  1. Meaningful control of the policy-side comparison. Q2 keeps the action architecture and initial parameter state, trunk architecture, temporal slots, token budget, data, and fusion design fixed. The paper explicitly characterizes complete representation variants rather than an encoder-only intervention and discloses the distinct Pixel-WAM flow recipe. Appendix A also addresses future leakage by constructing V-JEPA’s conditioning slot from a clip repeating only the first frame. These choices make the representation comparison informative and reduce several important control-side confounds.

  2. Useful closed-loop evidence with transparent limiting results. The paired LIBERO-Plus comparisons show a substantial V-JEPA advantage under MoT-joint fusion. The DINOv2–VAE tie, insignificant pixel bottleneck improvement, and failure of all evaluated variants under RoboTwin randomization are reported rather than suppressed. The fusion study also demonstrates that access to a video prior depends substantially on how it is connected to the action expert. These findings provide useful practical guidance while avoiding a blanket claim that all semantic representations outperform reconstruction latents.

Critical weaknesses
  1. The general representation recommendation and mechanism exceed the demonstrated scope. Q1 identifies hidden-state fusion as strongest within VAE-WAM, but Q2 fixes MoT-joint. Figure 5’s hidden-state LIBERO-Plus points appear to rank VAE slightly above V-JEPA, yet their exact values and matched training conditions are not given. Equal four-frame aggregation does not isolate temporal predictive structure because encoder pretraining, architecture, feature dimensions, normalization, and temporal construction also differ. The codec probe does not explain the V-JEPA–VAE gap, as the authors acknowledge. These results support a particular representation configuration more securely than a fusion-independent preference or the proposed mechanism. This distinction is consequential because the paper’s main contribution is a general design principle for choosing prediction spaces.

  2. The additional validation and implementation are insufficiently specified. Appendix Table 2 does not explain how the 64-channel V-JEPA features are constructed or how semantic representations are decoded for Figure 5’s PSNR/SSIM/LPIPS measurements. A complete training and xDoF pretraining recipe is not provided. The real-robot section and Appendix B omit trial counts, completion criteria, ID/OOD definitions, error-bar meanings, and baseline adaptation data and budgets. Consequently, the fairness and uncertainty of the real-robot advantage cannot be assessed. The LIBERO statistics establish fixed-model evaluation differences, but not training-seed stability; the paper appropriately acknowledges this limitation without resolving it. These omissions limit both reproducibility and the strength of the claimed transfer evidence.

Questions for authors
  1. Representation–fusion interaction: Please provide exact values and initialization/training protocols for all Setting A models in Figure 5. Does the apparent VAE–V-JEPA reversal under hidden-state fusion persist under matched conditions, and how does it affect the representation recommendation?

  2. Mechanism and implementation: How are the 64-channel V-JEPA features and semantic decoders constructed, and what are the complete training/xDoF recipes? What controlled evidence separates temporal predictive structure from other encoder and interface differences, and how stable is the central ranking across training seeds?

  3. Real-robot validation: What are the trial counts, completion criteria, ID/OOD changes, and error-bar definitions? Which checkpoints, observations, demonstrations, and adaptation budgets were used for each baseline?

Reviewer_1juNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how different future-prediction representation spaces affect the performance of World-Action Models (WAMs). The authors compare pixel-space prediction, reconstruction-oriented VAE latents, and semantic representations from DINOv2 and V-JEPA within a shared world-action framework. They also investigate six video-action fusion architectures and evaluate the resulting models on LIBERO-Plus, RoboTwin, and real-world bimanual manipulation tasks.

Critical strengths
  1. The paper studies an important but relatively underexplored design choice in world-action models: the representation space used for future prediction.
  2. The authors evaluate multiple representation spaces and fusion architectures, including closed-loop evaluations on LIBERO-Plus, RoboTwin, and a real dual-arm robot platform.
Critical weaknesses
  1. The paper suffers from several presentation issues that affect readability. For example, Q1 and Q2 are repeatedly referenced in Section 3 before being formally introduced in Section 4. Figure 1 is never referenced or discussed in the main text, and its descriptions (e.g., "Tangled Mapping" and "Smooth, Order-Preserving") are neither clearly explained nor substantiated. Moreover, the experimental analysis is overly dense with numerical results and statistical details, often repeating information from tables rather than highlighting the main findings and their implications. These issues make the paper difficult to follow and obscure its key insights.
  2. The main representation comparison uses MoT-joint fusion, which Table 1 shows is clearly weaker than hidden-state fusion within VAE-WAM, so the ranking may not hold under the best design. Moreover, the best LIBERO-Plus result in the paper is a pretrained VAE-WAM, which exceeds every RAE-WAM result on that benchmark and is hard to reconcile with the conclusion against VAE latents.
  3. The real-robot experiments only evaluate the selected V-JEPA RAE-WAM against π0.5 and DreamZero, without comparing Pixel-WAM and VAE-WAM under the same real-world setup. While these experiments demonstrate promising overall policy performance, they do not directly establish that the representation-space ranking observed in simulation transfers to real-world manipulation.
Questions for authors
  1. Given that hidden-state fusion significantly outperforms MoT-joint fusion in Table 1, can the authors evaluate the representation variants under hidden-state fusion on LIBERO-Plus? Does the same representation ranking hold?
  2. Can the authors evaluate additional representation variants on the real robot, or clarify the extent to which the simulation findings generalize to physical manipulation?
  3. The reported shared-path latency excludes representation-specific encoder costs. End-to-end inference latency would be useful for understanding the practical trade-offs among Pixel-WAM, VAE-WAM, and RAE-WAM.
Reviewer_UbSnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the representation space used for future visual prediction affects the closed-loop control performance of world-action models (WAMs). Within a shared video–action modeling framework, the authors compare pixels, VAE latents, and semantic features from DINOv2 and V-JEPA, alongside six video–action fusion designs. V-JEPA achieves the highest success rate under MoT-joint fusion on LIBERO-Plus, with additional robot-video pretraining further improving its performance. Hidden-state fusion performs best in the evaluated VAE-based pretraining setting, while the V-JEPA variant also demonstrates competitive performance on clean RoboTwin and real-world bimanual manipulation tasks. The study aims to provide empirical guidance for representation selection and fusion design in WAMs.

Critical strengths
  1. The research question is practically relevant. Treating the future prediction space as an explicit design variable helps clarify how representation choices affect closed-loop control performance in WAMs.
  2. The main comparison includes several useful controls. Q2 shares the action-expert initialization, backbone architecture, token budget, and evaluation tasks across variants, and reports paired statistical tests. The approximately 9.7 percentage-point improvement of V-JEPA over VAE is noteworthy.
  3. The paper is transparent about several limitations. The authors explicitly acknowledge that DINOv2 and VAE perform similarly overall, that the representation probe does not establish a causal mechanism, and that all evaluated variants nearly fail under severe visual randomization in RoboTwin. These negative results are also informative.
Critical weaknesses
  1. The interaction between representation choice and fusion design remains insufficiently explained. Q1 identifies hidden-state fusion as the best-performing design, whereas the main representation comparison in Q2 uses MoT-joint fusion. In the LIBERO-Plus hidden-state fusion results shown in Figure 5, VAE and V-JEPA achieve similar success rates, with VAE having a slightly higher point estimate. This differs from the substantial advantage of V-JEPA under MoT-joint fusion. It therefore remains unclear to what extent V-JEPA’s advantage depends on the fusion design.
  2. The real-robot experimental protocol is insufficiently specified to assess the fairness and statistical reliability of the comparison. The paper reports that the selected model outperforms π0.5 and DreamZero on three tasks, but does not clearly specify the baselines’ fine-tuning protocols, training data quantities, ID/OOD settings, number of evaluation trials, or the definition of completion rate. The meaning of the error bars in the figure also needs clarification. Please provide these details.
Questions for authors

Please see the weakness section

Reviewer_qjMNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks which space a world-action model should predict the future in. The authors keep one framework (a video DiT trunk plus an action DiT) and only swap the visual codec between raw pixels, Wan2.1 VAE latents and frozen DINOv2 or V-JEPA 2.1 features, with no auxiliary losses. They first compare six ways of fusing the video and action streams on LIBERO-Plus with VAE-WAM, and find that hidden-state fusion with a stop-gradient is best, both with LingBot-Video init (81.06) and from scratch (58.47). Then they compare the representations using MoT-joint fusion, trained from scratch. V-JEPA comes out on top at 55.15, DINOv2 and VAE are basically tied around 45.5, and pixels are last at 37.21. Pretraining the V-JEPA model on ABC-130K brings it up to 62.96. On clean RoboTwin the same ordering shows up (V-JEPA 80.86, VAE 76.00, Pixel 47.80), although everything drops below 2% once randomization is on. On a real dual-arm setup their model beats π0.5 and DreamZero on three tasks. The conclusion they draw is that WAMs should predict in a video-pretrained semantic space rather than in pixels or VAE latents.

Critical strengths
  1. I appreciated how controlled the setup is, more so than most WAM papers I've read. The trunk, action pathway and token budget are shared, the action DiT init is literally copied across arms, and they use paired McNemar tests and even re-run the same weights to estimate eval noise. They are also upfront about what they could not control, e.g. the separate Pixel recipe, and they don't oversell the DINOv2 vs VAE tie.

  2. For me the Q1 fusion study is the most useful part. Hidden-state fusion with stop-gradient wins with and without pretraining, and not by a small margin (81.06 vs 74.09 for the next best, IDM). I also liked that hidden-state fusion has the worst open-loop action MSE of the six designs but the best closed-loop success. Good reminder not to tune on offline metrics.

Critical weaknesses
  1. My main concern is the fusion choice in Q2. Q1 shows hidden-state fusion is clearly the best design, and MoT-joint is actually the worst one from scratch (45.43). But the LIBERO-Plus representation comparison in Q2 is run with MoT-joint, and I couldn't find a reason for this choice in the paper. So the main result is obtained under a fusion design the paper itself shows is weak, and it's not clear the ranking would hold under hidden-state fusion. Figure 5 actually shows hidden-state results on LIBERO-Plus, where VAE appears slightly ahead of V-JEPA, although the difference is small and no significance test is reported. On clean RoboTwin, V-JEPA leads VAE (80.86 vs 76.00), but all models fail once randomization is on. On top of that, the best LIBERO-Plus number in the paper (81.06) is a VAE-WAM, though with LingBot-Video init. So I'm not convinced the "simple principle" in the abstract is supported yet.

  2. Even if the ranking held, I don't think the paper isolates why V-JEPA wins. V-JEPA 2.1 and DINOv2 differ in many ways besides temporal structure, like pretraining data, feature dimension and how frames are encoded (V-JEPA sees the whole clip jointly, while DINOv2 encodes frames one by one and averages them). No training-seed variance is reported either. So attributing the gap to "video-predictive temporal structure" feels like a stretch to me. Related work the paper cites (OpenWAM, Nilaksh et al., ARPS) already points in a similar direction, so a cleaner isolation is what would really make this paper stand out.

  3. Minor stuff. The appendix looks rushed. Section 3 says the optimization configs are in Table 2, but there is no learning rate, batch size or number of training steps in it. Appendix B is called "Real-Robot Setup and Tasks" but it's just a latency paragraph and two figures, nothing about the protocol or number of trials. The figures are also pretty blurry, Figure 5 especially. And there is a broken sentence around line 79 ("depends on the semantic representation encodes").

Questions for authors
  1. Could you report the Q2 comparison of all four representations with hidden-state fusion on LIBERO-Plus? If the ranking changes there, the main conclusion would need to change too. This is the question most likely to change my score.

  2. Could you give more detail on how the V-JEPA features are extracted? Table 2 lists 64 channels for V-JEPA 2.1 against 1024 for DINOv2, and it's not clear to me where that comes from.

  3. Any chance of a second training seed for the Q2 arms? The eval noise analysis is nice, but it doesn't say anything about training variance.

  4. On the real robot, how many demos and trials per task? Were π0.5 and DreamZero fine-tuned on the same data with a similar budget? And is the real-robot model the ABC-130K pretrained one?

1 comment

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.