acceptodds
Withdrawn from ICLR 2027

ReWAM: Rethinking Representations for World-Action Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World-action models jointly learn robot policies and predict future observations, making the representation space an interface between control and prediction. We study the design of this space through controlled comparisons, finding that neither reconstruction fidelity nor pre-trained perceptual features alone ensure effective policy learning. These findings motivate ReWAM, a representation-centric world-action model built on pre-trained DINO features. Feature Calibration and a Temporal Representation Bottleneck organize these features into compact world states suited to dynamics modeling. Action-Grounded Representation Shaping routes only action-loss gradients to the bottleneck, thereby letting the policy shape what the representation encodes while the world model learns how it evolves. Without generative video pre-training, ReWAM achieves 93.6% success on RoboTwin 2.0. On RoboDojo, it achieves an average score of 12.29 and a success rate of 8.28% using approximately 600 hours of embodied pre-training data.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World-action models, Representation learning, Robot manipulation, World models.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5LNZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on representations of world-action models. Its core idea is to use the action objective to form a compact temporal representation. The world model predicts its evolution, with prediction loss forbidden from modifying the representation. Built on fastWAM’s framework, the model substitutes vision features with frozen DINOv3 features. It incorporates feature calibration using multi-layer aggregation, normalization and a representation-aware noise schedule and a Temporal Representation Bottleneck to encode continuous multi-view observations into compact states. It also implements Action-Grounded Representation Shaping, which restricts bottleneck updates exclusively to gradients from the action loss. This design lets the policy decide what information the representation holds, while the world model learns how this information evolves over time. During inference, ReWAM produces actions directly, balancing the inference efficiency and manipulation performance.

Critical strengths
  • This work presents a clear principle for representation design. It studies representation construction for joint world prediction and robot control. Its core idea is to use the action objective to shape representations while the world model learns dynamics.
  • Ablation studies show strong benefits for this controlled representation design. The experiments test feature calibration, temporal compression, representation dimension, and gradient routing. Specifically, the gradient-routing ablation validates AGRS' effectiveness in this framework.
  • Both simulation and real robots show effectiveness of the proposed method. The paper uses two simulation benchmarks and real robotic manipulation tasks. Real-robot results indicate advantages for deformable objects and multi-stage tasks.
Critical weaknesses
  • Generalization to other encoders and architectures remains unproven. The results are mainly for the DINO + Fast-WAM setup. TRB and AGRS have not been validated on VAE latents or other pre-trained features, which limits the method’s applicability as a general representation approach for WAMs.
  • The contribution of world co-training to the complete method remains unclear. Table 9 removes the world prediction loss only from the raw DINO configuration, showing a modest benefit in that setting. It does not establish whether world co-training remains beneficial after feature calibration, TRB, and AGRS are introduced, leaving its contribution to the final ReWAM performance unresolved.
  • The statistical significance of simulation performance gains is unclear. Table 1 shows a small improvement from 92.8% to 93.6% over the strongest baseline. Without results across independent training seeds or confidence intervals, we cannot tell whether this gain comes from the method itself or random noise. The nearly saturated performance further weakens this comparison. Evaluations in less saturated settings — for instance, training on clean demonstrations and testing on both clean and randomized scenes — would better demonstrate robustness and generalization.
Questions for authors
  • Regarding generality, which components are DINO-specific, and which should transfer to other encoders or WAM architectures? Could the authors provide a targeted alternative-encoder experiment or clarify the architectural assumptions behind AGRS?
  • To address the attribution concerns, how does complete ReWAM perform without the world loss? Additionally, in Table 4, reconstruction-pretrained bottlenecks are frozen during WAM training, whereas AGRS updates the bottleneck using action loss. How would reconstruction-pretrained bottlenecks perform if they were also allowed to adapt through action loss?
  • Do representation and gradient-routing gains persist with longer training and across independent seeds? Please also clarify whether the Fast-WAM baselines match model capacity, initialization, and training budget, distinguishing generative video pretraining from embodied pretraining. Because fastWAM performances in Table 1 and Table 3 are inconsistent.
Reviewer_pW8GOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the representation space of world-action models (WAMs). It argues, through controlled comparisons on RoboTwin 2.0, that neither reconstruction-oriented Video-VAE latents nor raw DINO features are ideal. ReWAM builds on Fast-WAM's world/action DiT architecture over frozen DINOv3 features, adding (i) RAE-style feature calibration (multi-layer aggregation, normalization, a dimension-aware noise schedule), (ii) a Temporal Representation Bottleneck (TRB) that compresses frame pairs into 128-d tokens, and (iii) Action-Grounded Representation Shaping (AGRS), which stops world-loss gradients at the TRB so that only the action loss shapes it. Without generative video pre-training, ReWAM reaches 93.6% on RoboTwin 2.0, scores 9.42/5.82% on RoboDojo, and beats Fast-WAM on three real-robot tasks

Critical strengths
  1. A useful controlled study of the representation space. Under the same 1-epoch budget, it separates reconstruction fidelity from control utility. Raw DINO beats Video-VAE latents despite lower SSIM, and a DINO reconstruction codec beats an RGB one. It also shows perceptual features aren't ready-made prediction targets. This challenges the default of building WAMs on video-generator latents.

  2. An actionable finding on who should shape the representation. The more world-loss gradients are allowed into the representation, the worse it gets;; Shaping it with the action loss alone still works at 64 dimensions, and the resulting representation encodes motion direction better than Video-VAE's. The design rule is concrete: the policy decides what gets encoded, and the world model only learns how it evolves.

Critical weaknesses
  1. The results of this paper are not surprising for world models or representation learning, e.g., reconstruction fidelity ≠ control utility, collapse when the world loss can reshape the representation, action-only training aligns the representation with action direction. Too many papers (JEPA, BYOL, WAMs) directly or indirectly demonstrate them.

  2. The analysis is too narrow and confounded to support its general claims. All representation comparisons and ablations come from one benchmark (RoboTwin 2.0), one training epoch, and no reported seeds, while many differences are 1–4 points. The "reconstruction vs. control" conclusion rests on one Video-VAE and one perceptual encoder (DINOv3-L), which also differ in capacity, dimensionality, and pre-training data, and the VAE is tested without the generative pre-training it is designed for. SSIM from a detached probe is the only fidelity measure, and the structure analysis uses t-SNE and 10-NN agreement on two tasks. Other encoders (e.g., SigLIP, V-JEPA), a second benchmark, and multiple seeds are needed before these findings can guide WAM design.

  3. The "world model" half of the conclusion is not established. The paper's design rule is that the world model learns how the representation evolves. Yet AGRS blocks world-loss gradients from the representation; future and action tokens cannot attend to each other, and no future is predicted at inference. The only world-loss ablation (App. C) is on the raw DINO baseline, not on ReWAM. Without it, the analysis cannot tell whether world prediction contributes at all, which undercuts the paper's framing as a study of world-action models.

Questions for authors
  1. How does ReWAM perform with the world loss removed, all else fixed? This decides whether world prediction contributes at all.
  2. What's the clear novelty of ReWAM compared to prior work?
Reviewer_xUztOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies representation design for world-action models and proposes ReWAM, which builds compact states from frozen DINOv3 features through feature calibration and a Temporal Representation Bottleneck (TRB). Its central design, Action-Grounded Representation Shaping (AGRS), allows only action-loss gradients to update the TRB, while stopping world-loss gradients through both current and future representations. The model jointly learns future-state prediction and action generation. Following a Fast-WAM-style inference procedure, it generates actions from current-state features without explicitly generating future states. The approach is evaluated on RoboTwin 2.0, RoboDojo, and three real-robot manipulation tasks.

Critical strengths
  1. The paper addresses a relevant representation-design problem. World prediction and policy learning share a representation interface but may favor different information. Studying feature calibration, temporal compression, and gradient routing within this setting provides useful practical insights.
  2. The key design choices receive meaningful ablation support. Table 5 shows that preventing world-loss gradients from updating the TRB substantially improves policy performance. Table 6 further shows that AGRS outperforms fixed random projection at the same 128-dimensional token width, supporting the effectiveness of action-supervised representation adaptation.
  3. The evaluation is broad and the results are competitive. The paper includes two simulation benchmarks and real-robot experiments, as well as a comparison with Fast-WAM using the same embodied pre-training dataset. The results demonstrate strong manipulation performance without generative video pre-training.
Critical weaknesses
  1. The methodological contribution appears relatively incremental. Prior work such as LDA-1B already performs future-state prediction and action modeling in DINO feature space. The main additions therefore lie in TRB and AGRS. TRB combines spatiotemporal patchification, attention, and linear compression, while AGRS introduces an action-supervised bottleneck with selective gradient blocking. I acknowledge the empirical effectiveness of these choices. However, I view the contribution primarily as representation adaptation and optimization within an existing WAM framework, with relatively limited methodological novelty. This is my main reservation.
  2. The contribution of world co-training in the complete model remains insufficiently isolated. Appendix C provides useful evidence that the world loss improves the raw DINO baseline, but this configuration does not include TRB or AGRS. Introducing an action-supervised bottleneck may change the interaction between policy learning and world prediction. A comparison that disables the world loss in the complete ReWAM configuration, while retaining the architecture and current-state processing path, would help establish whether world co-training provides additional policy benefits once the proposed representation adaptation is enabled.
  3. The reported improvements would be more convincing with confidence intervals or multi-seed experiments.
Questions for authors

Please see the weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.