- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents a systematic empirical study of how different visual representation spaces affect World Action Models (WAMs). Using an existing cascaded WAM framework, the authors compare four frozen visual encoders: Stable Diffusion 3.5 VAE, DINOv2, DINOv3, and SigLIP 2. Flow matching is used to predict future visual features, which are then converted into actions through an inverse dynamics model (IDM). The paper investigates future prediction, action decoding, and closed-loop control performance, as well as the effects of training data, computational budgets, and pretraining on additional tasks. Experiments suggest that semantic representations generally outperform VAE latents in the evaluated settings, particularly for free-object manipulation.
Critical strengths
(1) Valuable research question. The paper investigates which visual representations are more suitable for WAMs and how representation choice affects future prediction and action generation. This is an important question for robotic world modeling.
(2) Systematic empirical evaluation. The authors compare DINOv2, DINOv3, SigLIP 2, and VAE representations under different training data scales, computational budgets, and pretraining configurations. Results across multiple random seeds further strengthen the evaluation.
(3) Meaningful analysis of performance differences. Table 4 shows that the performance gap between VAE and semantic representations becomes substantially smaller when the IDM is provided with ground-truth future features. This experiment helps identify the importance of future prediction rather than relying solely on final task success rates.
(4) Evaluation across different settings. Beyond RoboCasa365, the paper trains on real-world DROID demonstrations and evaluates zero-shot generalization on RoboLab, providing additional evidence across different training and evaluation settings.
Critical weaknesses
(1) Limited methodological novelty. The paper mainly adopts an existing cascaded WAM framework, standard flow matching, and pretrained visual encoders. Its main investigation consists of replacing visual encoders and comparing their performance. No clearly new algorithm, learning objective, or modeling mechanism is introduced. Therefore, the contribution is primarily empirical rather than methodological.
(2) The investigation of different representation spaces largely amounts to an encoder comparison. Different visual encoders naturally map images into different feature representations. However, changing the pretrained encoder within an existing WAM does not itself constitute a new technical method. The authors should clarify whether the paper provides any independent methodological contribution beyond encoder replacement or establishes sufficiently distinctive empirical findings.
(3) Insufficient differentiation from prior work. Previous studies, including DINO-WM, EgoWAM, and OpenWAM, have already investigated semantic representations in world modeling or WAMs. The paper does not sufficiently clarify which important new insights its systematic comparison establishes beyond these existing studies.
(4) Incomplete isolation of representation-space effects. The compared encoders differ substantially in feature dimensions, parameter counts, and pretraining objectives. Some flow-matching training configurations are also tuned separately. Therefore, the reported performance differences may reflect not only semantic representation properties but also representation capacity and optimization choices.
(5) Limited representativeness of the VAE baseline. The paper uses only Stable Diffusion 3.5 VAE as the representative reconstruction-oriented encoder. Other VAEs, particularly those used in existing video-generation-based WAMs, are not evaluated. Consequently, the current evidence does not fully establish that semantic representation spaces generally outperform VAE latent spaces.
(6) Insufficient controlled ablations. Although the paper provides extensive performance analyses, additional controlled experiments are needed to separate the effects of feature dimensionality, training configurations, and representation types. Without these controls, the underlying causes of the observed performance differences remain incompletely understood.
Questions for authors
(1) Methodological novelty and contribution. The paper primarily adopts an existing cascaded WAM framework, flow matching, and pretrained visual encoders. Could the authors clarify the independent contribution beyond systematically comparing these encoders? In particular, what important findings does this study establish that have not already been demonstrated by DINO-WM, EgoWAM, or OpenWAM?
(2) Representation spaces versus encoder differences. The paper attributes performance differences to representation spaces, but the evaluated encoders also differ in feature dimensionality, parameter counts, and pretraining objectives. Could the authors conduct dimension-matched or other controlled comparisons to better distinguish the effects of semantic information from those of representation capacity?
(3) Representativeness of the VAE baseline. Could the authors include additional reconstruction-oriented encoders, particularly VAEs used by existing video-generation-based WAMs? This would help determine whether the observed results reflect a general difference between semantic and reconstruction-oriented representations or are primarily associated with the selected SD3.5 VAE.
(4) Influence of training configurations. The appendix indicates that different flow-matching time-shift settings are used for semantic representations and the VAE. Could the authors provide a sensitivity analysis of this parameter and clarify whether the representation ranking remains consistent across reasonable optimization settings?
(5) Understanding the performance gap. Table 4 shows that the performance gap between VAE and semantic representations becomes substantially smaller when the IDM receives ground-truth future features. Could the authors further investigate whether the remaining differences primarily arise from the predictability of future representations, optimization difficulty across feature spaces, or other factors? Such an analysis would strengthen the central empirical findings of the paper.
3 comments
W4, W6 & Q2: controlled comparisons
Training configurations. Every representation received the same hyperparameter search (see Q4), and this search selected the same configuration for the three semantic encoders. In the end, a single hyperparameter differs, between the semantic encoders and the VAE: the time shift (64 against 1).
What is controlled. The architecture of the predictor and of the IDM, their trainable parameters (1.01B and 321M, within 1% across representations), the number of tokens per frame (256, and 324 for DINOv2), the data and the number of training steps are the same. The three semantic encoders are also matched with each other in size and feature dimension: all three are ViT-L models with 1024-dimensional features. The encoders are frozen, so their parameters do not enter the trainable budget.
What cannot be controlled, and what we claim. We acknowledge that a perfectly matched comparison of pretrained encoders is not possible: each was trained on different data with a different objective. We made the comparison as fair as is achievable with existing encoders, and we agree with the reviewer on the consequence: our conclusions concern these four spaces as they are available to practitioners, and we do not claim to isolate "semantics" as the causal property.
Two observations argue against a capacity explanation. First, dimensionality works against the semantic spaces: their targets are 16 times larger per token (1024 against 64 dimensions) for the same predictor, and they still reach higher success. The compactness of the VAE is an advantage, and it shows at low compute (Figure 5). Second, the three semantic encoders differ from each other in objective (self-supervised or vision-language) and in pretraining data, and all three outperform the VAE.
A dimension-matched control, such as semantic features compressed to 64 dimensions, requires an additional stage, the training of a compression module for each encoder, which is outside the scope of this study. Our interest is in generating directly in the high-dimensional feature space, with the recent architectural developments that make it possible. This control is also beyond the compute available during the rebuttal period: the training recipe depends on the dimension of the target (the time shift in particular), so each new dimension requires its own search, followed by the training of a 1B-parameter predictor and an IDM. Finally, compression changes the representation itself, so it would not isolate dimensionality either.
W5 & Q3: a single VAE
This is a valid point. The VAEs of video generators add temporal compression, which is a confounding factor: it changes the number of predicted tokens and the inputs of the IDM. We therefore compare image encoders only, so that all four spaces are computed per frame, and we chose the SD3.5 VAE as a recent 16-channel image VAE with spatial compression. A fair extension to video would require video encoders on both sides, video VAEs and semantic video encoders, and we do not have the compute to run it within the rebuttal period. We therefore restrict our claim to the VAE we tested.
Q4: time-shift sensitivity
All four representations were searched over the same nine values of the time shift (0.5, 1, 2, 4, 8, 16, 32, 64 and 128), and the value was selected per representation on the validation latent L1, without any closed-loop evaluation. The optimum of the VAE, , lies inside the grid, so the VAE result is not an artifact of an untuned schedule. The sensitivity table is in our response to Reviewer hGCJ (W2 & Q2).
The search also selected the same configuration for the three semantic encoders. On the ranking across settings: it can only be measured in success rate, since latent L1 is not comparable across representations, and we report closed-loop results at the selected setting of each representation.
Q5: source of the gap
Action decoding and optimization difficulty are both ruled out: with ground-truth future latents, the VAE is within about 2 points of the semantic encoders (Table 4), and each representation is trained at its own optimum of the same search (Q4). The gap therefore comes from the predicted futures (Figures 6 and 7; details in our response to Reviewer hGCJ, W1 & Q1). What remains open is how much comes from the prediction errors themselves and how much from the IDM's sensitivity to them.
Changes in the revision. Section 2: differences with DINO-WM, EgoWAM and OpenWAM. Conclusion and limitations: the claims concern these four spaces and the SD3.5 VAE only, with no causal claim about "semantics"; the missing dimension-matched control is listed. We no longer write "VAE-based WAMs" in general.
We hope that these clarifications and additions address the reviewer's concerns. If so, we would be grateful if the reviewer would consider raising their score.
We wanted to check whether our response addressed your concerns. In particular, we listed the findings this study establishes (W1, W2 & Q1) and clarified how it differs from DINO-WM, EgoWAM and OpenWAM, the latter reports the opposite result for uncompressed semantic features.
We also restricted our claims to the four spaces tested, with no causal claim about “semantics”. We are happy to clarify anything in the remaining time.
We thank the reviewer for the detailed review, and for recognizing the value of the research question and of the evaluation.
W1, W2 & Q1: contribution
The reviewer is right that our contribution is empirical. The components we build on exist, namely the cascaded WAM and the recipe for flow matching in high-dimensional feature spaces of RAE (Zheng et al., 2025) and FlowWM (Porcher et al., 2026). What had not been shown is that flow matching in semantic feature spaces yields better WAMs. The current consensus goes the other way: state-of-the-art WAMs such as Cosmos 3 and FLUX 3 Action predict the future in the VAE space of their video generator. Testing this consensus under matched conditions is our contribution. We establish that:
This comparison is also costly, with more than 150 training runs of models of up to 1B parameters on 32 H200 GPUs, which is why we believe that reporting it is useful to the community.
W3 & Q1: DINO-WM, EgoWAM, OpenWAM
EgoWAM and OpenWAM are arXiv preprints, and OpenWAM appeared nine days before the submission deadline, so both fall under the ICLR policy on contemporaneous work. We nevertheless discuss them.