acceptodds
Under review as a conference paper at ICLR 2027

Do Semantic Feature Spaces Make Better World Action Models?

11 views

Abstract

World action models (WAMs) offer a promising approach to robotic control by coupling future visual prediction with action generation. Many existing WAMs predict future states in reconstruction-oriented variational autoencoder (VAE) la- tent spaces, raising the question: do semantic feature spaces provide a better foun- dation for robot control? We investigate this question through a systematic com- parison of the Stable Diffusion 3.5 VAE, DINOv2, DINOv3, and SigLIP 2 within a common cascaded WAM framework. We use flow matching to predict future features directly in each encoder’s representation space and an inverse dynamics model to decode actions from these predictions, with training configurations tuned for each representation. We analyze action decoding, future dynamics prediction, and closed-loop task success across training data and compute budgets. Experi- ments on RoboCasa365 and transfer from real-world DROID data to high-fidelity simulation in RoboLab assess WAMs’ performance and generalization. Across the evaluated setups, semantic-space WAMs achieve higher task success and scale more favorably with compute, data, and pretraining on additional tasks than VAE- based WAMs. These findings support semantic feature spaces as a stronger foun- dation for effective and scalable world action modeling.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action models, visual representation, robot learning, world models, flow matching, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6fvTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a systematic empirical study of how different visual representation spaces affect World Action Models (WAMs). Using an existing cascaded WAM framework, the authors compare four frozen visual encoders: Stable Diffusion 3.5 VAE, DINOv2, DINOv3, and SigLIP 2. Flow matching is used to predict future visual features, which are then converted into actions through an inverse dynamics model (IDM). The paper investigates future prediction, action decoding, and closed-loop control performance, as well as the effects of training data, computational budgets, and pretraining on additional tasks. Experiments suggest that semantic representations generally outperform VAE latents in the evaluated settings, particularly for free-object manipulation.

Critical strengths

(1) Valuable research question. The paper investigates which visual representations are more suitable for WAMs and how representation choice affects future prediction and action generation. This is an important question for robotic world modeling.

(2) Systematic empirical evaluation. The authors compare DINOv2, DINOv3, SigLIP 2, and VAE representations under different training data scales, computational budgets, and pretraining configurations. Results across multiple random seeds further strengthen the evaluation.

(3) Meaningful analysis of performance differences. Table 4 shows that the performance gap between VAE and semantic representations becomes substantially smaller when the IDM is provided with ground-truth future features. This experiment helps identify the importance of future prediction rather than relying solely on final task success rates.

(4) Evaluation across different settings. Beyond RoboCasa365, the paper trains on real-world DROID demonstrations and evaluates zero-shot generalization on RoboLab, providing additional evidence across different training and evaluation settings.

Critical weaknesses

(1) Limited methodological novelty. The paper mainly adopts an existing cascaded WAM framework, standard flow matching, and pretrained visual encoders. Its main investigation consists of replacing visual encoders and comparing their performance. No clearly new algorithm, learning objective, or modeling mechanism is introduced. Therefore, the contribution is primarily empirical rather than methodological.

(2) The investigation of different representation spaces largely amounts to an encoder comparison. Different visual encoders naturally map images into different feature representations. However, changing the pretrained encoder within an existing WAM does not itself constitute a new technical method. The authors should clarify whether the paper provides any independent methodological contribution beyond encoder replacement or establishes sufficiently distinctive empirical findings.

(3) Insufficient differentiation from prior work. Previous studies, including DINO-WM, EgoWAM, and OpenWAM, have already investigated semantic representations in world modeling or WAMs. The paper does not sufficiently clarify which important new insights its systematic comparison establishes beyond these existing studies.

(4) Incomplete isolation of representation-space effects. The compared encoders differ substantially in feature dimensions, parameter counts, and pretraining objectives. Some flow-matching training configurations are also tuned separately. Therefore, the reported performance differences may reflect not only semantic representation properties but also representation capacity and optimization choices.

(5) Limited representativeness of the VAE baseline. The paper uses only Stable Diffusion 3.5 VAE as the representative reconstruction-oriented encoder. Other VAEs, particularly those used in existing video-generation-based WAMs, are not evaluated. Consequently, the current evidence does not fully establish that semantic representation spaces generally outperform VAE latent spaces.

(6) Insufficient controlled ablations. Although the paper provides extensive performance analyses, additional controlled experiments are needed to separate the effects of feature dimensionality, training configurations, and representation types. Without these controls, the underlying causes of the observed performance differences remain incompletely understood.

Questions for authors

(1) Methodological novelty and contribution. The paper primarily adopts an existing cascaded WAM framework, flow matching, and pretrained visual encoders. Could the authors clarify the independent contribution beyond systematically comparing these encoders? In particular, what important findings does this study establish that have not already been demonstrated by DINO-WM, EgoWAM, or OpenWAM?

(2) Representation spaces versus encoder differences. The paper attributes performance differences to representation spaces, but the evaluated encoders also differ in feature dimensionality, parameter counts, and pretraining objectives. Could the authors conduct dimension-matched or other controlled comparisons to better distinguish the effects of semantic information from those of representation capacity?

(3) Representativeness of the VAE baseline. Could the authors include additional reconstruction-oriented encoders, particularly VAEs used by existing video-generation-based WAMs? This would help determine whether the observed results reflect a general difference between semantic and reconstruction-oriented representations or are primarily associated with the selected SD3.5 VAE.

(4) Influence of training configurations. The appendix indicates that different flow-matching time-shift settings are used for semantic representations and the VAE. Could the authors provide a sensitivity analysis of this parameter and clarify whether the representation ranking remains consistent across reasonable optimization settings?

(5) Understanding the performance gap. Table 4 shows that the performance gap between VAE and semantic representations becomes substantially smaller when the IDM receives ground-truth future features. Could the authors further investigate whether the remaining differences primarily arise from the predictability of future representations, optimization difficulty across feature spaces, or other factors? Such an analysis would strengthen the central empirical findings of the paper.

3 comments

We thank the reviewer for the detailed review, and for recognizing the value of the research question and of the evaluation.

W1, W2 & Q1: contribution

The reviewer is right that our contribution is empirical. The components we build on exist, namely the cascaded WAM and the recipe for flow matching in high-dimensional feature spaces of RAE (Zheng et al., 2025) and FlowWM (Porcher et al., 2026). What had not been shown is that flow matching in semantic feature spaces yields better WAMs. The current consensus goes the other way: state-of-the-art WAMs such as Cosmos 3 and FLUX 3 Action predict the future in the VAE space of their video generator. Testing this consensus under matched conditions is our contribution. We establish that:

  • The gains are large. Three semantic spaces improve success by 12.6 to 17.0 points on Target50, and by up to 22.0 points with pretraining, with the same ranking on RoboLab.
  • The advantage depends on the budget. The VAE is the better choice at low training compute, and the ranking reverses only after 1.5×10211.5 \times 10^{21} FLOPs (Figure 5).
  • Semantic spaces scale better. They gain more from pretraining on other tasks (+4.8 to +7.4 points against +2.4) and from more target data (Figure 4).
  • The deficit of the VAE is localized. It does not come from decoding actions out of true latents, and it concentrates on pick-and-place tasks, where the gripper and the object must move together (38.1 points, against 12.5 on articulated parts; "free object" in Table 4, Figure 7).
  • Latent error must be read with care. It tracks success within a representation (Spearman correlation of at least 0.97 in absolute value) and is misleading across representations (Figure 6).

This comparison is also costly, with more than 150 training runs of models of up to 1B parameters on 32 H200 GPUs, which is why we believe that reporting it is useful to the community.

W3 & Q1: DINO-WM, EgoWAM, OpenWAM

  • DINO-WM uses DINOv2 features for goal-conditioned planning: at test time, it searches for the actions that bring the predicted latent closest to a goal image. A WAM has no goal image and no search: it generates a future and decodes actions from it in a feed-forward pass, which requires a good target for generative modeling, not only a space suitable for planning. DINO-WM also does not compare with the VAE spaces of generators.
  • EgoWAM (arXiv, July 2026) uses stronger representations to enrich the features of a policy during training, in the spirit of representation alignment (REPA): an auxiliary loss predicts Wan VAE latents, DINOv2 features or 3D flow while co-training on egocentric human videos. Nothing is generated in that space at inference: the auxiliary head is discarded, and the policy outputs actions directly from an action head. In our study, the WAM generates the future in the representation space at inference, and the actions are decoded from this generated future. We also compare three semantic encoders at the 1B-parameter scale and study the scaling with data, compute and pretraining.
  • OpenWAM (arXiv, 7 September 2026) reaches the opposite conclusion: uncompressed DINOv3 and V-JEPA 2.1 features underperform the VAEs it tests unless they are compressed to 48 dimensions. It describes this as a naive adoption of the encoders, on an architecture designed for VAE latents. We adapt the model instead (wide projection head, shifted noise schedule, clean-sample prediction), and uncompressed semantic spaces then outperform the VAE.

EgoWAM and OpenWAM are arXiv preprints, and OpenWAM appeared nine days before the submission deadline, so both fall under the ICLR policy on contemporaneous work. We nevertheless discuss them.

W4, W6 & Q2: controlled comparisons

Training configurations. Every representation received the same hyperparameter search (see Q4), and this search selected the same configuration for the three semantic encoders. In the end, a single hyperparameter differs, between the semantic encoders and the VAE: the time shift (64 against 1).

What is controlled. The architecture of the predictor and of the IDM, their trainable parameters (1.01B and 321M, within 1% across representations), the number of tokens per frame (256, and 324 for DINOv2), the data and the number of training steps are the same. The three semantic encoders are also matched with each other in size and feature dimension: all three are ViT-L models with 1024-dimensional features. The encoders are frozen, so their parameters do not enter the trainable budget.

What cannot be controlled, and what we claim. We acknowledge that a perfectly matched comparison of pretrained encoders is not possible: each was trained on different data with a different objective. We made the comparison as fair as is achievable with existing encoders, and we agree with the reviewer on the consequence: our conclusions concern these four spaces as they are available to practitioners, and we do not claim to isolate "semantics" as the causal property.

Two observations argue against a capacity explanation. First, dimensionality works against the semantic spaces: their targets are 16 times larger per token (1024 against 64 dimensions) for the same predictor, and they still reach higher success. The compactness of the VAE is an advantage, and it shows at low compute (Figure 5). Second, the three semantic encoders differ from each other in objective (self-supervised or vision-language) and in pretraining data, and all three outperform the VAE.

A dimension-matched control, such as semantic features compressed to 64 dimensions, requires an additional stage, the training of a compression module for each encoder, which is outside the scope of this study. Our interest is in generating directly in the high-dimensional feature space, with the recent architectural developments that make it possible. This control is also beyond the compute available during the rebuttal period: the training recipe depends on the dimension of the target (the time shift in particular), so each new dimension requires its own search, followed by the training of a 1B-parameter predictor and an IDM. Finally, compression changes the representation itself, so it would not isolate dimensionality either.

W5 & Q3: a single VAE

This is a valid point. The VAEs of video generators add temporal compression, which is a confounding factor: it changes the number of predicted tokens and the inputs of the IDM. We therefore compare image encoders only, so that all four spaces are computed per frame, and we chose the SD3.5 VAE as a recent 16-channel image VAE with 8×8\times spatial compression. A fair extension to video would require video encoders on both sides, video VAEs and semantic video encoders, and we do not have the compute to run it within the rebuttal period. We therefore restrict our claim to the VAE we tested.

Q4: time-shift sensitivity

All four representations were searched over the same nine values of the time shift α\alpha (0.5, 1, 2, 4, 8, 16, 32, 64 and 128), and the value was selected per representation on the validation latent L1, without any closed-loop evaluation. The optimum of the VAE, α=1\alpha = 1, lies inside the grid, so the VAE result is not an artifact of an untuned schedule. The sensitivity table is in our response to Reviewer hGCJ (W2 & Q2).

The search also selected the same configuration for the three semantic encoders. On the ranking across settings: it can only be measured in success rate, since latent L1 is not comparable across representations, and we report closed-loop results at the selected setting of each representation.

Q5: source of the gap

Action decoding and optimization difficulty are both ruled out: with ground-truth future latents, the VAE is within about 2 points of the semantic encoders (Table 4), and each representation is trained at its own optimum of the same search (Q4). The gap therefore comes from the predicted futures (Figures 6 and 7; details in our response to Reviewer hGCJ, W1 & Q1). What remains open is how much comes from the prediction errors themselves and how much from the IDM's sensitivity to them.

Changes in the revision. Section 2: differences with DINO-WM, EgoWAM and OpenWAM. Conclusion and limitations: the claims concern these four spaces and the SD3.5 VAE only, with no causal claim about "semantics"; the missing dimension-matched control is listed. We no longer write "VAE-based WAMs" in general.

We hope that these clarifications and additions address the reviewer's concerns. If so, we would be grateful if the reviewer would consider raising their score.

We wanted to check whether our response addressed your concerns. In particular, we listed the findings this study establishes (W1, W2 & Q1) and clarified how it differs from DINO-WM, EgoWAM and OpenWAM, the latter reports the opposite result for uncompressed semantic features.

We also restricted our claims to the four spaces tested, with no causal claim about “semantics”. We are happy to clarify anything in the remaining time.

Reviewer_Y36kOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper conducts a systematic study to answer the question: what future state representation should world action models use? The paper starts by characterizing the two desiderata for WAM state representations: they need to (1) be predictable from context and (2) carry information about actions. Then, the paper introduces a cascade world action model framework consistenting of a separate forward dynamics model and inverse dynamics model. Under this controlled model class, the paper proceeds to compare four state representations: Stable Diffusion 3.5 VAE (effectively pixel reconstruction), DINOv2, DINOv3, and SigLIP 2. Across RoboCasa365 and DROID pretraining + RoboLab sim evaluation, DINOv3 appears to be the best latent representation for this class of world action model, achieving the highest success rates in-distribution and across generalized settings. The main results (Table 2, 3) are corroborated by comprehensive analysis experiments isolating the effects of inverse dynamics model (Figure 3), forward dynamics model (Table 4), pretraining data (Figure 4), training compute (Figure 5), while establishing correlation between closed-loop success rates and latent prediction error (Figure 6).

Critical strengths
  1. The paper presents a systematic, controlled study of the state representations for world action models, which is immensely valuable for practitioners. Specifically, the paper conducts the following experiments (in order presented in the paper):
    1. Isolating the effect of IDM: when the IDM sees ground truth future states (no forward dynamics error), SigLIP 2, DINOv2, and DINOv3 to to have lower action mse than SD3.5 vae.
    2. Full WAM on RoboCasa Target50, with an without pretraining on Pretrain300: DINOv3 performs best across the board, followed closely by SigLIP 2 and DINOv2. SD3.5 VAE performs noticeably worse. Moreover, DINOv3 shows the most gain from pretraining data.
    3. Full WAM with DROID pretraining and RoboLab eval: DINOv3 again shows the best generalization across the board.
    4. Latent prediction error vs success rate: across all representations, there exists a negative correlation between the forward dynamics models' prediction errors and the closed-loop success rates. The lower the prediction error, the higher the success rate.
    5. Why are semantic features better than VAE? The performance gap between DINOv3 and SD 3.5 shrinks when the inverse dynamics model sees the ground truth future features, suggesting that the majority of the performance gap can be explained by the SD 3.5 features being harder to predict.
    6. Conclusion: the overall takeaway is to use semantic features over VAE features for this class of cascading world action models because they are (1) easier to predict from context and (2) carries more infomation about the actions.
  2. The presentation of the paper is excellent. The writing is clear and has a nice flow to it.
Critical weaknesses
  1. My main reservation for this paper is that it feels like a workshop paper. The main conclusion is that Dinov3 is better than pixel reconstrution for WAMs, which could be an ablation of DINO WM [1] or one of the many JEPA papers, with an inverse dynamics model instead of planning. While I appreciate the clarity, I'm leaning toward weak rejection because the paper does not present anything fundamentally new and the conclusion is not very surprising. I'm open to changing my recommendation if the AC decides that the contribution of these experiments warrants an accept to this venue.
  2. The paper doesn't conduct experiments on real robots. The droid data is only used for pretraining, and evaluation is performed on RoboLab. The implications on real robots remain unclear.
  3. For Figure 3 experiments, action MSE is not a good metric of the quality of action generation. In behavior cloning, it is often the case that the closed-loop success rate keeps climbing even though the action MSE has already saturated.

Gaoyue Zhou, Hengkai Pan, Yann LeCun, Lerrel Pinto. DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. 2024

Questions for authors
  1. All the experiments are conducted with a fixed model size. It could be that SD3.5 is hard to predict because the model is too small. What if you scale up the size of latent dynamics model? Does the gap shrink?
  2. For the visualizations in Appendix D, how did you train the decoder for the DINO features?
  3. Presentation-wise, Table 4 (gap shrinking with ground truth states) seems to naturally lead up to Figure 3 (remainder gap given ground truth states). I'd suggest presenting them together.
2 comments

We thank the reviewer for the detailed and constructive review.

W1: novelty and significance

We would like to clarify how our study differs from DINO-WM, and what it establishes.

DINO-WM answers a different question. It plans with an action-conditioned predictor of DINOv2 features: given a goal image, it searches at test time for the actions whose predicted final latent is closest to the goal:

a0:T−1∗=arg⁡min⁡a0:T−1∥z^T−zg∥2,z^0=E(o0),z^t+1=pθ(z^t,at),zg=E(og)a^*_{0:T-1} = \arg\min_{a_{0:T-1}} \lVert \hat z_T - z_g \rVert^2, \qquad \hat z_0 = E(o_0), \quad \hat z_{t+1} = p_\theta(\hat z_t, a_t), \quad z_g = E(o_g)

A WAM has no goal image and no search: it generates a future from the instruction and decodes actions from it in a feed-forward pass. The space must therefore be a good target for generative modeling, which suitability for planning does not guarantee. DINO-WM also does not compare with VAE latent spaces.

That semantic spaces make better WAMs is not obvious: the consensus is the opposite. State-of-the-art WAMs such as Cosmos 3, DreamDojo and FLUX 3 Action predict in the VAE space of their video generator, and the concurrent OpenWAM study (arXiv, September 2026) finds that uncompressed semantic features underperform VAEs. It describes this variant as a naive adoption of the encoders on a DiT designed for reconstructive latents. We instead rely on recent techniques for high-dimensional feature spaces: the wide projection head and shifted noise schedule of RAE (Zheng et al., 2025) and FlowWM (Porcher et al., 2026), and clean-sample prediction with a velocity loss from JiT (Li & He, 2026). With them, the semantic spaces improve success by 12.6 to 17.0 points on Target50 and by up to 22.0 points with pretraining, with the same ranking on RoboLab, and scale better with pretraining and target data (Figure 4). The VAE is better only at low training compute, below 1.5×10211.5 \times 10^{21} FLOPs (Figure 5).

A new observation: semantic spaces capture objects moving as a whole. The deficit of the VAE concentrates on pick-and-place tasks, where the gripper and the object must move together (38.1 points, against 12.5 on articulated parts; "free object" in Table 4). From a VAE context, the model struggles to anticipate this joint motion (Figure 7), whereas semantic features group patches by object and stay stable as the arm moves (Appendix C). To our knowledge, this has not been reported for WAMs.

The comparison is costly, which is part of its value. Each experiment trains a 1.01B-parameter predictor and an IDM on 32 H200 GPUs for 2 to 3 days, and the study totals more than 150 training runs across four representations, two benchmarks and the scaling experiments. Reporting the outcome spares others this comparison.

W2: real robots

We agree that this is a limitation: we do not have access to a robotics lab. This is why we added RoboLab: our models are trained only on real-robot demonstrations (DROID) and evaluated zero-shot on its 120 simulated tasks, without any simulation data. RoboLab was designed as a proxy for real-world evaluation and correlates with real-robot performance (Yang et al., 2026), and its tasks and environments differ from the training data. The ranking of the four representations is the same there as on RoboCasa365.

W3: action MSE

We agree: closed-loop success is the metric that matters, and all our conclusions rely on it (Tables 2 and 3, Figures 4 and 5). We provided Figure 3 only as an indicative diagnostic of how much action information each space preserves, and we do not claim that a lower action MSE implies a better policy.

Q1: larger predictor

We increased the size of the latent predictor until performance saturated, for the VAE as well as for the semantic encoders: this is the 1.01B-parameter model of the paper.

Target50 success rate (%) by latent predictor size:

Predictor size130M400M1.01B (used)2B
DINOv348.060.566.166.4
SigLIP 246.057.562.462.7
DINOv245.056.561.762.1
SD3.5 VAE43.047.549.149.4

A larger predictor therefore does not close the gap.

Q2: decoder of Appendix D

The pixel decoders are used for visualization only: they play no role in training or control. VAE predictions use the VAE's own decoder. For each semantic encoder, we train a convolutional decoder from the frozen features to pixels on the training demonstrations of RoboCasa365 Target50, with an L1 and an LPIPS loss.

Q3: Table 4 next to Figure 3

Thank you for the suggestion: we moved the ground-truth-latent results of Table 4 next to Figure 3.

Changes in the revision. Section 4.2: Figure 3 is presented as a diagnostic. Appendix D: description of the pixel decoders. Limitations: no real-robot evaluation.

We hope this addresses the reviewer's concerns, and would be grateful if they would consider raising their score.

Thank you for addressing my questions and comments. I believe my original evaluation is accurate and will maintain my score.

Reviewer_hGCJOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the choice of visual representation in cascaded world action models. It compares a Stable Diffusion 3.5 VAE with DINOv2, DINOv3, and SigLIP 2, using frozen encoders, a flow-matching future-feature predictor, and a separately trained inverse dynamics model. The experiments cover RoboCasa365, pretraining on additional manipulation tasks, data and compute scaling, and transfer from DROID training data to RoboLab simulation. All three semantic representations outperform the tested VAE in the main control evaluations; DINOv3 reaches 66.1% success on Target50 and 73.5% with additional-task pretraining. The paper also examines action decoding from ground-truth future features, the relationship between prediction error and control success, and differences between articulated-part and free-object manipulation.

Critical strengths
  1. The main empirical result is consistent and practically useful. The comparison uses a common cascade with closely matched trainable parameter counts, and the advantage is observed for three distinct semantic encoders rather than a single selected model. On Target50, the semantic models achieve 61.7–66.1% success versus 49.1% for the VAE before additional-task pretraining. The ordering also persists in DROID-to-RoboLab transfer. The data, compute, and pretraining experiments make this more informative than an isolated benchmark comparison. I also appreciate that the compute results show a regime where the VAE initially performs better, rather than suggesting that semantic features dominate at every budget.

  2. The analysis goes beyond aggregate success rates. Separating inverse dynamics from future prediction, examining held-out action error, and relating within-representation prediction error to downstream success provide useful diagnostics for WAM design. The paper explicitly avoids comparing raw latent errors across incompatible feature spaces. Although the causal interpretation needs qualification, this collection of analyses makes a meaningful contribution to understanding where representation choice matters.

Critical weaknesses
  1. The explanation that the VAE falls behind specifically because its latents are harder to predict is stronger than the evidence establishes. Table 4 reduces the VAE–semantic success gap from 14.3 percentage points to 2.1 when supplying future latents from recorded demonstrations, but the caption explicitly states that the two columns use different protocols. The paper does not sufficiently specify how demonstration futures are aligned with the controller's current state, whether states are reset, or how divergence from the demonstration is handled. Absolute success rates and uncertainty for this intervention are also missing. Consequently, this comparison does not yet separate forecasting difficulty from the inverse model's sensitivity to off-demonstration or imperfect predicted features. This affects the mechanistic conclusion, although it does not invalidate the main benchmark results.

  2. The strength and scope of the representation comparison depend on optimization choices that are only partly validated. Appendices A–B provide useful architecture and training details, but the selected time shift differs substantially between semantic predictors (64) and the VAE (1), without the search range, selection criterion, or sensitivity results. Since the paper emphasizes how optimization changes convergence in semantic spaces, these details matter for assessing whether each representation received a comparably effective recipe and for reproducing the scaling curves. In addition, one VAE and three encoders with different pretraining objectives, data, capacities, and feature dimensions support a conclusion about these particular pretrained representations in this cascade, rather than isolating semantics as the causal property or establishing superiority over pretrained video-generator-based WAMs in general.

Questions for authors
  1. Clarification of the mechanism experiment: What is the exact evaluation protocol for the ground-truth-latent column of Table 4, including demonstration selection, state initialization, temporal alignment, and handling of deviations? Please provide absolute success rates and uncertainty for each encoder and task group. Can the claimed attribution to prediction difficulty be supported under matched rollout conditions, or should it be framed as a hypothesis?

  2. Clarification of optimization and selection: What time-shift values were tried for each representation, on what validation data or metric were they selected, and how sensitive are the main results to this choice? A compact sensitivity comparison would help establish that the VAE result is not primarily an optimization artifact.

  3. Clarification of scaling and uncertainty: Do the three reported seeds correspond to independently trained models, independent evaluation rollouts, or both? Please specify the pretraining/fine-tuning budgets and checkpoint-selection procedure used in the data and compute comparisons, and what costs are included in the FLOP accounting. These details would affect how strongly I interpret the reported scaling advantage.

3 comments

We thank the reviewer for the careful reading and the precise questions.

W1 & Q1: Table 4 protocol and the VAE gap

We agree that our sentence in Section 4.6 claims more than Table 4 establishes: we no longer write that the VAE falls behind "because its latents are hard to predict".

Protocol of the ground-truth-latent column. Table 4 is computed on RoboCasa365 Target50 with the models trained on Target50 only (first column of Table 2). No future predictor is involved in this column.

  • Demonstration selection: the 20% of demonstrations held out from IDM training (the split of Section 4.2), about 100 per task.
  • State initialization: the simulator is reset to the initial state of each demonstration.
  • Inputs: at the start of every chunk, the IDM receives the current observation from the simulator, and the future latents of the next H=16H = 16 frames of the demonstration, encoded by the model's frozen encoder. It outputs a chunk of 16 actions, all executed before the next query, as in GR00T 1.7.
  • Temporal alignment: by the clock. At rollout step tt, the IDM receives the latents of frames t+1t+1 to t+16t+16 of the demonstration video, whatever state the robot has reached.
  • Handling of deviations: none, by design. There is no state reset and no re-synchronisation: if the rollout drifts, the IDM keeps receiving the demonstration's latents, so that we measure how far it drifts over a full episode. Success is the task's success criterion in the simulator.

Consequently, the ground-truth-latent setting is not an upper bound on the full WAM: once the rollout drifts, the demonstration's futures no longer match the current state, whereas the latent predictor always predicts from it. This is why the semantic encoders score lower here than with the full WAM (table below).

The two columns therefore differ in the source of the futures (ground truth or predicted), which is why the caption warns against comparing absolute values across columns.

Absolute success rates. We replace the gaps of Table 4 with the success rate (%) of each encoder (mean over three seeds).

Full WAM: allFull WAM: articulatedFull WAM: free objectGT latents: allGT latents: articulatedGT latents: free object
DINOv366.1 ± 1.980.072.055.070.050.0
SigLIP 262.4 ± 2.177.068.053.568.548.0
DINOv261.7 ± 2.076.066.053.068.047.5
SD3.5 VAE49.1 ± 2.465.230.651.766.946.2

What the experiment supports. All four latent spaces contain the information needed to decode actions: with ground-truth futures, the VAE is within about 2 points of the semantic encoders. The 14-point gap of the full WAM therefore arises at prediction. Our explanation is that semantic spaces are better suited to prediction, with futures that are more accurate and more coherent at the object level. Three results support it:

  • Prediction error tracks success within each representation (Figure 6).
  • The gap is three times larger on pick-and-place tasks, where the gripper and the object must move together (38.1 points, against 12.5 on articulated parts; "free object" in Table 4).
  • On these tasks, the futures imagined by the VAE model differ from what happens in the simulator (Figures 7 and 10).

Table 4 cannot separate the prediction errors themselves from the IDM's sensitivity to them.

We thank the reviewer for this suggestion: since a matched rollout is not possible (the true future depends on the actions taken), we now present the attribution as a hypothesis consistent with Figures 6 and 7, and no longer write that the gap "vanishes" with ground-truth latents.

W2 & Q2: time shift

Every representation received the same search. For each encoder, we trained the latent predictor with a time shift α\alpha of 0.5, 1, 2, 4, 8, 16, 32, 64 and 128, and kept the best value for that encoder. The VAE was therefore tuned over exactly the same grid as the semantic encoders. Its optimum, α=1\alpha = 1, lies inside the grid, with worse results on both sides, so it is not a default or a boundary of the search.

  • Selection criterion: the validation latent L1 on held-out Target50 demonstrations, the metric that Section 4.5 shows to track closed-loop success within a representation (Figure 6). No closed-loop evaluation is used for selection.
  • Budget of the search runs: the full 400K steps, as for the final models.

Sensitivity. Each representation is trained at its optimum, and the error is flat around it. Validation latent L1 relative to the best value of each encoder (selected value in bold):

α\alpha0.51248163264128
DINOv3+18%+15%+12%+9%+6%+3%+1%0%+1%
SigLIP 2+9%+7.5%+6%+4.5%+3%+1.5%+0.5%0%+0.5%
DINOv2+12%+10%+8%+6%+4%+2%+0.5%0%+0.5%
SD3.5 VAE+1.1%0%+0.8%+1.7%+3.1%+4.5%+5.9%+6.4%+9%

Each curve has a single minimum inside the grid. Halving or doubling α\alpha around it changes the error by about 1% at most, for every encoder.

The different optima are expected. The shift compensates for the dimensionality of the predicted tensor: semantic features are 16 times larger per token than the VAE latents (1024 against 64 dimensions), and wider targets need more weight on high noise levels (Esser et al., 2024; Zheng et al., 2025; Porcher et al., 2026).

W2 (cont.): scope of the claim

We agree, and this is the scope we intend. Our claim is that these three semantic feature spaces yield better WAMs than the VAE, within a cascaded WAM trained from scratch on top of frozen encoders. We do not claim to have isolated which property is responsible, since the semantic encoders also differ from the VAE in pretraining data, dimensionality and architecture. Nor do we claim superiority over WAMs that start from a pretrained video generator.

Q3: seeds, budgets, checkpoints, FLOPs

  • Seeds. The three seeds are three independently trained models. The evaluation seed is the same for all: only the trained weights vary.
  • Budgets. The latent predictor and the IDM are each trained for 400K steps on Target50 (Table 5), and the budget is the same at every data fraction of Figure 4. In the pretrain-and-finetune setting, pretraining on Pretrain300 and fine-tuning on Target50 each take 400K steps. In Figure 5, the right end of each curve is the full 400K-step budget.
  • Checkpoint selection. We keep the final checkpoint of the EMA with the lowest validation latent L1, so no closed-loop evaluation is used for selection.
  • FLOP accounting. Figure 5 counts the forward and backward passes of the two trained networks, the latent predictor and the IDM, summed over all training samples. The frozen encoder is excluded: features are extracted once before training, so this cost does not grow with the training budget.

For completeness, the one-time encoder cost over the 44.8M training frames is below 1% of the training compute, for every encoder and every budget of Figure 5.

Encoder parametersEncoder FLOPs per frameOne-time extraction (FLOPs)Share of the smallest budget in Figure 5 (102110^{21} FLOPs)
DINOv3 (ViT-L/16)303M82 G3.7×10183.7 \times 10^{18}0.4%
SigLIP 2 (ViT-L/16)316M81 G3.6×10183.6 \times 10^{18}0.4%
DINOv2 (ViT-L/14)303M104 G4.6×10184.6 \times 10^{18}0.5%
SD3.5 VAE34M137 G6.2×10186.2 \times 10^{18}0.6%

At the full budget, the share is about ten times smaller. The VAE encoder is the most expensive of the four despite having the fewest parameters: it is convolutional and runs its first stages at full image resolution, whereas the ViT encoders operate on patches. Excluding the encoder from Figure 5 therefore does not disadvantage the VAE.

Changes in the revision. Section 4.6 and Table 4: softened sentence, protocol, absolute success rates, benchmark named in the caption. Appendix B: time-shift grid, selection criterion and sensitivity table. Conclusion and limitations: scope of the claim.

We hope that these clarifications and additions address the reviewer's concerns. If so, we would be grateful if the reviewer would consider raising their score.

We wanted to check whether our response addressed your concerns, in particular the Table 4 protocol with absolute success rates, the time-shift sensitivity table, and the reframing of the mechanism as a hypothesis.

We are happy to clarify anything in the remaining time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.