acceptodds
Under review as a conference paper at ICLR 2027

On the Representation of World Action Models

Abstract

World action models (WAMs) use visual prediction to learn intermediate representations (e.g., global or dense semantics, geometry, and spatiotemporal dynamics) for robot action prediction. Learning these representations depends on which visual features are used as inputs/outputs and how these features are predicted. In this paper, we conduct comprehensive analyses on how these two choices shape intermediate representations and thus help action prediction. Our study yields two key findings:1) V-JEPA 2.1 features (which are good at spatiotemporal dynamics) outperform Wan VAE (pixel reconstruction), DINOv3 (dense semantics), and SigLIP-2 (global semantics) features for robot action prediction. 2) using frame-wise autoregression for visual feature prediction performs better than diffusion, i.e., shaping the intermediate representations better for action prediction. On RoboTwin 2.0, our representation recipe (V-JEPA 2.1 + frame-wise autoregression) achieves an average success rate of 85.9%, compared with 38.0% for the baseline recipe (Wan VAE + diffusion), without additional robot pretraining. This advantage persists after additional pre-training on 3,300 hours robot data, with success rates of 91.5% (our recipe) and 75.1% (baseline recipe). We hope these findings provide helpful guidances for developing more effective representation recipes for WAMs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Representation Learning, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kqAwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies two design choices in world-action models: the pretrained features used as visual inputs and prediction targets, and the objective used to predict those features. A common mixture-of-transformers policy is evaluated with several visual encoders and two different modeling objectives: frame-wise autoregressive regression or diffusion. V-JEPA 2.1 features perform best among the tested diffusion configurations, while autoregression improves action success with both Wan VAE and V-JEPA features. The resulting recipe reaches 85.9% average RoboTwin 2.0 success without additional robot pretraining and 91.5% after 3,300 hours of robot pretraining. Offline probes and gradient-alignment measurements provide possible explanations for these differences.

Critical strengths
  1. A useful empirical question with substantial observed effects. Table 2 provides a two-by-two comparison: autoregression raises Wan VAE success from 38.0% to 73.9% and V-JEPA success from 74.2% to 85.9%. The improvement is therefore not confined to changing the encoder alone.

  2. Broad evaluation within the selected benchmark. The study covers all 50 RoboTwin tasks and both clean and randomized conditions, with 100 episodes per task and condition. The additional robot-pretraining experiment shows that the combined recipe's advantage persists beyond task-only training.

  3. The representation diagnostics go beyond a leaderboard comparison. The paper connects control performance to semantic, geometric, and action probes, and measures both intermediate-representation quality and visual/action gradient alignment. Appendix E documents episode-level splits and bootstrap intervals for the linear-probe analysis.

  4. Detailed implementation is described. The appendices specify model dimensions, observation windows, feature normalization, attention masks, optimizer settings, and action execution. The text appropriately describes gradient alignment as a possible explanation and acknowledges that the noise presets also change loss normalization.

Critical weaknesses
  1. The contribution of future-feature supervision is not isolated. Using the same encoder for observations and prediction targets is a reasonable design, and the comparison supports the proposed representation recipe. However, without an action-only control for each encoder, it remains unclear how much of the advantage comes from stronger input features versus the additional predictive supervision. Keeping the architecture and encoder fixed while removing the visual prediction loss would clarify whether learning future dynamics provides a consistent control benefit.

  2. The autoregression-versus-diffusion comparison changes several factors together. Clean teacher-forced L1 regression and noise-weighted velocity MSE differ in loss, conditioning, masking, and effective supervision strength. Fixed weights of one do not equalize their gradient contributions. The final-checkpoint cosine measurements are informative but do not establish causality. Matched direct-regression losses, visual-loss-weight sweeps, and measurements during training would strengthen the explanation.

  3. Full-data RoboTwin evaluation does not establish robustness to domain shift. Appendix C.1 uses the full 27,500-episode pool, with 50 Clean and 500 Randomized demonstrations per task, before reserving 1% for validation. Thus, evaluation on Randomized conditions after training on randomized data is not a clean-to-randomized (C2R) generalization test. The results do not rule out reliance on benchmark-specific visual or task regularities, although they do not by themselves demonstrate overfitting. A C2R setting, trained only on Clean demonstrations and evaluated on Randomized conditions without adaptation, is needed to test whether the encoder and objective rankings survive a domain shift. Low-data comparisons and multiple training seeds would further strengthen this conclusion.

  4. The scaling comparison supports the tested recipe, not a general efficiency or scaling law. Only two robot-data budgets are tested, and both controlled vision experts start from random initialization. This differs from WAMs that inherit pretrained video-transformer weights. Table 4 also includes heterogeneous external systems and explicitly guessed pretraining durations. Those rows should remain context rather than evidence of superior data efficiency; compute, convergence curves, and a pretrained-video-backbone control would be informative.

Questions for authors
  1. How much does future-feature prediction improve over action-only training with the same current encoder and model capacity?
  2. Do the V-JEPA and autoregression advantages persist in a clean-to-randomized (C2R) setting, with training restricted to Clean demonstrations and no adaptation to Randomized conditions?
  3. Does autoregression remain better after tuning visual-loss weights and matching the direct-prediction loss? How does gradient alignment evolve during training?
  4. Is the ordering of the main encoder/objective configurations stable across reduced demonstration budgets and multiple training seeds?
3 comments

We sincerely thank Reviewer kqAw for appreciating our two-by-two recipe comparison, 50-task closed-loop evaluation, and detailed diagnostics. We address each of your points below.

1. Future-feature prediction vs. action-only training (λv=0\lambda_v = 0) with the same encoder. (W1, Q1) Due to limited rebuttal time, we conducted a controlled three-way comparison trained from scratch on a matched 2,500-episode Clean training split (roughly 1/11th of the main paper's 27,500-episode training data) with the same frozen V-JEPA 2.1-G encoder. Because these models are trained from scratch on 1/11th of the data, their absolute success rates are lower than the full-data models in Tables 1–2, while the comparison within this group is strictly controlled:

Visual-Training Objective (Fixed V-JEPA 2.1-G, 1/11th Data)Clean Success (%)vs. Diffusionvs. Action-Only
Diffusion visual prediction28.4%—−5.0 pp
Action-only baseline (λv=0\lambda_v = 0)33.4%+5.0 pp—
Frame-wise autoregression (Ours)49.3%+20.9 pp+15.9 pp

Frame-wise autoregression outperforms action-only training by +15.9 percentage points, demonstrating that predicting clean future visual features provides a substantial inductive bias beyond simply passing strong frozen features to the action expert. Conversely, noisy diffusion prediction underperforms action-only training by 5.0 pp, suggesting that denoising noisy future latents competes for capacity within the shared vision expert without aiding action generation. (These matched supplementary controls will be reported in their own subsection, clearly separated from the full-data Tables 1–2.)

2. Matching loss weights and gradient alignment dynamics. (W2, Q3)

  • Role of λv=λa=1\lambda_v = \lambda_a = 1: Setting λv=λa=1\lambda_v = \lambda_a = 1 defines the standard, un-tuned recipe for both objectives rather than assuming identical gradient norms between L1 and velocity MSE. At the evaluated checkpoint, the directional alignment metric in Equation 4 (cosine similarity) is invariant to positive scalar rescaling of per-example losses, meaning the directional gap (0.154450.15445 for AR vs. 0.000230.00023 for diffusion) is not simply a static units-of-loss artifact; nonetheless, we recognize that loss weighting still influences relative gradient magnitudes and optimization trajectories over training.
  • Scope of Checkpoint Diagnostics: We evaluate both models at the exact same training step (19,570 updates) to characterize the learned representations and local loss geometry of the converged policies. While tracking gradient alignment trajectories across training is an exciting direction for future work, our causal intervention (λv=0\lambda_v = 0 vs. AR vs. diffusion above) directly establishes that the AR objective drives the performance gain. We will make this distinction explicit in Section 4.2.

3. Clean-to-randomized generalization and full-data protocol. (W3, Q2)

  • Full-Data Training Protocol: In our main experiments (Tables 1–4), WAM training uses the full RoboTwin 2.0 training corpus containing both Clean (50 episodes/task) and Randomized (500 episodes/task) demonstrations across all 50 tasks (Appendix C.1). We will state this explicitly in Section 2.2 so readers do not need to look up the appendix.
  • Dedicated Clean-to-Randomized (C2R) Evaluation: To directly evaluate zero-shot transfer from Clean training to Randomized evaluation within the rebuttal period, we trained both the diffusion baseline and our frame-wise autoregression model from scratch (with the same frozen V-JEPA 2.1-G encoder) exclusively on 2,500 Clean episodes (~1/11th of the full training data) for 1,400 optimizer updates and evaluated them across all 50 Randomized tasks (20 episodes per task, 1,000 total episodes):
Clean-to-Randomized (C2R) MethodRandomized Success (%)Total Episodes Succeeded
Fast-WAM (External Reproduction on RoboTwin Leaderboard) [1, 2]1.9%—
Diffusion visual prediction (Our Baseline, 1/11th Data, 1,400 updates)2.5%25 / 1,000
Frame-wise autoregression (Ours, 1/11th Data, 1,400 updates)2.7%27 / 1,000
  • Interpretation & Limitations under Severe Domain Shift: In our matched 1,400-update control, frame-wise autoregression succeeds on 27/1,000 episodes (2.7%) versus 25/1,000 episodes (2.5%) for our diffusion baseline (for context, an external reproduction of Fast-WAM on the official RoboTwin leaderboard reports 1.9% under a separate protocol [1, 2]). Because this difference is only 2 episodes out of 1,000 and absolute C2R performance remains very low across all models, we do not claim a reliable zero-shot C2R generalization advantage from this experiment. Instead, these results indicate that training solely on Clean demonstrations under a current-only inference interface (z0z_0) is insufficient to bridge strong visual domain randomization. Relatedly, Zhao et al. [3] show that preserving inference-time future conditioning can substantially improve out-of-distribution robustness under domain shift; exploring how our frame-wise AR representation objective interacts with inference-time future conditioning is a promising direction for future work. We will report these C2R numbers and this candid limitation in the revised manuscript.

4. Reduced demonstration budgets and multiple training seeds. (W3, Q4)

  • Data-Efficiency Evidence Across Budgets: Our experiments demonstrate consistent gains across three distinct data regimes:
    1. Low-data Clean-only regime (2,500 episodes, ~1/11th data, trained from scratch): AR outperforms diffusion by +20.9 pp (49.3% vs. 28.4%) and action-only by +15.9 pp (49.3% vs. 33.4%).
    2. Full RoboTwin task regime (27,500 episodes): AR outperforms diffusion by +11.7 pp on V-JEPA 2.1-G (85.9% vs. 74.2%) and +35.9 pp on Wan VAE (73.9% vs. 38.0%).
    3. Large-scale robot pretraining regime (3,300 hours InternData + RoboTwin): Our combined recipe (V-JEPA 2.1 + frame-wise AR) maintains a +16.4 pp lead over the Wan VAE + diffusion baseline (91.5% vs. 75.1%).
  • Multiple Training Seeds: Due to the high computational cost of training 28-layer MoT models and evaluating 10,000 closed-loop rollouts per checkpoint (50 tasks ×\times 200 episodes), we did not have enough time during the rebuttal period to run additional random seeds. Currently, all main runs use reproducible seed 42 (Table 5), while Figure 2(a) reports 95% bootstrap confidence intervals across held-out episodes—which reflect episode-level variation rather than training-seed variance. We will add multi-seed training runs in the revised manuscript to quantify seed-to-seed variance directly.

5. Initialization and robot-data pretraining. (W4) Training task-specific latent dynamics and control modules from random initialization on top of frozen perceptual encoders is an established practice in world-model and policy learning—for example, UWM [4] trains its video-action diffusion transformer from scratch over a frozen pretrained visual VAE, and DINO-WM [5] learns latent transition dynamics from scratch on frozen pretrained patch representations. Initializing the vision expert from scratch in our study ensures a strictly fair comparison between AR and diffusion: a pretrained video-diffusion checkpoint (e.g., Wan2.2-5B) cannot be dropped into a frame-wise AR V-JEPA feature predictor without mismatching both the latent space and the causal attention structure. Crucially, when we provide both recipes with the exact same 3,300-hour InternData robot pretraining prior to RoboTwin fine-tuning (Table 4), our combined recipe (V-JEPA 2.1 + frame-wise AR) still achieves 91.5% vs. 75.1% for the Wan VAE + diffusion baseline (+16.4 pp), confirming that its advantage persists when scaling up with pretrained dynamics weights. We will clarify this initialization rationale and cite these precedents in Section 2.1 and Section 5.

References:

  • [1] Yuan et al., Fast-WAM: Do World Action Models Need Test-time Future Imagination?, arXiv:2603.16666, 2026.
  • [2] RoboTwin Official Leaderboard (Clean-to-Randomized Evaluation), https://robotwin-platform.github.io/leaderboard.
  • [3] Zhao et al., Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Models, arXiv:2608.04404, 2026.
  • [4] Zhu et al., Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets, arXiv:2504.02792, 2025.
  • [5] Zhou et al., DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning, Proceedings of ICML, 2025.
Reviewer_KuGtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates visual representations and prediction objectives for world action models through eight encoder configurations, frozen-feature probes, closed-loop control, and intermediate-representation analysis. It finds strong performance from V-JEPA features and frame-wise autoregression, with additional experiments on robot-data pretraining and diffusion noise presets. The contribution is an empirical design study rather than a new policy architecture.

Critical strengths
  1. Evaluating representations through executed control, rather than reconstruction quality alone, is valuable. The autoregressive recipe improves success from 74.2% to 85.9% with V-JEPA and from 38.0% to 73.9% with Wan VAE (Table 2). The advantage of the combined recipe also persists after additional robot-data pretraining. These are substantial empirical observations.
  2. The paper goes beyond a single ranking by examining action probes, geometry probes, gradient alignment, and several diffusion noise presets. Section 4.3 is a genuine attempt to investigate the objective, and the authors explicitly identify the gradient-based explanation as a possible explanation. The paper should receive credit for both the additional analysis and its qualifications.
Critical weaknesses
  1. The study does not yet turn its encoder ranking into a sufficiently isolated representation-level conclusion. Appendix B.2 varies temporal inputs, target counts, model sizes, and pretraining across encoders. The observed ranking therefore identifies successful complete configurations, but cannot distinguish geometry, temporal information, and pretraining as explanations. The already-cited study [1] address closely related representation questions, although with different evaluation endpoints. I do not require comparisons to these preprints; rather, their overlap clarifies that this paper's added value must come from its closed-loop analysis and the reliability of the design insight it establishes.
  2. The autoregression comparison does not sufficiently separate prediction paradigm from supervision design. Clean teacher forcing with L1 prediction is compared against noisy velocity prediction with weighted MSE and different within-vision masking. Table 3 usefully changes diffusion presets, but each preset changes both sampling and loss normalization; it does not isolate the component responsible for the large gap. Gradient cosines of 0.15445 versus 0.00023 are descriptive, not an intervention on the proposed explanation. With one reported closed-loop evidence confined to RoboTwin, the stability of the proposed design rule remains insufficiently established.

[1] Jha, Saurav, Artem Zholus, and Sarath Chandar. "Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models." arXiv preprint arXiv:2605.06388 (2026).

Questions for authors
  1. Can the authors provide a focused encoder control matching temporal coverage and target budget, or explicitly restrict the representation conclusions to the tested configurations? Any available leave-one-encoder-out sensitivity analysis would also clarify how much the probe correlations depend on individual encoders.
  2. What evidence separates the benefit of autoregression from that of clean-feature supervision and loss scaling? A focused matched-objective comparison, together with any existing independent-training results, would be more decision-relevant than additional unmatched encoder rankings.
2 comments

We sincerely thank Reviewer KuGt for your thoughtful assessment and for highlighting the value of closed-loop evaluation and careful diagnostic qualification.

1. Representation conclusions, temporal coverage, and relation to prior work. (W1, Q1)

  • Encoder Selection & Controls: As detailed in our Global Response (Point 1), our goal is to guide practical encoder selection for WAMs under a unified downstream training and control protocol. Capacity-matched comparisons (V-JEPA 2.1-L vs. SigLIP2-L at 300M parameters; V-JEPA 2.1-L vs. 2B V-JEPA 2.1-G) and our new single-frame vs. two-frame V-JEPA 2.1-L control (37.8% vs. 30.9%, trained from scratch on a 2,500-episode clean split—1/11th of the main training data—due to limited rebuttal time) confirm that neither parameter count nor extra historical frames account for V-JEPA's advantage.
  • Relation to Prior World-Model Representation Studies: Thank you for highlighting Nilaksh, Jha et al. [1] (cited in our manuscript). Whereas Nilaksh, Jha et al. [1] study action recoverability and policy evaluation inside action-conditioned generative world models, we investigate jointly trained WAM policies where the action expert directly reads the vision expert's intermediate representations for closed-loop control, and we further show how changing the visual prediction objective (AR vs. diffusion) reshapes those internal representations. We will sharpen this distinction in Section 6.

2. Leave-one-out sensitivity of probe correlations. (Q1) Following your insightful suggestion, we performed a leave-one-configuration-out sensitivity analysis across all eight representations in Table 1 (with Depth and Normal errors sign-reversed so higher values indicate better performance):

Offline ProbeAll 8 Encoders (ρ\rho)Leave-One-Out Rangeρ\rho Without Wan VAE
ImageNet classification0.262−0.107 to 0.429−0.107
Scene classification (SUN397)0.286−0.071 to 0.464−0.071
Dense semantics (ADE20K)0.3570.036 to 0.6430.036
Cross-view correspondence (NAVI)0.3810.071 to 0.7140.071
Depth (NYUv2)0.5710.357 to 0.7140.357
Surface normals (NYUv2)0.5950.393 to 0.7500.393
Inverse dynamics (IDM 5f)0.7140.571 to 0.8570.571
Action prediction (BridgeData)0.7860.679 to 0.9290.679

Action prediction remains the strongest (or tied-strongest) correlate of closed-loop control success in every single leave-one-out subset, and retains a strong positive correlation (ρ=0.679\rho = 0.679) even when the reconstruction outlier (Wan VAE) is excluded. This confirms that our probe correlation is robust and not an artifact of a single weak baseline. At the same time, we fully agree that correlations over N=8N=8 encoders should be interpreted descriptively rather than as a universal statistical law, and we will add this sensitivity table and qualification to Section 3.2.

3. Autoregression vs. action-only baseline, loss scaling, gradient alignment, and multi-seed variance. (W2, Q2)

  • Action-Only Control (λv=0\lambda_v = 0): As shown in our Global Response (Point 2), with V-JEPA 2.1-G held fixed in our matched rebuttal control (trained from scratch on ~1/11th of the main training episodes, explaining the lower absolute scores across all three models), AR beats action-only training by +15.9 pp (49.3% vs. 33.4%), while action-only beats diffusion by +5.0 pp (33.4% vs. 28.4%). This confirms that the AR gain comes from beneficial auxiliary supervision rather than merely relying on strong frozen input features.
  • Interpreting Gradient Alignment & Noise Presets: We agree completely on careful interpretation. At a fixed checkpoint, per-example scalar loss rescaling leaves cosine similarity unchanged, so Figure 2(b) captures directional compatibility between native visual and action gradients rather than a static units-of-loss effect, while a near-zero mean cosine (0.000230.00023 for diffusion) indicates unaligned updates rather than direct gradient opposition. However, we recognize that loss weighting still influences gradient magnitudes and optimization trajectories during training. Similarly, Table 3 evaluates complete noise-sampling presets together with their theoretically derived loss normalizers (Appendix B.4), rather than isolating individual hyperparameters. We will refine our wording in Sections 4.2–4.3 to frame these diagnostics strictly as supporting explanatory evidence.

References:

  • [1] Nilaksh, Saurav Jha, Artem Zholus, and Sarath Chandar, Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models, arXiv:2605.06388, 2026.

Thank you for the additional experiments and analysis. The leave-one-out results reduce my concern that the probe correlations are driven by a single encoder, and the action-only control shows that the AR recipe provides benefits beyond using strong frozen features. However, my main attribution concern remains: the comparison still changes clean-feature supervision, loss formulation, noise conditioning, and masking together with the prediction paradigm. I will maintain my rating of 2.

Reviewer_6MNFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how visual representations and visual prediction objectives influence robot action prediction in World Action Models (WAMs). The authors compare pretrained visual encoders, including Wan VAE, DINOv3, SigLIP2, and V-JEPA 2.1, using representation probes and closed-loop manipulation on RoboTwin 2.0. They further compare frame-wise autoregression against diffusion-based visual prediction and analyze intermediate action-prediction performance and gradient alignment. Combining V-JEPA 2.1 with autoregression achieves 85.9% average success without additional robot pretraining, compared with 38.0% for the Wan VAE diffusion baseline. With 3,300 hours of additional robot pretraining, the respective success rates reach 91.5% and 75.1%.

Critical strengths

1.Systematically compare a number of pretrained visual representations and prediction objectives within a unified WAM framework. The closed-loop evaluation over 50 RoboTwin tasks gives practically relevant evidence and large improvements achieved by autoregressive prediction are especially noteworthy. 2. More than just final success rates analysis. The authors explore intermediate representations with action probes and gradient alignment between visual and action objectives . These analyzes yield useful hypotheses as to how auxiliary visual supervision affects robot control.

Critical weaknesses
  1. Insufficiently distinct representation properties. The visual encoders considered vary in model capacity, temporal inputs, feature dimensions, and prediction horizons. For example, V-JEPA relies on temporal observations while other encoders are working on individual frames. Therefore, the experiments do not provide conclusive evidence that V-JEPA has an advantage due to its representation properties or other architectural and input differences.
  2. As an article proposing a new model, it lacks sufficient innovation. However, as a summary article, it is more complete in comparison and involves more organization and analysis.
Questions for authors
  1. Can the authors provide more controlled ablations separating the effects of autoregressive prediction, clean-feature regression, loss functions, and noise conditioning?
  2. How does performance change when the auxiliary visual prediction loss is removed (λ_v = 0)?
  3. How sensitive are the representation comparisons to temporal input differences and prediction horizons?
  4. Can the authors report uncertainty across independently trained models and clarify why the V-JEPA diffusion baseline differs between Tables 2 and 3?
2 comments

We sincerely thank Reviewer 6MNF for recognizing the value of our systematic closed-loop benchmarks and intermediate-representation analyses. We address your specific comments and questions below.

1. Representation properties, temporal inputs and model capacity. (W1, Q3)

  • Practical Design Framing: Rather than retraining disparate vision backbones from scratch—which would alter the very representations practitioners actually deploy—our study evaluates frozen pretrained encoders under a strictly controlled downstream WAM protocol (identical MoT architecture, action expert, optimization hyperparameters, and 50-task closed-loop evaluation).
  • Capacity & Temporal Controls: As detailed in our Global Response (Point 1), capacity differences do not explain the ranking: V-JEPA 2.1-L (300M) outperforms capacity-matched SigLIP2-L (304M) by +5.7 pp and matches the ~2B V-JEPA 2.1-G. Furthermore, in our new matched temporal control (trained from scratch on a 2,500-episode clean split—approximately 1/11th of the main paper's training data—due to limited rebuttal time, resulting in lower absolute rates across both runs), single-frame V-JEPA 2.1-L achieves 37.8% vs. 30.9% for two-frame V-JEPA 2.1-L under identical target offsets. Also note that in our original Table 1 setup, the current observation z0z_0 for V-JEPA is encoded from a repeated current frame (0, 0), meaning the action expert never receives an extra distinct historical frame at inference.

2. Novelty and distinct contribution of the study. (W2)

We respectfully emphasize that our work goes well beyond benchmarking existing encoders; it introduces and validates a novel visual-training recipe for WAMs backed by mechanistic evidence:

  • Systematic Objective-Replacement Study in WAMs: While concurrent methods such as Fast-WAM [1] retain diffusion/flow-matching visual training and simply skip future generation at test time, we demonstrate that replacing the diffusion visual objective with clean frame-wise autoregressive representation prediction dramatically improves closed-loop control (+35.9 pp on Wan VAE; +11.7 pp on V-JEPA 2.1-G).
  • Action-Centric Representation Diagnostics: We show that offline action probes predict downstream control success far better than standard semantic probes (Spearman ρ=0.79\rho = 0.79 vs. 0.26–0.360.26\text{--}0.36) and that AR yields lower future-action prediction error across all five evaluated vision-expert layers alongside substantially higher visual/action gradient alignment (0.154450.15445 vs. 0.000230.00023).
  • Validated Scalability: Unlike studies restricted to small-scale task data, we verify that our combined recipe (V-JEPA 2.1 + frame-wise AR) retains a +16.4 pp advantage (91.5% vs. 75.1% over the Wan VAE + diffusion baseline) after 3,300 hours of robot pretraining. We will sharpen this positioning and discuss concurrent autoregressive WAMs (e.g., Cai et al. [2]) in Section 1 and Section 6.

3. Separating autoregression, clean-feature regression, losses, and noise conditioning. (Q1)

Our Table 2 comparison evaluates two complete, well-defined visual-prediction paradigms within an identical WAM framework: clean teacher-forced autoregressive L1 regression vs. noise-corrupted flow-matching velocity MSE. Because the encoder, action objective, and action inference schedule remain fixed within each pair, the gains across both Wan VAE and V-JEPA confirm that the improvement stems from the visual prediction recipe rather than encoder selection.

Regarding our diagnostics in Section 4.2 and Table 3: we view the layer-wise action probes and gradient cosine alignment as explanatory mechanistic evidence rather than an isolated ablation of every mathematical sub-component. At a fixed checkpoint, cosine similarity is invariant to positive scalar rescaling of per-example losses, so the directional gap (0.154450.15445 vs. 0.000230.00023) is not merely a static units-of-loss effect; however, we recognize that loss weighting still affects gradient magnitudes and optimization trajectories during training. In the revision, we will explicitly distinguish our empirical recipe claims (Table 2) from our mechanistic hypotheses (Section 4.2–4.3).

4. Removing the auxiliary visual prediction loss (λv=0\lambda_v = 0). (Q2)

As reported in our Global Response (Point 2), we ran the exact controlled experiment you suggested with a fixed V-JEPA 2.1-G encoder, training all models from scratch on a matched 2,500-episode clean split (~1/11th of the full training budget, chosen to complete the multi-way comparison within the limited rebuttal period; hence the lower absolute success rates compared to Tables 1–2):

  • Action-only (λv=0\lambda_v = 0) achieves 33.4%, outperforming diffusion visual prediction (28.4%) by +5.0 pp.
  • Frame-wise autoregression achieves 49.3%, outperforming action-only training by +15.9 pp and diffusion by +20.9 pp.

This result directly resolves your question: adding an auxiliary visual objective is not automatically helpful—noisy velocity prediction diverts shared vision-expert capacity toward denoising tasks that slightly hinder action learning, whereas clean frame-wise prediction provides strong, action-aligned representation supervision. We will include this controlled ablation in Section 4.

5. Independent-training uncertainty and baseline normalization in Tables 2 and 3. (Q4)

  • Multi-Seed Uncertainty Reporting: Due to the high computational cost of training 28-layer MoT models and running 10,000 closed-loop rollouts per checkpoint, we were unable to complete additional independent training seeds within the limited rebuttal period. We will clarify this distinction explicitly in the text and add multi-seed training runs in the revised manuscript.
  • Cross-Table Baseline Difference (74.2% vs. 72.9%): Thank you for catching this detail. The V-JEPA 2.1-G diffusion baseline in Table 2 (74.2%) applies per-sample, per-channel, per-encoded-state spatial standardization (Equation 5, Appendix B.2), whereas the noise-preset sweep in Table 3 (Baseline: 72.9%) uses precomputed fixed global normalization statistics. Because these two tables were run as separate experimental groups with different normalization protocols rather than identical replicate runs, we do not attribute the 1.3 pp cross-table difference solely to normalization; due to time and compute constraints during the rebuttal period, we could not re-run the full-data sweep under a unified setting. Crucially, within each table, all compared methods follow the exact same protocol, ensuring internal fairness. We will explicitly document this in Section 4.3, Appendix B.2, and the table captions.

References:

  • [1] Yuan et al., Fast-WAM: Do World Action Models Need Test-time Future Imagination?, arXiv:2603.16666, 2026.
  • [2] Cai et al., Devol-ONE: One Autoregressive Mixture of Transformers to Unify Vision-Language-Action and Latent World Modeling, arXiv:2609.32193, 2026.

Comments on the paper

2 comments

We sincerely thank all reviewers for their thorough, constructive feedback and for recognizing the practical value and empirical rigor of our study. In particular, we appreciate the reviewers highlighting:

  • Systematic Closed-Loop Evaluation: Comparing visual representations and prediction objectives through executed control across all 50 RoboTwin 2.0 tasks (Reviewers 6MNF, KuGt, kqAw).
  • Substantial Gains from Frame-Wise Autoregression: Demonstrating consistent, large improvements over diffusion across both pixel-reconstruction (Wan VAE) and spatiotemporal (V-JEPA 2.1) representations (Reviewers 6MNF, KuGt, kqAw).
  • Comprehensive Representation Diagnostics: Connecting closed-loop control success to offline action/geometry probes and intermediate gradient alignment (Reviewers 6MNF, KuGt, kqAw).
  • Scalability with Robot Pretraining: Confirming that the combined recipe maintains a substantial advantage (+16.4 percentage points) after 3,300 hours of robot-data pretraining (Reviewers KuGt, kqAw).

To complete targeted ablations within the limited rebuttal period, we conducted new controlled experiments trained from scratch on a matched 2,500-episode Clean training split (approximately 1/11th of the main paper's 27,500-episode training budget), alongside statistical sensitivity analyses, to directly address the reviewers' core questions. Because these supplementary models are trained from scratch on 1/11th of the data, their absolute success rates are naturally lower than those in Tables 1–2; accordingly, all supplementary results are evaluated strictly within their own matched experimental group. Below, we summarize our primary clarifications and new findings:

1. Why Evaluate Pretrained Encoders with Their Native Configurations? (Reviewers 6MNF, KuGt)

Our study is designed to answer a central, practical question for World Action Model (WAM) developers: among existing pretrained visual encoders, which provides the most effective observation representations and prediction targets? In practice, WAMs build upon frozen, off-the-shelf visual encoders rather than retraining every candidate from scratch under an artificial, unified pretraining recipe. Consequently, an encoder's native architecture, pretraining data, and temporal interface are intrinsic properties of the candidate representation being selected. To ensure a fair downstream comparison, we freeze all encoders and connect them via input/output projections to an identical Mixture-of-Transformers (MoT) WAM architecture, shared action expert, unified optimization schedule, and identical closed-loop evaluation protocol.

Furthermore, encoder capacity and multi-frame inputs do not explain our ranking:

  • Capacity-Matched Comparison: In Table 1, V-JEPA 2.1-L (300M parameters, 1,024 channels) outperforms SigLIP2-L (304M parameters, 1,024 channels) by +5.7 percentage points (74.6% vs. 68.9%) and outperforms the ~306M Qwen3-VL and XR-1 visual encoders. Moreover, V-JEPA 2.1-L matches the ~2B-parameter V-JEPA 2.1-G (74.6% vs. 74.2%).
  • New Temporal-Input Control (Single-Frame vs. Two-Frame): To isolate whether V-JEPA's two-frame causal interface inflates its performance within the rebuttal timeframe, we trained single-frame and two-frame V-JEPA 2.1-L diffusion models from scratch in a matched controlled setting (2,500 Clean training episodes, ~1/11th of the full training data, evaluated across all 50 Clean tasks with seen instructions, 20 episodes/task):
V-JEPA 2.1-L Diffusion Configuration (Matched Control, 1/11th Data)Clean Success (%)
Two-frame causal encoding30.9%
Single-frame encoding37.8%

Adding the extra frame yields no advantage (+6.9 pp for single-frame in this setting), confirming that V-JEPA's superior control performance stems from its learned spatiotemporal features rather than extra input frames. (Note that to ensure strict comparability, supplementary controls trained from scratch on the 2,500-episode clean split will be reported in their own subsection, separate from the full-data results in the revision.)

2. Isolating the Visual-Training Recipe: AR vs. Diffusion vs. Action-Only (λv=0\lambda_v = 0) (Reviewers 6MNF, KuGt, kqAw)

A key question raised by all reviewers is whether frame-wise autoregression (AR) genuinely provides beneficial auxiliary supervision beyond simply using strong frozen features (i.e., action-only training with λv=0\lambda_v = 0). To isolate the effect of the visual prediction objective while holding the encoder fixed (V-JEPA 2.1-G) within the available rebuttal time, we trained all three variants from scratch on the matched 2,500-episode Clean training split (~1/11th of the full training budget):

Visual-Training Objective (Fixed V-JEPA 2.1-G, 1/11th Data)Clean Success (%)vs. Diffusionvs. Action-Only
Diffusion visual prediction28.4%—−5.0 pp
Action-only baseline (λv=0\lambda_v = 0)33.4%+5.0 pp—
Frame-wise autoregression (Ours)49.3%+20.9 pp+15.9 pp

This ordering provides direct causal evidence for our paper's central thesis: visual supervision is only beneficial when its prediction objective is compatible with action learning. Removing the noisy diffusion objective actually improves performance by +5.0 pp (33.4% vs. 28.4%), indicating that noise-dependent velocity prediction can interfere with action learning in the shared vision expert. Conversely, clean frame-wise autoregression outperforms action-only training by +15.9 percentage points, proving that predicting clean future features supplies a strongly constructive inductive bias for action generation.

We will incorporate all new controls, sensitivity analyses, and methodological clarifications into the revised manuscript.

References:

  • [1] Yuan et al., Fast-WAM: Do World Action Models Need Test-time Future Imagination?, arXiv:2603.16666, 2026.
  • [2] RoboTwin Official Leaderboard (Clean-to-Randomized Evaluation), https://robotwin-platform.github.io/leaderboard.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.