acceptodds
Withdrawn from ICLR 2027

What Matters for World Modeling in Vision-Language-Action Models?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Integrating world models into Vision-Language-Action models (VLAs) has emerged as a promising direction, enabling policies to anticipate future dynamics beyond reactive observation-to-action mapping. However, existing approaches vary simultaneously in their world representations, integration paradigms, and policy backbones, making it difficult to determine what truly matters for effective world model integration. To disentangle these factors, we categorize existing efforts into four representative paradigms, namely Implicit Future Alignment, Explicit Future Query, MoT Future Expert, and External World Predictor. We instantiate each within a unified framework and systematically evaluate these paradigms with diverse world representations and policy backbones across in-domain, out-of-domain, long-horizon, dynamic, few-shot, and real-world settings. Our study yields four main findings: 1) world representation effectiveness is paradigm-dependent, whereas 2D semantic representations offer robust, cross-paradigm gains; 2) no single paradigm is universally optimal, yet the decoupled MoT Future Expert offers competitive overall performance, particularly in OOD robustness; 3) world modeling consistently benefits policies built on standard VLMs, while decoupled designs exhibit greater compatibility with pretrained VLAs; and 4) world modeling improves performance over action-only baselines across multiple settings, with notable gains on long-horizon and dynamic tasks. Project page and video demonstrations are available at https://anonymous-demo123.github.io/iclr2027-project-page/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, World Models, Empirical Study

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Lgk1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how future-representation supervision should be integrated into VLA policies. It compares four representative integrations and six frozen visual encoders, primarily using a Qwen3-VL-based policy, with additional pi0.5 experiments. Evaluation spans in-domain and perturbed manipulation, task horizons, moving objects, reduced demonstration budgets, and four physical-robot tasks. The main contribution is an empirical comparison and its resulting design guidance, rather than a new world-modeling architecture.

Critical strengths
  1. The comparison addresses an important practical question that individual architecture papers leave difficult to answer. The representation-by-integration matrix reveals useful interactions, including the opposite preferences of WM-MoT and WM-Ext for V-JEPA versus DINOv3. The evaluation also reports failures, such as WM-Query under limited data and WM-Align when adapting pi0.5, rather than presenting only favorable settings. These observations could guide model design if the evaluated implementations are unambiguously specified.

  2. The appendix contains meaningful controls beyond aggregate benchmark scores. Table B.1 repeats central positive and negative findings across three training seeds. Table B.2 separates additional DINOv3 input, model capacity, current-target supervision, and future-target supervision: within WM-MoT, the latter reaches 76.2 percent OOD success versus 73.6 without the auxiliary loss and 74.0 with current-target supervision. Table B.3 further separates input and target encoders. Together with measured deployment costs and real-robot results, these strengthen the empirical contribution substantially.

Critical weaknesses
  1. The central representation comparison cannot currently be reconciled with the supplied extraction code. Section 3.1 and Appendix A.2 state that all six encoders receive the same four future frames at offsets {4, 8, 12, 16}. However, examples/Foresight/extract_vjepa21_features.py explicitly constructs the target from the two-frame pair [t, t+16], and its extraction call uses encode_pair on precisely these two images; the online extractor follows the same procedure. Thus, the supplied V-JEPA pipeline changes the temporal information and target construction as well as the encoder. This matters directly to the V-JEPA ranking reversal in Figure 3 and the factorial attribution in Table B.3. The issue is not that different encoders produce different token counts, but that the asserted common temporal input is not what the provided implementation uses. The exact evaluated pipeline needs to be established before these results support the stated controlled comparison.

  2. The supplied action-conditioning topology also differs materially from the formulation being compared. Equation 9 and Appendix A.3 describe WM-MoT actions attending to both VLM and Future Expert context at each layer. In starVLA/model/framework/VLM4A/MoTFutureExpert.py, the registered implementation instead uses AlternatingRoutingMixin, which masks future keys on one block and VLM keys on the next, with an interval of two in the supplied configuration. The external predictor likewise alternates context access in its action head, unlike the combined context in Equation 12. These are consequential routing choices for a paper whose contribution is to compare integrations under a unified policy framework. The capacity and supervision controls in B.2 are valuable, but do not resolve which routing implementation generated the cross-paradigm rankings. This discrepancy does not demonstrate that the measured gains are false; it leaves the experimental object and the scope of the resulting design guidance unclear.

Questions for authors
  1. Which exact feature-extraction pipeline generated the V-JEPA results in Figure 3 and Tables B.1/B.3/C.2: the four future frames specified in the manuscript or the supplied [t, t+16] pair? Please provide the evaluated configuration and clarify which conclusions remain supported if temporal inputs differ.

  2. Were WM-MoT and WM-Ext evaluated using the simultaneous conditioning described by Equations 9/12, or the alternating routing in the supplied code? Please identify the evaluated implementation and explain whether correcting the manuscript or replacing the supplementary implementation is sufficient, or whether any comparisons require reevaluation.

  3. Can you provide the actual run configurations linking the reported tables to backbone checkpoints, training steps, encoder profiles, and routing settings? The supplied RoboTwin defaults select Qwen3-VL-4B and 150,000 steps, whereas the main testbed is described as Qwen3-VL-2B and B.1 describes full 50,000-step replications. I recognize that A.4 permits benchmark-specific schedules and the launcher accepts overrides; the missing information is which overrides produced the reported experiments.

Reviewer_3VAoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical analysis of different ways of adding in a world model based representation learning auxiliary loss built on top of a VLA architecture. They compared four different ways of injecting this world model supervision objective on 4 simulation suites as well as real world tasks. Their results find that the representations caused by the world modeling auxiliary loss improves overall performance especially on longer horizon tasks although there was not a large gap between the different paradigms tested.

Critical strengths

I think that their evaluation covered a diverse set of simulation and real tasks, and I found that their presentation of results and their comparisons were clear to follow.

Critical weaknesses

I did not find that this paper had clear evidence of one paradigm being superior to another. Additionally, with the small gap in most of their environments between the different methods and lack of statistical significance analysis, I think it is hard to draw meaningful comparisons between these different world model approaches.

These current comparisons don't include controlled results across several world action modeling approaches such as fast wam, or mimic video, or dream zero style pretraining objectives which are different ways of doing a MoT style future expert or directly using video pretraining instead of VLA pretraining. These comparisons would be especially valuable as they are the most widely used methods of using world model style pretraining in robotics.

Questions for authors

Why are the absolute success rates very low on domino?

For the robotwin 2.0 with pi 05, how does the VLA pretraining affect the auxiliary objectives? In some cases after post training it became worse, why did this happen?

How does this work ensure fair comparisons with the action only baseline and different methods in terms of compute costs and inference costs as they each used different compute budgets?

Reviewer_HJREOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper is a study on how we should incorporate world modeling (or future visual prediction) into VLA models, and tries out various combinations of prediction methodology (feature alignment, explicit queries, MoT future expert, etc.), visual representations, and benchmarks. The outcome is rather scattered, with paradigm-dependent best choices.

Critical strengths

A robust comparison between the various design choices for incorporating world modeling is a important problem, since just attaching WM capabilities to a VLA doesn't automatically increase downstream performance. The authors' identified problem and motivation is at least a important problem in my opinion.

Critical weaknesses

However, on whether the authors actually delivered on a robust and extensive comparison that is actually helpful to the research community, I am skeptical. In the paper, the authors provide a long list of empirical rankings and few remarks on the results here and there, but there is no real strong scientific conclusion from them. The architectures and objectives are all exitsing methods, so for this paper to have merit, there should be some insight that I can actually utilize, which I don't see here.

Questions for authors

Why use only cosine alignment with shared loss weight across all the target representations? Some features maybe more suited to other loss types, especially ones where magnitude may matter.

To actually show this paper is useful, can you derive some sort of design principle from your experiments and try out its parameters on a new architecture or benchmark? If this is not shown, I don't think we can derive any scientific merit out of this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.