acceptodds
Under review as a conference paper at ICLR 2027

Devol-ONE: One Autoregressive Mixture of Transformers to Unify Vision-Language-Action and Latent World Modeling

Abstract

Vision Language Action (VLA) models condition actions directly on current visual and language context, without an explicit account of how the scene evolves under candidate actions. World Action Models (WAM) attempt to address this limitation by predicting future states, but existing designs keep prediction and policy learning architecturally separate, connecting them only through the predicted output, whether through pixel space video generation or a latent forecasting module trained independently of the policy. We present Devol-ONE, a Mixture of Transformers architecture that unifies vision language understanding, latent world dynamics prediction, and action generation within a single autoregressive framework. Instead of encoding vision language tokens once and feeding them to the action expert, Devol-ONE runs autoregressive prediction jointly across a vision language stream and a V-JEPA pretrained dynamics stream, attending to the vision language key-value cache at every layer to forecast future latent states under language guidance. The action expert is in turn shaped continuously by semantic reasoning and predicted physical dynamics rather than by a fixed representation computed in advance. Extensive experiments are conducted on LIBERO, LIBERO-PLUS, RoboTwin2.0 along with real-world evaluation on Flexiv single-arm and dual-arm setups. Ablation studies show the effectiveness of dynamic stream prediction and layer-wise unified attention to validate our model architectural coherency.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-language-Action, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q77JOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Devol-ONE, a Mixture-of-Transformers architecture integrating a vision-language stream, a V-JEPA latent dynamics stream, and a continuous action expert. The streams retain separate parameters and communicate through layerwise directed attention: the dynamics stream accesses vision-language context, while the action stream accesses both vision-language and dynamics context. The model autoregressively forecasts future latent states and conditions action-chunk generation on these predictions. The authors also introduce Autoregressive Latent Rollout Training (ALRT), which switches from a teacher-forced warm-up to self-generated latent rollouts to reduce the training–inference distribution mismatch.

Critical strengths

Using predictive latent states for action generation without synthesizing future video pixels is a reasonable direction for incorporating predictive information into deployable robotic policies.

Maintaining specialized streams for semantic processing, latent prediction

Experiments are well-performed. The paper is well-written.

Critical weaknesses

I find that in the paper ALRT is presented as a major contribution, but Tables 5, 6, and 10 test conditioning sources, fusion mechanisms, or prediction horizon—not teacher forcing versus ALRT. Consequently, the current evidence does not establish how much of the improvement comes from training on self-generated latent trajectories.

The claim that ALRT benefits not only the predictor but also the policy conditioned on its rollouts remains untested.

Page 2, lines 106–107, reports Clean and Randomized success rates of 61.1% and 63.7% on 34 evaluated tasks. In contrast, Table 8 and its accompanying discussion report results on all 50 tasks, with averages of 58.0% and 62.2%, and an overall average of 60.1%, below GigaBrain-0.7’s 67.4%. Recomputing the averages from the 50 rows of Table 8 yields approximately 58.02% and 62.16%, consistent with the appendix.

Questions for authors

What is the precise difference from VLA-JEPA? Please specify the action heads’ training and inference conditioning in both methods, explain the claimed propagation of exposure bias to the action head, and clarify the LIBERO-Plus baseline selection and averaging convention.

Reviewer_pKeJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Devol-ONE combines vision-language understanding, latent dynamics prediction, and action generation through three transformer streams connected by directed, layerwise attention. The dynamics stream predicts future V-JEPA features that condition the action expert. Autoregressive Latent Rollout Training (ALRT) exposes the model to its own predictions during training. Experiments cover LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-world manipulation, with reported success rates of 98.4% on LIBERO and 71.4% on zero-shot LIBERO-Plus.

Critical strengths
  1. Predicting future V-JEPA features provides a useful way to condition actions on anticipated scene changes without generating future images. The action expert can access both semantic and predictive features throughout decoding.
  2. The conditioning ablations support the use of layerwise interaction. In Table 5, LIBERO-Long success increases from 92.1% with final-layer fusion to 97.0% with layerwise attention. The evaluation also covers several simulation benchmarks and both single-arm and dual-arm real-world tasks.
Critical weaknesses
  1. My main concern is what the architecture adds to earlier MoT world-action models. Motus (Bi et al., 2025) already connects understanding, video generation, and action experts through joint attention, while MotuBrain (2026) uses a three-stream design with asymmetric video–action attention. Devol-ONE cites Motus but gives little discussion of how its cross-stream interaction differs. Predicting JEPA features is a clear distinction, but the contribution of the attention design itself needs a closer comparison. Table 5 shows the benefit of layerwise attention over final-layer fusion; it does not address this comparison with existing MoT designs.
  2. ALRT is presented as a main contribution, but the ablations examine other design choices: conditioning in Tables 5 and 10, and prediction horizon in Table 6. These experiments support parts of the framework, but do not show how much the rollout training contributes. A comparison with teacher-forced training under the same architecture, data, and training budget is needed to assess that contribution.
Questions for authors
  1. Which aspects of the cross-stream interaction differ from Motus and MotuBrain? Is there an ablation of the attention directions with the JEPA prediction targets held fixed?
  2. What are the task-success rates with ALRT and teacher forcing under otherwise identical training conditions? How does latent prediction error vary across rollout steps in the two settings?
  3. Table 8 uses per-task fine-tuning for Devol-ONE and joint training on all 50 tasks for the baselines. Are results under matched training protocols available? How do the 34-task results in the introduction relate to the 50-task results in Table 8?
  4. Equation (11) specifies noise prediction, whereas Appendix A.4 describes a flow-matching action head. Which training objective and sampling procedure were used in the reported experiments?
Reviewer_eWPEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Devol-ONE combines a vision-language stream, a latent dynamics predictor, and an action expert in a Mixture-of-Transformers architecture. The dynamics stream predicts future V-JEPA features under vision-language conditioning, and the action expert reads both semantic and dynamics features through directed layerwise attention. The paper also proposes Autoregressive Latent Rollout Training (ALRT), which moves from teacher-forced inputs to self-generated latent rollouts during training. Experiments on LIBERO, LIBERO-Plus, RoboTwin, and real-world manipulation examine the overall system and some of its design choices.

Critical strengths
  1. The layerwise integration is well motivated. The action expert can access task semantics and predicted dynamics throughout its computation, rather than receiving only a final representation. The fusion ablation, particularly on LIBERO-Long, provides encouraging evidence that this choice is useful. Predicting in latent space also avoids the need to synthesize future video frames.

  2. The evaluation covers several relevant settings. In addition to standard LIBERO, the authors test perturbations in LIBERO-Plus and demonstrate single-arm and dual-arm execution on real robots. These results provide useful evidence that the architecture can support practical control, and the internal ablations add more than an overall benchmark ranking.

Critical weaknesses
  1. The RoboTwin comparison is not sufficiently matched. Table 8 and the appendix's limitations state that Devol-ONE uses single-task fine-tuning, whereas the listed RoboTwin baselines are co-trained on all 50 tasks. Single-task specialization and multi-task transfer can affect performance differently, so the comparison does not tell us how much of the observed difference comes from the architecture. The appendix acknowledges this limitation, which is helpful, but the main paper still uses these results as important evidence for robustness. This matters because robustness under distribution shift is one of the paper's main motivations. A comparison with a strong baseline under the same task-training regime would make that conclusion much more convincing. Table 8 already provides the full 50-task summary (58.0% Clean / 62.2% Randomized); the introduction instead gives 34-task averages (61.1% / 63.7%). Please clarify the evaluated task sets and bring the full-suite summary into the main text alongside the four selected tasks.

  2. The interpretation of the dynamics model needs clarification. The motivation discusses how the scene evolves under candidate actions, but the dynamics stream predicts future latents from observations and task context without explicitly conditioning on those actions. This is a reasonable design for providing task-conditioned future features, but it does not directly evaluate what would happen under different candidate actions. I do not regard this as an architectural error; the paper should explain the intended role of the forecast and frame its contribution accordingly. ALRT is also a sensible way to expose the policy to prediction errors during training. Its additional benefit would be easier to assess with a direct teacher-forcing comparison, since the existing fusion and horizon studies address other design choices.

Questions for authors
  1. Can authors compare Devol-ONE with a strong RoboTwin baseline under the same task-training setting? A focused comparison on a clearly specified set of tasks would already be informative. Please also explain the choice of tasks highlighted in the main paper, reconcile the 34-task averages in the introduction with the 50-task summary in Table 8, and report the full-suite summary in the main text.

  2. Should the dynamics predictions be understood as task-conditioned forecasts rather than predictions of candidate-action consequences? Please explain the reason for the directed dependency and how this choice relates to the paper's motivation. A clearer account of this point would help position the methodological contribution.

  3. If feasible, could you add a direct teacher-forcing versus ALRT comparison with the rest of the model and training setup fixed? This would clarify the benefit of the proposed training strategy without requiring an extensive new experimental study.

2 comments

We hereby genuinely be grateful for your careful review.

Q1 (RoboTwin). Thank you for pointing this out. We were trying to release the full RoboTwin results in time, but the full 50-task run finished very late, right before the deadline. That is why the Introduction still has the 34-task average (61.1% / 63.7%), while Table 8 has the full 50-task result (58.0% Clean / 62.2% Randomized). The 34-task number is only the average over the tasks finished at that time, and the 50-task number is the final one. The full details were too long for the main paper, so we put them in the appendix. In the camera-ready version we will make the numbers consistent, put the 50-task summary in the main text, and explain how the four tasks in the main paper were chosen. We also agree that single-task fine-tuning for ours and co-training for the baselines is not the same protocol, so we will reword the RoboTwin claims and not use them as direct evidence for the architecture itself.

Q2 Yes, the dynamics prediction should be understood as a task-conditioned forecast, not a prediction of candidate-action consequences. We do not feed actions into the JEPA flow predictor and we do not use action latents as supervision for it. The predictor only predicts future V-JEPA 2 latents from the current observation and the language instruction. So it cannot tell what would happen under different candidate actions.

The forecast is meant to tell the policy what the scene is expected to look like as the task progresses. The action expert reads it at every layer through directed attention, so the action chunk is generated given the anticipated future, not the future derived from the action. We keep the dependency directed (VL → dynamics → action) because the predictor is only trained to output the future latent distribution of V-JEPA 2, and at deployment only the current frame and the instruction are available, so the forecast has to be computed without knowing the action. Table 5 shows this information is useful: JEPA-only conditioning beats VL-only (96.9 vs. 96.0 average, 94.2 vs. 92.5 on LIBERO-Long), and layerwise fusion is the best (98.4 average, 97.0 on LIBERO-Long).

In the revision we will change the motivation, abstract and contributions to "task-conditioned latent forecasting", add a paragraph in the Method section explaining the information flow, and list action-conditioned dynamics as a limitation and future work.

Q3 (teacher forcing vs. ALRT). We will add a direct comparison between teacher forcing and ALRT in the camera-ready version, on LIBERO and zero-shot LIBERO-Plus, with the same architecture, data and training budget.

For the comparison, student forcing (ALRT) means the predictor runs on its own outputs for the whole rollout. Teacher forcing means we directly use the ground-truth output of the V-JEPA 2 encoder, since the predictor's goal is just to predict future latents from the V-JEPA 2 output distribution. Inference is the same for both, because a VLA only sees the current frame at deployment. Our expectation is that if the predictor itself is good enough, teacher forcing is not necessary. It may give some improvement, but it relies on ground-truth future frames that do not exist in real use.

Thank you for the response and for clarifying the points raised in my review.

RoboTwin comparison The explanation of the different task-count summaries is helpful. I also appreciate the commitment to include the full-suite summary in the main text and explain the selected tasks. Regarding the training protocols, I agree that narrowing the claims is appropriate: the current results can demonstrate system performance, but they cannot clearly separate architectural improvements from differences between per-task fine-tuning and joint training. The response therefore addresses how the comparison should be interpreted, while the comparative evidence itself remains limited.

Role of the dynamics model The explanation that the model provides task-conditioned forecasts, rather than predictions of candidate-action consequences, resolves my question about its intended role. This is a reasonable design for supplying predictive context to the action expert. I would regard this point as clarified, with the motivation and contribution statements revised accordingly as proposed.

ALRT I appreciate the explanation of the training setup and the plan to compare ALRT with teacher forcing. The response does not yet include results from that comparison, so the additional benefit of ALRT remains uncertain. The existing fusion and horizon studies are useful, but address different design choices. A direct comparison would strengthen this part of the paper; I do not interpret its absence as evidence that ALRT is ineffective. Overall, the response has clarified the intended method and some reporting issues. I continue to acknowledge the value of the layerwise fusion studies and real-world results. However, the limitations of the RoboTwin comparison and the remaining uncertainty about ALRT still leave me insufficiently convinced of the strength of the improvement claims. I therefore maintain my rating of 2/4 (Weak rejection), while recognizing the useful clarifications provided by the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.