acceptodds
Under review as a conference paper at ICLR 2027

Joint World Action Models through Differentiable Trajectory Optimization

Abstract

Joint world action models (WAMs) generate future actions and states together, modeling world-action trajectories rather than actions alone. However, trajectory imitation does not ensure that these trajectories are task-effective or consistent with dynamics. We present world action trajectory optimization (WATO), which optimizes generated actions and future states jointly for control. WATO uses both as references in a dynamics-constrained trajectory optimization problem and differentiates the task loss through the optimized trajectory to train the generator. This makes generated trajectories task-aware and dynamics-consistent while using trajectory optimization only during training. At deployment, WATO directly generates candidate trajectories and ranks them with a world model, without iterative trajectory optimization. Across visual control environments, WATO improves average success from for the imitation joint WAM to , outperforming strong model-based, goal-conditioned RL, and generative control baselines. Compared with cross-entropy method (CEM), WATO is on average faster and uses fewer world model evaluations at deployment.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, World Models, Flow Matching, Trajectory Optimization, Visual Offline Goal-Conditioned Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RJioOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes WATO, a method that trains joint World Action Models (WAMs) through differentiable trajectory optimization to improve the task effectiveness and dynamics consistency of generated trajectories. The method is evaluated across seven visual goal-conditioned control environments. Experimental results show that WATO improves the average success rate of the imitation-based joint WAM from 79.9% to 91.7%. Additionally, WATO achieves an average 11.5x deployment speedup over CEM while substantially reducing the number of world model evaluations.

Critical strengths
  1. WATO performs computationally expensive trajectory optimization only during training, providing a clear advantage in deployment efficiency compared to CEM.
  2. The method is supported by a relatively clear mathematical formulation and theoretical analysis.
Critical weaknesses
  1. OGBench is designed to evaluate challenging capabilities such as long-horizon navigation and multi-stage manipulation, with episode lengths ranging from 200 to 4,000 steps across environments [1]. However, the authors define the goal as the state only 25 steps after the initial state. This substantially changes the task difficulty and deviates from the intended evaluation setting of OGBench. Consequently, I am concerned that the reported results on these short-horizon tasks are insufficient to establish the effectiveness of WATO, particularly for long-horizon goal-reaching problems. The authors should evaluate WATO under the standard OGBench evaluation protocol and provide comparisons with relevant methods under the same setting.
  2. A primary motivation of this paper is that trajectory imitation alone cannot guarantee task effectiveness. However, existing offline goal-conditioned RL methods have already explicitly addressed goal-reaching performance through value learning, hierarchical subgoal planning, and temporal abstraction [2, 3]. The paper does not sufficiently clarify the advantages of WATO over these established approaches or explain what additional capabilities are enabled by differentiable trajectory optimization.
  3. WATO jointly generates actions and future states and optimizes the resulting trajectories. However, HIQL [2] has demonstrated that predicting compact subgoal, together with a goal-conditioned low-level policy, can already achieve strong performance on long-horizon tasks. Therefore, it remains unclear whether generating complete future-state trajectories is necessary for the reported improvements, or whether simpler subgoal-based guidance could achieve comparable performance with lower modeling complexity.

[1] Park S, Frans K, Eysenbach B, et al. Ogbench: Benchmarking offline goal-conditioned rl[C]//International Conference on Learning Representations. 2025, 2025: 94937-94982.

[2] Park S, Ghosh D, Eysenbach B, et al. Hiql: Offline goal-conditioned rl with latent states as actions[J]. Advances in Neural Information Processing Systems, 2023, 36: 34866-34891.

[3] Ahn H, Choi H, Han J, et al. Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning[J]. Advances in Neural Information Processing Systems, 2026, 38: 99833-99861.

Questions for authors
  1. What irreplaceable advantages does the full Joint WAM have over simpler future-state guidance mechanisms such as HIQL? And can authors further compare it with state references provided by independent subgoal predictors?
Reviewer_hgPNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces world action trajectory optimization (WATO), composed of 3 critical stages. First, Joint WAM jointly generates action trajectories and future state trajectories based on the current state and goal. Second, WATO optimizes both references and backpropagates through the optimized solution. Finally, the deployment ranks the candidates with a world model without online optimization. For evaluation, WATO outperforms baselines across 7 visual control environments.

Critical strengths

By integrating joint generation for both action and state reference with differentiable trajectory optimization, the designed ablation shows the benefits of each component.

Critical weaknesses

The conceptual novelty is limited. The paper seems to just combine existing techniques, since jointly action-state generation and differentiable trajectory optimization have been explored in prior work.

Questions for authors

Table 1 reports 94 ± 1% success on scene-play, whereas Table 10 reports 99% under the same setting (L=25,D=25). Could the authors clarify this discrepancy?

1 comment

Thank you for the positive assessment of our presentation, empirical results, and ablations. We appreciate the concerns about conceptual novelty and the discrepancy between Tables 1 and 10. We clarify both below.

Q1. Is WATO simply a combination of existing techniques?

We agree that joint world-action generation and differentiable trajectory optimization are not individually new. WATO's contribution is how these components are coupled to train the joint generator through its optimized solution, rather than merely using optimization to refine generated trajectories.

Specifically, generated actions and future states parameterize the optimization objective:

J(\mathcal R_\theta(z_t,A),g) +\lambda_A d_A(A,\tilde A) +\lambda_Z d_Z(\mathcal R_\theta(z_t,A),\tilde Z).$$ This provides three key properties: 1. **Generated futures guide action optimization.** The state trajectory is not an auxiliary prediction or merely an initialization. It directly influences optimized actions through the world-model rollout Jacobian, as established in Proposition 1. 2. **Learning through the optimized solution.** WATO differentiates the task loss through the optimizer's implicit solution sensitivity, providing task gradients to both generated actions and future states. This differs from direct rollout training, consistency regularization, and optimization distillation. 3. **Optimization amortization.** The iterative optimizer improves the generator during training and is removed at deployment. Only generation, world-model evaluation, and lightweight best-of-$K$ selection remain. Unlike inference-time trajectory refinement, WATO uses differentiable optimization as a **learning mechanism for joint generative models**. ### Q2. Does this distinction provide measurable benefits? Yes. We isolate these mechanisms through controlled ablations using the same joint WAF architecture, offline data, and pretrained world model. **Training-objective ablation (Fig. 3 and Table 9):** | Training objective | Success (%) | |---|---:| | Joint WAF (imitation) | 79.9 | | Direct rollout | 85.7 | | Direct + consistency | 87.4 | | Distillation | 88.0 | | **WATO** | **91.7** | Direct rollout already backpropagates task gradients through the world model. Consistency additionally aligns generated states with predicted states. Distillation uses optimized trajectories as detached targets. **WATO outperforms all three alternatives.** This isolates the benefit of differentiating through the optimized solution rather than simply combining trajectory imitation with task optimization. **Reference-design ablation (Table 9):** | Reference design | Success (%) | |---|---:| | Action only | 85.7 | | State only | 85.0 | | Rollout-derived state | 87.9 | | **Full WATO** | **91.7** | Replacing the generated future with the world-model rollout retains dynamics-aware regularization but reduces success by 3.8 percentage points. This supports the benefit of using **generated future states as control references**, beyond action generation or dynamics regularization alone. Furthermore, Table 2 shows that applying the same trajectory optimizer improves success by 10.4 percentage points before WATO training but only 1.3 points afterward. Thus, most of the optimizer's benefit has been transferred into the generator. At deployment, WATO achieves **11.5x lower latency and 600.5x fewer world-model evaluations than CEM**. Together, these results support our central contribution: **trajectory optimization is not merely applied to a WAM; it becomes a mechanism for learning better world-action generation.** ### Q3. Why does scene-play report 94 ± 1% in Table 1 but 99% in Table 10? Thank you for catching this error! The 99% in Table 10 is a reporting mistake. During table preparation, we accidentally entered the result from **a single seed (99%, seed 3)** instead of the mean across four seeds. The correct result for $L_{\mathrm{train}}=25$, $D=25$ is **94%**, consistent with the **94 ± 1%** reported in Table 1. We will correct the corresponding entry in Table 10 from **99 to 94** in the revised manuscript. This is a transcription error, not a difference in the experimental protocol. **Table 1 already reports the correct four-seed result**, and neither the main performance comparison nor the reported 91.7% overall WATO success is affected.
Reviewer_onNEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes WATO, a novel optimization framework to control generated actions and future states of WAMs jointly. After pretraining a world action flow on offline trajectory dataset, one can apply WATO to improve generated trajectories to better match the task objective. The method is evaluated on 7 vision control environments, improving success rate and speeding up planning efficiency compared to prior baselines.

Critical strengths
  • The proposed framework effectively proposes a new paradigm for trajectory optimization using joint WAMs. While prior works either focus on only action or state optimization, WATO jointly optimizes both for better control.
  • The paper is well-structured and includes thorough ablation studies (e.g., objective ablation, amortization through optimization). Quantitative results show that the WATO improves task performance and efficiency. Qualitative results show how this is achieved with better exploration space and optimized trajectory generation.
Critical weaknesses
  • Although the paper provides a theoretical motivation for optimizing joint world-action trajectories, it remains unclear under what conditions WATO reliably improves control. In particular, the framework depends on the accuracy of the pretrained world model and the suitability of its latent representation for the task objective. The paper do not systematically investigate how they may affect performance of WATO.
Questions for authors
  • While WATO is formulated independently of the generative model architecture, all experiments rely on LeWM as the pretrained dynamics model. What assumptions does WATO make about the pretrained world model, particularly its prediction accuracy and latent-space geometry? Would the framework remain effective with a representation trained under a different objective, or with a less accurate world model?
  • One potential advantage of generative WAMs is their ability to synthesize novel trajectories beyond those explicitly observed in the training data. However, WATO fine-tunes the generator toward task-specific objectives, potentially changing the diversity and coverage of its learned trajectory distribution. Does this optimization compromise generalization to unseen goals or novel combinations of behaviors?
2 comments

Thank you for the positive assessment of WATO's formulation, theories, and results. We appreciate the questions about world model dependence and trajectory diversity, as they help clarify the conditions under which WATO is expected to work.

Q1. What does WATO assume about the pretrained world model? Would it work with other representations or less accurate dynamics?

WATO is not necessarily tied to LeWM, but it sure depends on the quality of action-conditioned dynamics and the task representation.

The joint WAM proposes actions and intended futures, while the world model predicts the consequences of those actions. WATO uses

Z(A)=Rθ(zt,A)Z(A)=\mathcal{R}_\theta(z_t,A)

to optimize the generated trajectory and differentiate task performance through the optimized solution.

The formulation requires (1) differentiable action-conditioned rollouts, (2) a latent task cost that meaningfully reflects goal achievement, and (3) locally regular trajectory optimization for implicit differentiation. It does not require LeWM's particular architecture or representation-learning objective.

Importantly, dynamics consistency in WATO is defined relative to the learned world model, not the true environment. A less accurate model may produce misleading optimization gradients or favor trajectories that exploit prediction errors. WATO does not guarantee improved real-world control under arbitrary model error.

We explicitly analyze this issue in Appendix C.4. Under bounded model error, the deployment candidate-selection bound separates three effects: projected candidate quality, the gap between generated and optimized trajectories, and world-model error. Thus, better optimization alone cannot eliminate inaccurate dynamics.

All seven environments use the LeWM model family, including both released checkpoints and models trained in this work. These results establish effectiveness across different environments and trained model instances, not robustness across arbitrary world-model architectures, objectives, or accuracy levels. We agree that systematically varying dynamics accuracy and representation quality would strengthen this analysis.

Q2. Does task optimization reduce trajectory diversity or generalization to unseen goals?

This is an important tradeoff. WATO does not replace generative imitation with pure task optimization. Its training objective is:

= \mathcal{L}_{\mathrm{WAF}} + \lambda_{\mathrm{task}} \mathbb{E}[J(Z^\star,g)].$$ The flow matching term continues to model offline world-action trajectories, while the task term favors proposals that lead to better optimized outcomes. We use a small task weight, $\lambda_{\mathrm{task}}=5\times10^{-3}$, to retain imitation supervision during fine-tuning. Our evaluation uses held-out start--goal pairs, so the observed improvement is not limited to reproducing training trajectories. WATO improves mean success from **79.9% to 91.7%** over the imitation-only joint WAF. WATO also achieves **88.7% with $K=1$**, showing that improved control is not solely a result of selecting among more candidates. However, **the imitation term does not formally guarantee diversity preservation**. Our current results demonstrate improved goal-reaching generalization within the evaluated goal distribution, but do not isolate changes in trajectory diversity or establish generalization to unsupported goals and novel combinations of behaviors. A direct follow-up would measure generated action/state diversity and goal coverage before and after WATO fine-tuning, alongside evaluations on compositional goals outside the training distribution. These are distinct from the controller-quality improvements established by our current experiments.
Reviewer_onNEOpenReview ↗

Reply to authors

Thank you for the detailed rebuttal. I appreciate the authors' effort to address my concerns, especially on relationship between WATO and performance of the world model itself and trajectory diversity, respectively. While I understand that systematic analysis of different pretrained models is not within the scope of the paper, I suggest the authors to add explanation in either the abstract or introduction that the paper uses LeWM for all experiments.

Overall, the rebuttal has clarified my concerns, and I maintain my positive rating of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.