- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes WATO, a method that trains joint World Action Models (WAMs) through differentiable trajectory optimization to improve the task effectiveness and dynamics consistency of generated trajectories. The method is evaluated across seven visual goal-conditioned control environments. Experimental results show that WATO improves the average success rate of the imitation-based joint WAM from 79.9% to 91.7%. Additionally, WATO achieves an average 11.5x deployment speedup over CEM while substantially reducing the number of world model evaluations.
Critical strengths
- WATO performs computationally expensive trajectory optimization only during training, providing a clear advantage in deployment efficiency compared to CEM.
- The method is supported by a relatively clear mathematical formulation and theoretical analysis.
Critical weaknesses
- OGBench is designed to evaluate challenging capabilities such as long-horizon navigation and multi-stage manipulation, with episode lengths ranging from 200 to 4,000 steps across environments [1]. However, the authors define the goal as the state only 25 steps after the initial state. This substantially changes the task difficulty and deviates from the intended evaluation setting of OGBench. Consequently, I am concerned that the reported results on these short-horizon tasks are insufficient to establish the effectiveness of WATO, particularly for long-horizon goal-reaching problems. The authors should evaluate WATO under the standard OGBench evaluation protocol and provide comparisons with relevant methods under the same setting.
- A primary motivation of this paper is that trajectory imitation alone cannot guarantee task effectiveness. However, existing offline goal-conditioned RL methods have already explicitly addressed goal-reaching performance through value learning, hierarchical subgoal planning, and temporal abstraction [2, 3]. The paper does not sufficiently clarify the advantages of WATO over these established approaches or explain what additional capabilities are enabled by differentiable trajectory optimization.
- WATO jointly generates actions and future states and optimizes the resulting trajectories. However, HIQL [2] has demonstrated that predicting compact subgoal, together with a goal-conditioned low-level policy, can already achieve strong performance on long-horizon tasks. Therefore, it remains unclear whether generating complete future-state trajectories is necessary for the reported improvements, or whether simpler subgoal-based guidance could achieve comparable performance with lower modeling complexity.
[1] Park S, Frans K, Eysenbach B, et al. Ogbench: Benchmarking offline goal-conditioned rl[C]//International Conference on Learning Representations. 2025, 2025: 94937-94982.
[2] Park S, Ghosh D, Eysenbach B, et al. Hiql: Offline goal-conditioned rl with latent states as actions[J]. Advances in Neural Information Processing Systems, 2023, 36: 34866-34891.
[3] Ahn H, Choi H, Han J, et al. Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning[J]. Advances in Neural Information Processing Systems, 2026, 38: 99833-99861.
Questions for authors
- What irreplaceable advantages does the full Joint WAM have over simpler future-state guidance mechanisms such as HIQL? And can authors further compare it with state references provided by independent subgoal predictors?
Thank you for the positive assessment of our presentation, empirical results, and ablations. We appreciate the concerns about conceptual novelty and the discrepancy between Tables 1 and 10. We clarify both below.
Q1. Is WATO simply a combination of existing techniques?
We agree that joint world-action generation and differentiable trajectory optimization are not individually new. WATO's contribution is how these components are coupled to train the joint generator through its optimized solution, rather than merely using optimization to refine generated trajectories.
Specifically, generated actions and future states parameterize the optimization objective:
J(\mathcal R_\theta(z_t,A),g) +\lambda_A d_A(A,\tilde A) +\lambda_Z d_Z(\mathcal R_\theta(z_t,A),\tilde Z).$$ This provides three key properties: 1. **Generated futures guide action optimization.** The state trajectory is not an auxiliary prediction or merely an initialization. It directly influences optimized actions through the world-model rollout Jacobian, as established in Proposition 1. 2. **Learning through the optimized solution.** WATO differentiates the task loss through the optimizer's implicit solution sensitivity, providing task gradients to both generated actions and future states. This differs from direct rollout training, consistency regularization, and optimization distillation. 3. **Optimization amortization.** The iterative optimizer improves the generator during training and is removed at deployment. Only generation, world-model evaluation, and lightweight best-of-$K$ selection remain. Unlike inference-time trajectory refinement, WATO uses differentiable optimization as a **learning mechanism for joint generative models**. ### Q2. Does this distinction provide measurable benefits? Yes. We isolate these mechanisms through controlled ablations using the same joint WAF architecture, offline data, and pretrained world model. **Training-objective ablation (Fig. 3 and Table 9):** | Training objective | Success (%) | |---|---:| | Joint WAF (imitation) | 79.9 | | Direct rollout | 85.7 | | Direct + consistency | 87.4 | | Distillation | 88.0 | | **WATO** | **91.7** | Direct rollout already backpropagates task gradients through the world model. Consistency additionally aligns generated states with predicted states. Distillation uses optimized trajectories as detached targets. **WATO outperforms all three alternatives.** This isolates the benefit of differentiating through the optimized solution rather than simply combining trajectory imitation with task optimization. **Reference-design ablation (Table 9):** | Reference design | Success (%) | |---|---:| | Action only | 85.7 | | State only | 85.0 | | Rollout-derived state | 87.9 | | **Full WATO** | **91.7** | Replacing the generated future with the world-model rollout retains dynamics-aware regularization but reduces success by 3.8 percentage points. This supports the benefit of using **generated future states as control references**, beyond action generation or dynamics regularization alone. Furthermore, Table 2 shows that applying the same trajectory optimizer improves success by 10.4 percentage points before WATO training but only 1.3 points afterward. Thus, most of the optimizer's benefit has been transferred into the generator. At deployment, WATO achieves **11.5x lower latency and 600.5x fewer world-model evaluations than CEM**. Together, these results support our central contribution: **trajectory optimization is not merely applied to a WAM; it becomes a mechanism for learning better world-action generation.** ### Q3. Why does scene-play report 94 ± 1% in Table 1 but 99% in Table 10? Thank you for catching this error! The 99% in Table 10 is a reporting mistake. During table preparation, we accidentally entered the result from **a single seed (99%, seed 3)** instead of the mean across four seeds. The correct result for $L_{\mathrm{train}}=25$, $D=25$ is **94%**, consistent with the **94 ± 1%** reported in Table 1. We will correct the corresponding entry in Table 10 from **99 to 94** in the revised manuscript. This is a transcription error, not a difference in the experimental protocol. **Table 1 already reports the correct four-seed result**, and neither the main performance comparison nor the reported 91.7% overall WATO success is affected.