acceptodds
Under review as a conference paper at ICLR 2027

HiLo-WAM: Decoupling Long-Horizon Task Evolution and Local Dynamics in World Action Models

Abstract

World Action Models (WAMs) improve robotic manipulation by jointly modeling visual dynamics and robot actions. However, existing WAMs primarily predict dense videos over short temporal windows, providing strong local interaction modeling but limited understanding of long-horizon task evolution. Extending visual prediction over longer horizons can provide richer task context, but also increases prediction uncertainty and computation, while forcing task-level evolution and fine-grained local dynamics to be modeled within a single temporal process. More importantly, this entangles different temporal dynamics, obscuring their dependencies and distinct roles in action generation. We propose HiLo-WAM, which decouples long-horizon task evolution from local visual dynamics while jointly modeling their interaction for action generation. A long-horizon visual stream captures historical task context and coarse future evolution, whereas a local visual dynamics stream models dense short-horizon robot–object interactions. The two streams operate at different temporal scales within a unified dual-DiT architecture and jointly guide robot actions through structured cross-stream interaction. Progressive stream-wise inference further reduces unnecessary world-model computation. Experiments on simulation and real-world manipulation tasks demonstrate strong long-horizon manipulation performance. HiLo-WAM achieves a 93.70% average success rate on RoboTwin 2.0 and a 17.43 long-horizon progress score on RoboDojo, while also maintaining robust performance on the Fourier GR-3 robot under both in-domain and OOD settings. Ablations further verify the complementary contributions of long-horizon task evolution and local interaction dynamics to action generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, Multi-Scale Visual Streams, Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dV5fOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper decouples long-horizon manipulation in WAM into coarse long horizon prediction and fine-grained local dynamic prediction, using two streams that differ in their conditioning. The local stream, conditioned on the current observation, predicts dense-near future; the long horizon stream, conditioned on the chunk of the history observation, predicts sparse, downsampled futures over a brader horizon. Together, they yield actions that are both locally precise and aware of long-range dependencies.

Critical strengths

The paper is exceptionally well written. The method is presented concisely and without unnecessary detail, and the exposition is clear and reader friendly. The conclusions are easy to follow. In addition, the authors have made a substantial effort in implementing and evaluating a broad set of baselines, resulting in a thorough and convincing comparison.

Critical weaknesses
  1. Problem framing: Narrowing the problem to the long-horizon ability of WAMs seems unnecessary. A WAM is only a tool; long-horizon manipulation itself is the more important problem.
  2. Missing baselines: The paper does not compare against methods that are known to handle long-horizon tasks well, such as open-world TAMP systems and agent-based approaches. Without these comparisons, it is unclear why a WAM is necessary for these tasks.
  3. Insufficient evaluation of the long-horizon claim: General benchmarks such as RoboDojo are not sufficient to support the paper's long-horizon claim, and it remains unclear whether the gains come from the proposed design or simply from additional parameters.
Questions for authors
  1. Is handling long-horizon dependencies implicitly a better design than handling them explicitly (e.g., with TAMP)?
  2. Where is long-horizon reasoning required in the test cases? A case study on subtask dependencies (e.g., opening a drawer occluding later manipulation) would help.
  3. Do the gains come from the design rather than more parameters?
  4. How do the predictions and actions behave under multiple possible outcomes, such as a ball bouncing in different directions?
1 comment

I thank the authors for the additional experiments and for reporting the results candidly. They address most of my earlier concerns.

The two-stream policy is simple. Compared with the baselines, it improves on some dimensions and shows no gain on others. I see these mixed results as a strength, not a weakness: negative results are valuable when they are presented clearly. To get the most out of them, I encourage the authors to analyze when the two-stream design helps and when it does not, and to discuss possible reasons for each case. That analysis would turn the empirical findings into a real insight for the community.

I also encourage the authors to use the new terminology more carefully, especially “state evolution” and “local dynamics.” As currently presented, the evidence does not clearly support these interpretations.

I am willing to raise my score to 3 (weak accept) if the authors do either of the following:

  1. Provide evidence that the dense stream actually captures dynamics (how forces and torques relate to the motion they produce) and the sparse stream actually captures state (preconditions, effect, predicates) or
  2. Reframe the paper using established terminology, so that it describes the two-stream architecture plainly without claims about what each stream represents.
Reviewer_k2yeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HiLo-WAM, which separates long-horizon task evolution from short-horizon local dynamics in a World Action Model. The two visual streams operate at different temporal scales and jointly guide action generation through structured cross-stream attention, with progressive inference used to reduce computation. Experiments on RoboTwin 2.0, RoboDojo, and real-world tasks show improved long-horizon manipulation performance.

Critical strengths

1.Clear motivation and intuitive design. The separation between long-horizon task evolution and local interaction dynamics is easy to understand and well aligned with the challenges of long-horizon manipulation.

2.Reasonable analysis of the proposed components. The paper includes ablations on the long-horizon/local streams, temporal stride, and inference efficiency, which helps explain how the method works.

Critical weaknesses
  1. Limited Experimental Coverage

The simulation experiments mainly cover RoboTwin 2.0 and eight long-horizon RoboDojo tasks, without evaluation on standard long-horizon benchmarks such as CALVIN or LIBERO-Long. The RoboDojo comparison also misses recent relevant methods such as X-WAM and GigaWorld-Policy, making the comparison with current methods less complete.

  1. Limited Novelty over Recent Work

Recent works have already explored separating long-horizon task information from short-horizon/local execution or maintaining information at different temporal scales. HiLo-WAM provides a specific two-stream implementation, but the paper does not clearly show what fundamentally new advantage this decomposition brings over these closely related formulations.

Questions for authors

1.For RoboDojo, is a single multi-task checkpoint used for all eight long-horizon tasks, and what data are included in the 3,500 training episodes?

2.For the real-world experiments, are separate checkpoints trained for the three tasks, and how are the 50 hours of demonstrations distributed?

3.Does the efficiency gain of progressive inference mainly come from using fewer visual denoising steps? A matched reduced-step baseline would help clarify this.

4.How does the proposed asymmetric cross-stream attention compare with simpler alternatives such as full or symmetric attention?

1 comment

Thanks for the detailed responses and additional experiments. The new benchmark results, expanded RoboDojo comparisons, and additional ablations address most of my concerns. While some questions about novelty remain, the rebuttal has strengthened the paper, and I will raise my score to 3.

Reviewer_vU5fOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HiLo-WAM separates visual prediction into two temporal scales: sparse history and future frames capture long-horizon task evolution, while denser near-term predictions model local interactions. The streams share a video backbone and interact with an action transformer through asymmetric attention. Experiments cover RoboTwin, RoboDojo, and real-robot manipulation.

Critical strengths

The two-timescale design is well motivated, and sharing the video backbone avoids duplicating the full model. Results improve over Fast-WAM with the same pretrained video backbone, including RoboTwin success from 91.85% to 93.70% and RoboDojo progress score from 9.14 to 17.43. The real-robot results also show useful gains.

Critical weaknesses
  1. The component evidence is insufficient for the central long-horizon claim. In Table 3, history and the local stream add only 0.48 and 0.18 percentage points, respectively, without repeated-run variability. All component ablations are on RoboTwin, so they do not establish which components explain the RoboDojo gains. The claim that every component is essential is stronger than the evidence supports.

  2. The long-horizon comparison is incomplete. The RoboDojo benchmark paper reports π0.5 at 23.54 points and 14.67% success, above the submission's 17.43 and 6.50%. π0.5 is absent from Table 1 despite appearing in other experiments. These numbers require protocol alignment before a direct ranking, but omitting this strong baseline leaves the main comparison unresolved.

Questions for authors
  1. Can the authors clarify the RoboDojo versions, training data, and evaluation protocols, and include π0.5 under the same protocol?

  2. Do the history and local streams produce reliable gains on RoboDojo across repeated runs? Please also report success and latency together for early stopping versus full denoising; the default configuration lists both visual streams stopping at step 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.