acceptodds
Under review as a conference paper at ICLR 2027

AHA-WAM: Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routing

77 views

Abstract

World-action models have emerged as a promising paradigm for robot manipulation, jointly modeling visual scene dynamics and actions to inject physical priors into policy learning. However, existing world-action models couple world prediction and action execution at the same temporal resolution, forcing the world branch to model near-term frame variations that are redundant and weakly informative. We posit that strictly binding world prediction and action execution to the same temporal rhythm may underutilize the potential of the video branch for embodied control. Therefore, we propose AHA-WAM, an Asynchronous Horizon-Adaptive World-Action Model built on a dual Diffusion Transformer (DiT) architecture that reorganizes world-action modeling around this temporal asymmetry. AHA-WAM instantiates the video DiT as a low-frequency world planner that maintains rolling key-value memory over past observations and exposes reusable layerwise latent context encoding long-horizon scene evolution, while a high-frequency action DiT executes short action chunks in closed loop by querying this context through layerwise joint attention. To support asynchronous execution, we introduce horizon-adaptive offset training and Observation-Guided Video-Context Routing (OVCR), which together let the action expert exploit long-horizon world context while remaining responsive to real-time execution state without rerunning the video DiT. Experiments on RoboTwin and real-world manipulation tasks show that AHA-WAM achieves state-of-the-art performance without any robot-data pretraining, attaining **92.58%** average success on RoboTwin and **78.3%** success across 4 real-world tasks, while reaching **24.17** Hz closed-loop control with a **4.59** x speedup over Fast-WAM.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Embodied Manipulation, World-Action Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_1m1bOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper decouples video representation updates from action execution in world action models. AHA-WAM uses a low-frequency video DiT to maintain historical K/V memory and reusable layer-wise context, while a high-frequency action DiT predicts short action chunks from current observations and proprioception. Its main mechanism OVCR compresses current visual observations into queries and adapts cached video K/V through layer-wise retrieval and gated residual updates. Training also includes action-start phase randomization and rolling historical memory. AHA-WAM reports 92.80% average success across 50 RoboTwin 2.0 tasks and 78.33% across four real-world tasks, with component ablations, environmental-change evaluations, and deployment analyses. The optimized full model achieves 41.37 ms action-chunk inference latency. The distilled two-step AHA-WAM-Flash reduces latency to 17.56 ms while retaining 90.20% average success on RoboTwin.

Critical strengths
  1. Low latency with strong manipulation performance is the main practical contribution. The full model achieves 41.37 ms action-chunk latency and 92.80% average success on 50 RoboTwin tasks. The Flash version reduces latency to 17.56 ms while retaining 90.20% success, making large-video-model policies more attractive for deployment.
  2. OVCR is an effective cache adaptation mechanism and the main methodological contribution. It updates reused video K/V layer by layer using current observations, without rerunning the video DiT. Success drops from 92.80% to 88.71% without current observations and to 89.27% without layer-wise routing; direct asynchronous reuse achieves 88.60%.
  3. The complete system delivers practically valuable manipulation results. AHA-WAM achieves 78.33% across four real-world tasks involving deformable objects, tidying, tool use, and bimanual manipulation.
  4. Phase randomization is a simple, effective training improvement. It exposes the action model to misalignment between video windows and action chunks, helping it handle phase shifts during deployment.
Critical weaknesses
  1. The efficiency gain is not clearly attributable to asynchronous modeling. Fast-WAM’s official 190 ms latency is compared with AHA-WAM’s heavily optimized 41.37 ms on an RTX 5090D, while AHA-WAM’s eager latency is 415.77 ms.
  2. OVCR’s benefits are confounded by added capacity and visual representation. The new modules add 1.22B parameters, exceeding the 1.02B action DiT, without capacity-matched visual-conditioning or cross-attention controls.
  3. Generalization evidence is limited, and demo selection raises cherry-picking concerns. Baseline successes are omitted despite π0.5/Fast-WAM achieving 76.67%/68.33% in original conditions and π0.5 outperforming AHA-WAM overall on OOD tasks. Selection criteria and failure mechanisms need clarification.
  4. Typo issue. The text accompanying Tables 13 and 14 should be updated to match the current results reported in the tables.
Questions for authors

Could the author explain why video is preferred over semantic representations, memory, or future images for guiding actions?

Reviewer_rxjEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AHA-WAM separates world-action modeling into a low-frequency video DiT and a high-frequency action DiT. The video branch provides cached layer-wise context, while observation-guided video-context routing (OVCR) adapts this context using current visual observations before action prediction. Rolling KV memory and randomized action-start offsets support context reuse. Experiments evaluate manipulation success on RoboTwin and four real-world tasks, alongside deployment optimizations and an action-distilled variant.

Critical strengths

The paper addresses a practical limitation of world-action models: repeatedly running a large video backbone can restrict feedback frequency. Updating cached context using current observations is a sensible alternative to either recomputing the backbone or reusing stale features unchanged. The results show that the complete system maintains strong manipulation performance, and the OVCR ablations support the importance of current-observation conditioning and layer-wise interaction. Evaluation across 50 simulated tasks and physical manipulation tasks is valuable, and the detailed deployment breakdown makes the implementation contribution concrete.

Critical weaknesses
  1. The latency comparisons do not isolate the benefit of asynchronous modeling. Table 3 uses the official Fast-WAM latency, whereas AHA-WAM is measured on an RTX 5090 with extensive TensorRT, CUDA Graph, and runtime optimizations. Appendix D shows that these optimizations alone reduce AHA-WAM latency from 415.77 to 41.37 ms. The reported 4.59× speedup therefore cannot be attributed directly to the proposed architecture without comparable hardware and software treatment. The planner refresh rate, context age, and action latency under concurrent planner execution also need clearer reporting to establish sustained closed-loop performance.

  2. The claim that the interpretation of the video branch as a long-horizon world planner is stronger than the evidence establishes. The action branch cannot attend to future video tokens, and future prediction is removed at inference; it consumes observation-derived features trained with future-video supervision. This can produce useful predictive representations, but does not by itself demonstrate that the cached context functions as a future plan. Table 15 supports a benefit from longer video supervision, yet changes in auxiliary prediction difficulty and representation learning remain alternative explanations. Similarly, the offset-training ablation shows only a small aggregate improvement and does not establish robustness to arbitrary planner delays, as the appendix appropriately acknowledges.

  3. Several inconsistencies make the component and performance claims harder to assess. Table 2 describes progressively adding memory and OVCR, but the “+ OVCR” row scores 91.47%, leaving the additional change that produces the full model’s 92.80% unspecified. If these are independent ablations, the text should say so. The abstract also associates the simulation and real-world results with no robot-data pretraining, while Section 4.4 explicitly pretrains AHA-WAM on RoboCOIN for real-world deployment. Finally, the 78.33% versus 76.67% real-world comparison corresponds to one successful trial under the appendix’s protocol, so it supports competitive performance rather than a clear improvement over π0.5.

Questions for authors
  1. Are matched-hardware, comparably optimized Fast-WAM or synchronous-model timings available? Please report the planner refresh cadence and clarify whether action latency is measured with concurrent planner execution.

  2. What distinguishes the “+ OVCR” configuration from full AHA-WAM in Table 2? Please reconcile this with the component ablations and identify the contribution of randomized offsets.

  3. What existing evidence shows that the cached context provides long-horizon planning beyond observation features learned through future-prediction supervision?

  4. Please clarify the pretraining scope and real-world trial counts: Figure 3 states 30 trials, while Appendix C specifies 15 per task per condition. Please also provide available uncertainty estimates for the small success-rate differences.

Reviewer_J18zOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AHA-WAM, an asynchronous world-action model that decouples long-horizon world prediction from high-frequency action execution through a dual-DiT architecture. The video DiT serves as a low-frequency world planner, maintaining rolling K/V memory and reusable layerwise planner context, while the action DiT generates short action chunks in closed loop. To address the mismatch between cached planner context and the current execution state, the authors introduce Observation-Guided Video-Context Routing (OVCR), which adapts the cached video context using the latest visual observations without rerunning the video DiT. Horizon-adaptive offset training further supports asynchronous execution. Experiments on RoboTwin and real-world manipulation tasks demonstrate that AHA-WAM maintains competitive manipulation performance while substantially reducing inference latency, achieving 24.17 Hz closed-loop control with a reported 4.59× speedup over Fast-WAM.

Critical strengths
  1. Clear motivation and well-designed framework. The paper identifies a meaningful limitation in existing world-action models, where world prediction and action execution operate at the same temporal resolution. The proposed asynchronous dual-DiT architecture provides a reasonable approach to addressing this issue. In particular, the authors consider the potential temporal mismatch introduced by reusing cached planner context and develop OVCR and horizon-adaptive offset training to address it. The motivation and method design are clearly presented and form a coherent framework for efficient closed-loop control.

  2. Comprehensive experimental evaluation and promising practical performance. The paper provides extensive experiments on RoboTwin 2.0 and real-world robotic manipulation tasks, along with ablation studies of the proposed components. The results demonstrate competitive manipulation performance while achieving substantially reduced inference latency. The real-world deployment experiments are particularly valuable, as they demonstrate the practical potential of the proposed framework beyond simulation.

Critical weaknesses
  1. Unclear contribution of the asynchronous architecture to inference efficiency. The paper reports a 4.59× speedup over Fast-WAM, but the comparison is not conducted under equivalent inference optimization settings. As shown in Table 9, AHA-WAM benefits substantially from GPU inference optimizations, including TensorRT and CUDA Graphs, while the Fast-WAM latency is taken from its original reported results. Therefore, it remains unclear how much of the speedup comes from the proposed asynchronous architecture rather than these engineering optimizations. Since inference efficiency is a central contribution of this work, the lack of a controlled comparison makes it difficult to assess the actual efficiency gains attributable to the proposed method.

  2. Insufficient evidence that OVCR specifically addresses temporal misalignment. The paper introduces OVCR to address the mismatch between cached planner context and the current execution state during asynchronous execution. However, OVCR primarily relies on observation-guided queries and gated residual updates to adapt the cached video context. Although Table 13 demonstrates the effectiveness of its components, it does not clearly establish whether the improvements come from mitigating temporal misalignment or simply incorporating additional visual information. Furthermore, the paper lacks comparisons with simpler observation fusion approaches and systematic evaluations under different context staleness levels or planner refresh intervals. This makes it difficult to assess whether OVCR is necessary for addressing the proposed problem, which weakens the evidence supporting its core methodological contribution.

Questions for authors
  1. How much of the reported 4.59× speedup can be attributed to the proposed asynchronous architecture itself? Table 9 shows substantial latency reductions from GPU inference optimizations, including TensorRT and CUDA Graphs, while the Fast-WAM latency is taken from its original reported results. Could the authors provide a comparison under equivalent hardware and inference optimization settings, or clarify the efficiency gains specifically attributable to asynchronous execution?

  2. What evidence demonstrates that OVCR specifically addresses temporal misalignment rather than simply improving visual feature fusion? Although Table 13 provides component ablations, it remains unclear whether the proposed routing mechanism offers advantages over simpler observation fusion approaches. Could the authors provide such a comparison or evaluate OVCR under different levels of context staleness and planner refresh intervals to better support its effectiveness?

I am open to revising my rating upward if the authors can adequately address these concerns in their response.

1 comment

Thank you for the detailed response and additional experiments. My main concerns have been adequately addressed, and I am inclined to revise my rating upward to Weak Accept.

Reviewer_789COpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AHA-WAM, a world-action model that decouples low-frequency video-context updates from high-frequency action generation. A video DiT maintains rolling historical K/V memory, while an action DiT reuses its cached context across action chunks. Observation-Guided Video-Context Routing (OVCR) adapts this context using the latest observation, and randomized offset training addresses temporal misalignment between the two branches. The paper also introduces an action-distilled Flash variant and deployment optimizations. Evaluation covers RoboTwin 2.0, four real-world manipulation tasks, distribution shifts, component ablations, and latency measurements. The paper reports 92.80% success on RoboTwin and 78.33% on the real-world tasks, with optimized action-chunk latencies of 41.37 ms for AHA-WAM and 17.56 ms for AHA-WAM-Flash.

Critical strengths

1. A well-motivated design addressing practical bottlenecks in WAM deployment.
The paper tackles the tension between expensive video-model computation, historical context, and frequent feedback during action execution. Reusing planner context while adapting it to new observations provides a coherent approach to reducing repeated computation without eliminating visual feedback. This addresses an important deployment problem and makes the proposed interface practically relevant.

2. Substantial ablation coverage.
The study examines OVCR components, offset training, temporal horizons, distillation, and deployment optimizations. This provides a substantial empirical basis for assessing the system, and the reported manipulation performance and optimized latency indicate a promising performance–efficiency trade-off.

Critical weaknesses

1. Insufficient evidence for the proposed mechanisms.
The selected simulation benchmarks and real-world tasks do not specifically target scenarios demanding low-latency inference or substantial historical context, leaving AHA-WAM’s central claims insufficiently validated. Ambiguous component definitions in Table 2 and inconsistent reporting across Tables 13–15 further weaken attribution. Consequently, it remains unclear whether rolling memory and OVCR explain the performance gains.

2. The efficiency comparison mixes architectural and implementation gains.
Fast-WAM’s 190 ms latency is taken from its official report, whereas AHA-WAM receives extensive deployment optimizations. Table 9 shows that these optimizations reduce AHA-WAM’s latency from 415.77 to 41.37 ms. It remains unclear how much advantage would persist if comparable applicable optimizations were applied while retaining Fast-WAM’s inference scheme. Without this controlled comparison, the architectural contribution to the reported 4.59× speedup is difficult to establish.

Questions for authors

1. Suggested additional evaluation.
Could you report success rates on tasks specifically testing rapid reactions or dependence on historical information? Examples include dynamic manipulation benchmarks such as DOMINO and memory-focused benchmarks. If feasible, real-world experiments could include tasks requiring memory or rapid responses. These evaluations would more directly test AHA-WAM’s central claims.

2. Clarification of reporting inconsistencies.
The following inconsistencies undermine confidence in the reported results and make the ablation findings difficult to interpret. Could you correct or explain them?

  • Table 13, removing current-observation conditioning: The text reports a decrease of 3.87 percentage points, whereas the table gives 92.80 − 88.71 = 4.09.
  • Table 14, randomized offset training: The text reports an improvement of 0.33 percentage points, whereas the table gives 92.80 − 92.25 = 0.55.
  • Table 15, increasing HH from 32 to 64 with C=16C=16: The text reports an improvement of 2.31 percentage points, whereas the table gives 92.80 − 90.27 = 2.53.
  • Table 14 versus the main full-model results: Table 14 reports 93.00% Clean / 92.60% Randomized, while the main tables report 93.40% / 92.20%. Do these results correspond to different runs or configurations?

3. Controlled efficiency comparison.
How much inference-efficiency advantage remains over Fast-WAM under matched hardware, inference settings, and comparable applicable optimizations, while retaining Fast-WAM’s inference scheme?

4. Comparison with memory-augmented WAMs.
How does AHA-WAM compare with dedicated memory-augmented methods such as MemoryWAM? Comparisons on memory-dependent benchmarks such as RMBench, reporting both success rates and inference latency under comparable conditions, would help establish whether rolling memory and OVCR offer advantages over existing memory designs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.