acceptodds
Under review as a conference paper at ICLR 2027

RealtimeWAM: One-Step Asynchronous World Action Models

18 views

Abstract

World Action Models (WAMs) incorporate visual representations from video generation backbones to guide action prediction. Recent efficient WAMs adopt Mixture-of-Transformers (MoT) architectures and compute video representations once for reuse by the action expert. However, intra-expert iteration (\ie, multi-step action denoising) and inter-expert waiting (\ie, sequential execution of the video and action experts) still limit inference efficiency. To this end, we present RealtimeWAM, an extremely efficient WAM variant with one-step action generation and asynchronous inference, addressing these two bottlenecks. To reduce intra-expert iteration, we propose Teacher-Anchored Consistency Distillation (TACD) to address a local-global error gap: low local consistency error alone does not guarantee accurate final actions. TACD supplements local consistency with explicit supervision from the frozen teacher's multi-step rollout endpoint, enabling accurate one-step action generation. Additionally, we propose Cross-Expert Wavefront Pipelining (CEWP) to eliminate unnecessary expert-level waiting. It overlaps the two experts through block-wise sharing of the video KV cache, synchronizing only immediately before the corresponding action attention consumes it. Extensive experiments across diverse benchmarks (\eg, LIBERO, LIBERO-Plus and RoboTwin) and model variants (\eg, Fast-WAM and Faster-WAM) demonstrate the superiority of RealtimeWAM. Notably, RealtimeWAM maintains near-lossless performance (\ie, drop) across these benchmarks while delivering significant end-to-end speedup (\eg, on H100).

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_B9XdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to accelerate the infernce of WAMs using Mix-of-Transformers architectures. The proposed methods include TACD and CEWP. TACD improves the consistency between the student model and the endpoint while reducing denoising steps. CEWP mitigates the inter-expert waiting through a KV pipeline.

Critical strengths
  1. Accelerating WAMs is definitely a very important research topic.
  2. The paper has deeply analyzed the time comsumption during WAMs' inference, and further proposed two corresponding methods.
  3. The experimental results are promising, showing substantial speed-up while maintaining the accuracy.
Critical weaknesses
  1. To my opinion, the contribution and novelty of this paper is limited. (1) TACD seems to add an item of teacher anchor to the loss function of vanilla consistency distillation. (2) CEWP is the natural extension of the idea of pipeline operation.
  2. This paper provides no new ideas about the inference acceleration of WAMs, compared with strong baselines such as FastWAM and FasterWAM.
  3. The two main methods on this paper, TACD and CEWP, seems to be independent from each other, which makes this paper a combination of two irrelevant papers. If there is no other links between the two main parts, this paper should be treated as two submissions.
Questions for authors
  1. Please justify the contribution of this paper, addressing my concern in weakness #1 (1) and (2).
  2. What is the logical links between TACD and CWEP? I think these two methods seem to be independent, respectively accelerating the WAMs from two differenct points.
Reviewer_H3RrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims at solving the inference efficiency problems on WAMs. The authors propose Teacher-Anchored Consistency Distillation (TACD) and Cross Expert Wavefront Pipelining (CEWP) to save computation cost as well as maintaining the model performances. The authors evaluate the methods on different simulation benchmarks and show significant end-to-end speedup.

Critical strengths
  1. The authors propose two major techniques to improve WAM inference efficiency: Teacher-Anchored Consistency Distillation (TACD), which addresses the local-global error gap by augmenting local consistency supervision with a teacher endpoint constraint for one-step action generation; Cross-Expert Wavefront Pipelining (CEWP), which reduces unnecessary synchronization by overlapping block-wise execution of the Video and Action Experts according to their data dependencies. With these two techniques, the inference latency gets 4.56x-5.08x and 1.22x-1.34x improvement, respectively.
  2. The authors also introduce other techniques such as CUDA graph to further improve the inference latency, and the overall speedups can reach 24.5× and 18.42× over the baselines.
Critical weaknesses
  1. The performance improvement on the simulation benchmark is quite impressive, can the authors clarify whether real-wold experiments have been conducted to further validate its advantages?
  2. If the paper get accepted, will the authors make the source code publicly available upon acceptance to facilitate reproducibility and future research?
Questions for authors
  1. There is another paper that also names Faster-WAM by Ma etal., showing a dramatic decrease on inference latency of WAM, can the authors give a comparison with it and clarify the benefit of RealtimeWAM.
Reviewer_ehCjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the real-time inference of world action models. The paper introduces RealtimeWAM which is tailored for Mixture-of-Transformers (MoT) World Action Models (e.g., Fast-WAM, Faster-WAM). Based on the experimental observations, the paper idenitfied two latency bottlenecks, namely Intra-expert iteration and Inter-expert waiting. Accordingly, it proposes two novel ideas: Teacher-Anchored Consistency Distillation (TACD), which supplements local consistency loss (L_CD) with an explicit velocity constraint (L_TA) anchored to a 10-step rollout from the frozen teacher action expert to achieve 1-step action generation; and Cross-Expert Wavefront Pipelining (CEWP), using dual CUDA streams and per-block event synchronization so that the action expert begins its Q/K/V projections and consumes video KV cache block-by-block rather than waiting for the entire video expert to complete. The method is evaluated across diverse benchmarks and model variant, showing near-lossless performance while delivering significant end-to-end speedup.

Critical strengths
  • The paper pinpoints the two primary latency bottlenecks in Mixture-of-Transformers (MoT) World Action Models.
  • The two proposals sound technical which are mathematically and practically proved.
  • The paper uses an efficient post-training strategy by keeping the heavy video expert completely frozen and training only rank-128 LoRA parameters on the action expert. This reduces peak training VRAM from 65.98 GiB to 24.73 GiB (a 62.5% reduction) on 16 NVIDIA H100 GPUs, making post-training accessible.
  • Across LIBERO, LIBERO-Plus, and RoboTwin 2.0, RealtimeWAM achieves 1-step action generation with less than a 1% drop in task success rate compared to multi-step teachers. It also cuts per-call latency to 12.2 ms for Fast-WAM (24.55x speedup) and 11.9 ms for Faster-WAM (18.42x speedup), bringing policy inference well within the latency budget of 30 Hz real-time robot control.
Critical weaknesses
  • The claim “the first one-step WAM to achieve near-lossless performance” may be too strong without clearly definining what near-lossless is.
  • The contribution of CEWP may be overstated. The latency breakdown in Figure 5 reveals that CEWP accounts for only a minor fraction of the total speedup. The vast majority of the cumulative of 25x speedup comes from 1-step distillation (TACD: 4.6–5.1x), generic CUDA Graph capture (2.0–2.8x), and fused Triton kernels (1.4–1.5x).
  • The method is restricted strictly on MoT architecture while increasing the training time.
Questions for authors

In addition to the weaknesses:

  • How many trials per suite/task and how many seeds are behind Tables 1–4?
  • How was t sampled in TACD training, in particular near t = 0?
1 comment

Thank you for your clarification and additional experiments. I agree with reviewer B9Xd that two main contributions are not related to each other and indeed CEWP contributes little to the final performance. This is a critical point. Considering all factors, I would like to adjust my rating to weak rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.