acceptodds
Under review as a conference paper at ICLR 2027

Long-WAM: Scaling the Context of World-Action Models

41 views

Abstract

Real-time robot control demands enough visual history to infer motion and task progress, but processing that history can delay action. We present Long-WAM, a model–system framework for scaling the context of causal world-action models under real-time control constraints. Our central finding is that access to history is not the same as using it: longer histories pay off far more when the video foundation is pretrained autoregressively (AR). We first learn causal prediction from robot and egocentric videos without action labels, then preserve this history-to-future structure during world-action adaptation. On RoboCasa GR-1, increasing context from 0.0 to 19.2 seconds raises success from 63.3% to 78.7%, whereas a bidirectionally pretrained initialization shows no net gain; robot-domain AR pretraining further raises peak success on GR-1 and LIBERO-Long. Long-WAM also achieves the best results among compared methods on LIBERO-Long, RoboTwin2.0, and DOMINO. Streaming observation encoding, asynchronous execution, and hardware-specific acceleration enable deployment on RTX5090, DGX Spark, and Jetson AGX Thor without dropping future prediction; on RTX5090, each action chunk, including future-video latent prediction, takes 107.4,ms. Real-time deployment on Unitree G1 and YAM supports dynamic and long-horizon manipulation, including 95% success on dynamic cup stacking, where pi0.5 and Fast-WAM succeed in none of 20 trials. As a memory-informed executor, Long-WAM also complements higher-level planning in composite tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Long-context, Real-time

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_annUOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Long-WAM studies how real observation history affects world-action control. It continues autoregressive video pretraining on robot and egocentric videos, then adapts the foundation through asymmetric video-action coupling. Actions condition on observed history and partially denoised future visual latents. Each context window uses a separately trained policy, with forecast and action horizons fixed within each benchmark. The study finds substantial history gains for both AR initializations, with different useful windows on LIBERO-Long and RoboCasa GR-1. Streaming VAE encoding, asynchronous execution, and hardware-specific inference optimizations support deployment while retaining future prediction. Evaluation includes simulation benchmarks, dynamic manipulation on Unitree G1, and sustained tabletop manipulation on YAM.

Critical strengths
  1. Useful findings on observation history. Longer history substantially improves the AR-based policies, with different useful windows across tasks and much smaller gains for the bidirectional initialization (Figure 5). This helps clarify when history benefits world-action control.

  2. Strong deployment and real-robot results. Streaming VAE encoding, asynchronous execution, and hardware-specific optimizations substantially reduce inference cost while retaining future prediction and similar task success (Tables 5-7). Dynamic manipulation and sustained execution on G1 and YAM support the system's practical value (Figures 6-7).

Critical weaknesses

The future-denoising cutoff needs a clearer rationale. Section 2.2 fixes σ⋆=0.9\sigma_{\star}=0.9 for both the action-training surrogate and inference, so actions use visual KV from a partially denoised future at this noise level. Tables 1-2 support the complete predictive path, and Appendix C.2 explains the conditioning interface. However, the paper does not explain how 0.9 was selected or why the representation at this endpoint retains the predictive information needed for control. This leaves a central choice in the predict-then-act design difficult to assess.

Questions for authors
  1. How was σ⋆=0.9\sigma_{\star}=0.9 selected, and what makes the future representation at this endpoint useful for action generation? The selection rationale or existing evidence about predictive information retained in the visual KV would help justify this choice.

  2. Appendix F reasonably explains that stronger atomic execution helps Long-WAM benefit from high-level planning. Could you elaborate on which skills or execution failure modes most clearly distinguish Long-WAM + Astra from π0.5\pi_{0.5} + Astra? Existing task-level results or representative rollout observations, where available, would help clarify this interaction.

  3. Figure 5 suggests that useful history windows depend on the task. Could you explain the choice of history sampling interval and discuss whether multi-scale history (dense recent and sparse older observations) or selective sampling could preserve useful temporal coverage with fewer inputs? The selection rationale or existing task-level observations would help guide future work on context efficiency.

1 comment
AuthorsOpenReview ↗

Response to Reviewer annU

We sincerely thank you for the supportive review and for questions that go to the core of the design.

W / Q1 — How was σ⋆ = 0.9 selected, and why does the representation at this endpoint carry what control needs?

Selection: σ⋆ = 0.9 is the endpoint of a 4-step video denoising schedule fixed a priori (1 → 0.994 → 0.983 → 0.963 → 0.900 under the shifted schedule); it was not tuned and we do not claim it is optimal. Reading the future at high noise is shared with related designs: DreamZero-Flash conditions actions on a noise-dominated visual context (σ ≈ 0.875), and LingBot-VA trains its action decoder on noisy video latents and decodes actions when the video is denoised only halfway (s = 0.5).

Why it carries predictive information: the action expert does not read the noisy future latents; it reads the per-layer keys and values that the video expert computes on [clean history | future at σ⋆]. To predict the denoising direction, the video expert must estimate the clean future internally, and that estimate is what the cache carries. Under causal attention the future slots aggregate this forecast from the history at every noise level; the high-noise stage fixes coarse layout and motion, which is what control needs.

Evidence, new: (a) Probes. On 1,058 LIBERO-Long windows, ridge probes from the visual cache the action expert reads predict the end-effector displacement 1.55 s ahead with held-out R² = 0.94 from the future tokens at σ⋆ = 0.9, vs. 0.87 from the current-frame cache alone (0.8 s ahead: 0.91 vs. 0.85). R² is 0.94–0.95 for future tokens at σ ∈ {1.0, 0.9, 0.7}; in the early layers the partially denoised content adds information (0.7 > 0.9 > 1.0). Decoding the one-step x̂₀ of the σ⋆ = 0.9 future gives frames close to the ground truth 0.8 s and 1.6 s ahead. (b) Endpoint sweep. LIBERO-Long (10 tasks × 20 episodes): σ⋆ = 1.0 / 0.95 / 0.9 / 0.8 / 0.7 / 0.5 gives 99.5 / 99.5 / 99.0 / 96.5 / 98.5 / 97.5%; denoising to 0.7 or 0.5 while keeping the KV timestep at 0.9 gives 97.0 / 98.5%. GR-1 (19.2 s model, 24 tasks × 10 episodes, paired against the unmodified model in the same batch): σ⋆ = 1.0 changes success by +1.7 points (95% CI −5 to +9), while replacing the future slots by noise inconsistent with their timestep costs 20 points and freezing the forecast for the whole episode costs 23 points.

Q2 — Which skills or failure modes distinguish Long-WAM + Astra from π0.5 + Astra?

Atomic execution is the difference. Alone on Atomic-Seen, Long-WAM succeeds in 67.9% of episodes, π0.5 in 39.6%; composite tasks multiply along the chain (three steps: about 0.31 vs. 0.06), so stronger atomic execution is what lets the planner's decomposition pay off. The gain is decomposition, not decision frequency: without the planner and re-querying every 15 steps, Long-WAM reaches 84.4% on Atomic-Seen but 11.2% / 5.0% on the composite settings, vs. 38.8% / 35.0% with Astra. On unseen instructions the planner grounds abstract requests into mastered atomic steps (e.g. "place the tool for peeling cucumbers" → "take the peeler from the drawer and put it on the cutting board"). Failure modes: of 101 failed composite episodes, 67 are the planner repeatedly rewriting the sub-instruction; 4 tasks fail in every setting (e.g. pressing the button in PrepareCoffee), the current skill ceiling. Caveat: for π0.5 + Astra only aggregates are available and we cannot confirm the planning interface matches ours, so this is a system-level comparison; our composite results use 5 episodes per task. Revision: failure analysis and protocol in App. F.

Q3 — Choice of the history sampling interval; multi-scale or selective history?

Control is 20 Hz; history is sampled every 4th step (5 Hz); the causal VAE compresses time 4×, so one latent covers 0.8 s and the 19.2 s window is 25 clean latents plus 2 imagined future latents. Uniform sampling was chosen because (i) a constant step between latents lets the dynamics learned in video pretraining transfer directly, (ii) the VAE compresses consecutive frames, so non-uniform sampling changes what one latent represents, and (iii) tokens and latency grow linearly and predictably with the window (App. I). Your suggestion matches what we observe: LIBERO-Long saturates at 2.4 s, and the 19.2 s GR-1 window holds on average only 7.1 s of real history, so dense recent plus sparse older context is the natural next step. Revision: multi-scale and selective sampling as future work in the Discussion.

Thank you again for the support.

Reviewer_mKUzOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the scaling of the context history for WAMs and proposes optimization techniques to accelerate inference. It demonstrates that autoregressive video pretraining (LongLive) produces stronger performance than bidirectional pretrained video models (Wan2.2). While it provides some insight into the scaling of context length for WAM, the methodology contribution of the work is limited. Moreover, a key baseline is missing to better evaluate the benefit of proposed long-context conditioning.

Critical strengths

• The paper demonstrates that WAMs using causal instead of bidirectional video backbones benefit more from longer context.

• The paper conducts extensive experiments on different benchmarks, including LIBERO, RoboTwin, RoboCasa GR-1, DOMINO and Unitree platform. Ablations show the effect of context window lengths on model performance.

Critical weaknesses

• While the paper report useful findings regarding the context length scaling of WAM, the methodology contribution seems limited. The key components in the work, such as causal video backbone, IDM action generation and infrastructure acceleration, were introduced before. The paper should clearly identify which mechanism are new and distinguish its methodology contribution from the empirical context length findings.

• The performance is sensitive to video context length. While Long-WAM achieves best performance with 2.4 seconds of history for LIBERO-long, it requires upto 19.2 seconds history for RoboCasa GR-1. It is unclear whether a single checkpoint can operate across different context lengths, or you have to train separate models for each context window.

• The paper lacks a key baseline of Wan2.2 based WAM trained with bidirectional objective on the same robot-video corpus with LongLive2.0-Robot. Therefore, it is hard to conclude if the advantage we see in the results comes from the video backbones, causal objective or robot continue training recipe.

Questions for authors

• Do you have the result of another baseline result switching LongLive2.0-Robot with a bidirectional video backbone like Wan2.2, trained on same robot-video data with bidirectional objective while keeping all other settings same as Long-WAM? or at least do you agree such an experiment will strengthen the claim?

8 comments
AuthorsOpenReview ↗

Response to Reviewer mKUz

Thank you for the detailed review, and for recognizing the causal vs. bidirectional finding and the breadth of the experiments. We address each concern below and say plainly where we agree and where we see it differently.

W1 — Limited methodological contribution.

We agree that the individual components have precedents; Sec. 2.2 already says the interface follows DreamZero and LingBot-VA. We also agree that the paper should separate what is new from the empirical finding more clearly. Our contributions are on that question:

  1. AR pretraining is what makes context scale. To our knowledge, Long-WAM is the first WAM that starts from an AR video foundation model and stays causal throughout: continued AR pretraining on about 10,000 hours of action-free robot and egocentric video, followed by causal video-to-action adaptation. Existing WAMs (DreamZero, LingBot-VA, AHA-WAM) instead convert a bidirectional video model into a causal one. It is also, to our knowledge, the first controlled observation of context scaling in WAMs: with the same architecture, recipe and protocol, extending history to 19.2 s lifts the AR prior to 78.7% and leaves the bidirectional prior at 61.6%.
  2. AR context models are naturally suited to asynchronous execution.* Because actions are conditioned on temporally coherent predictions from the AR foundation, plain asynchronous execution, with no action blending, prefix guidance or extra module, costs only 0.2 points on RoboTwin 2.0 (94.4% → 94.2%) and gives lower handoff RMSE and jerk than Fast-WAM and LingBot-VA (Sec. 3.1). The causal VAE also lets us encode observations as a stream while the robot is still executing, so the latest observation is used without waiting at the handoff.
  3. Real-time long-context inference on edge hardware. Shared optimisations (W4A4 NVFP4 quantisation, KV reuse, CUDA Graph replay, shared Q/K/V input quantisation, segmented attention) plus device-specific kernel tuning give roughly 3–4× speedups over the BF16 eager baseline, down to 107.4 ms on RTX 5090, 328.2 ms on DGX Spark and 378.7 ms on Jetson AGX Thor including the observation VAE (Sec. 3.2), without removing future prediction and with accuracy preserved (RoboTwin 2.0: 94.4% → 93.5% with NVFP4, Table 5).

We believe these findings and system results are new and useful to the community: they indicate where further investment pays off (AR video priors and longer-episode data) and show that a long-context predictive WAM can run in real time on edge devices.

W2 — Performance is sensitive to context length; can one checkpoint serve different lengths?

Thank you for raising this, which is very helpful in making our manuscript more solid. Training one model per window was our controlled design, so that history length is the only variable behind each point of Fig. 5; it is not a requirement of the method. One checkpoint does serve different lengths, and we tested it (Table R1): the 19.2 s GR-1 checkpoint, with frames older than W replaced at test time by the training padding rule (same tokens, less visible history), changes by −2.5 points at W = 9.6 s and +0.8 points at 4.8 s (within noise), by −5.9 points at 2.4 s, and collapses to 2.1% at 0 s (24 tasks × 10 episodes per row, paired against the unmodified model in the same batch). So one long-window model covers the histories different tasks need; only the token budget, hence latency, is fixed by the training window (74.6 ms at 0 s vs. 341.0 ms at 19.2 s, App. I). The reverse does not hold: a short-window model cannot be given more history than it was trained on. On LIBERO-Long, longer windows do not reduce accuracy; they only add latency.

In summary, we share your wish for the bidirectional robot-pretrained baseline and state its absence as a limitation in the revision. We hope that the controlled pair in Fig. 5, which already separates the objective from the backbone and the robot recipe, together with the new experiments, addresses your concerns, and we would be grateful if you would take them into account in your final assessment.

Table R1. Test-time history truncation of the 19.2 s GR-1 model.

Visible real history WSuccess vs. unmodified 19.2 s model (same batch)
9.6 s−2.5 points
4.8 s+0.8 points
2.4 s−5.9 points
0 s (all history slots show the current frame)collapses to 2.1%

AuthorsOpenReview ↗

Response to Reviewer mKUz

W3 / Question — A Wan2.2-based WAM trained bidirectionally on the same robot-video corpus.

Yes, we agree that this model would strengthen the claim, and we would train it if we could: continued pretraining of a bidirectional 5B video model on the robot corpus costs about 30,720 H100 GPU-hours (about 20 days), which we cannot afford for a single ablation. We respectfully believe, however, that the paper already allows a conclusion on where the advantage comes from, because it contains a controlled pair:

  • LongLive-2.0 (AR) is Wan2.2-TI2V-5B fine-tuned into a causal model. Neither LongLive-2.0 nor Wan2.2 has seen robot video. Both go through the identical Long-WAM adaptation (Sec. 4.3). This pair isolates the causal objective from the backbone and from the robot continued-training recipe.
  • Under this pair, the AR prior uses history and the bidirectional prior does not: 65.7% vs. 61.7% at 2.4 s and 76.7% vs. 61.6% at 19.2 s on GR-1 (Fig. 5). The bidirectional curve peaks at 64.1% (9.6 s) and falls to 61.6% (19.2 s) while both AR curves keep rising. A domain difference shifts a whole curve; it cannot make a gap grow with the window.
  • Robot-domain pretraining is applied afterwards, to the prior already shown to use history: LongLive2.0-Robot raises 19.2 s from 76.7% to 78.7%.

What the pair does not control is the AR stage of LongLive-2.0 itself (its data and compute).

Reviewer_mKUzOpenReview ↗

Concerns are partially addressed

The rebuttal substantially clarifies checkpoint reuse and better articulates the paper’s empirical and systems contributions. However, the matched robot-pretrained bidirectional baseline remains absent, and the existing comparison does not isolate the causal objective from additional training data and compute. While the authors clarify the work’s value, the methodological contribution remains limited: the response primarily emphasizes the integration of existing components and empirical findings rather than a clearly identified new mechanism. I therefore consider my concerns partially addressed and maintain my original score.

Reviewer_kJXsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The work introduces Long-WAM, a world-action training framework, with autoregressive training on robot-video, causal video-to-action adaptation. They further carry out inference optimization for real-time robot control and showcase it on various hardware.

Claims:

  1. Longer visual histories help achieve better robot control.
  2. Streaming VAE encoding for asynchronous action executio
Critical strengths
  1. Effective Utilization of Visual History The paper successfully demonstrates how visual history directly benefits world-action models, rather than serving as passive features that a policy might struggle to leverage effectively.

  2. Promising Context-Scaling Results The comparison between autoregressive (AR) and bidirectional video initializations provides strong evidence for context-scaling gains. Specifically, on RoboCasa GR-1, the AR initialization achieves a 78.7% success rate with 19.2 seconds of context, whereas the bidirectional initialization shows no net performance gain.

  3. Impressive Inference Engineering The deployment and optimization pipeline for the NVIDIA RTX 5090 is commendable, effectively reducing latency without incurring a significant drop in task success rates.

  4. Extensive Real-World and Simulation Benchmarks The paper presents comprehensive empirical evaluations across both simulation environments and real-world robot platforms (specifically Unitree G1 and YAM).

  5. Well-Motivated Ablation Studies The ablations comparing Inverse Dynamics Models (IDM), co-denoising, and variants without future-video prediction—clearly quantify the specific contributions of predictive conditioning, particularly on long-horizon tasks.

Critical weaknesses
  1. Limited Long-Horizon Scaling Evidence The current context-scaling experiments are conducted in relatively small settings. While LIBERO-Long saturates quickly, a more detailed scaling curve is more prominent on RoboCasa GR-1. Because handling long-horizon tasks forms the central hypothesis of the paper, the work would benefit significantly from additional empirical evidence demonstrating performance in these extended regimes.

  2. Confounding Factors in Long-Context Evaluation (Padding & Coverage) Long-context evaluations are heavily confounded by padding and training coverage. For example, at 38.4 seconds, 80% of the sampled history frames consist of padding. This contamination with padding (or training data coverage limitations) leads to weaker and less conclusive results.

  3. Evaluation on Extended Training Data While the work provides evidence of how long-context helps across two benchmarks, it lacks direct evidence of training on genuinely longer-context data and testing Long-WAM on it. Exploring techniques such as scripted concatenation of robot data or scripted data generation with multiple subgoals would provide a clearer demonstration of extended-context effects.

  4. Controlled Comparison for Causal Temporal Factorization (Minor) Video initializations differ in domain adaptation and potentially other training factors, making it difficult to definitively attribute context-scaling gains specifically to causal temporal factorization. A more controlled autoregressive (AR) versus bidirectional comparison would strengthen this central claim (though this is acknowledged as a minor point given how difficult and expensive such an evaluation is to execute).

Questions for authors
  1. The contributions of the individual components could be better separated. The work proposed robot-data pretraining, future-latent conditioning, and other enginnering effort. The novelty is still unclear wrt to existing WAMs. A controlled ablation with the current WAM might be helpful to the reader.
  2. How much does the action-generation depend on the quality of video-generation? CoD vs IDM is explored in depth. However, controlled intervention or corruption of the video in IDM would be helpful to understand how the failures from the world model affect the robot control.
5 comments
AuthorsOpenReview ↗

Response to Reviewer kJXs

Thank you for the careful and positive review, and for recognising the context-scaling evidence, the inference engineering and the ablations. We answer each point below.

W1 — Limited long-horizon scaling evidence.

Our claim is not "longer history is always better". It is that the useful history follows the temporal structure of the task, and that whether history helps at all depends on the video prior: LIBERO-Long saturates at 2.4 s, while GR-1 improves monotonically to 19.2 s (Fig. 5).

What limits the range. The range of the scaling curve is set by the available training data, not by the method. GR-1 has 24,000 demonstrations averaging 12.1 s, so with a 19.2 s window 95% of training positions already see the full history of their episode, and doubling the window to 38.4 s adds only about 0.3 s of real history on average (see W2). Beyond this point there is no real history left to train on. We therefore do not claim that 19.2 s is an upper limit of the approach or that the benefit saturates there: it is where the coverage of existing data ends. To our knowledge, no current benchmark offers both large-scale training data and episodes long enough to train and evaluate substantially longer windows. Building such data is an open problem that the robotics community will need to solve together; our results give a reason to do so, since they show that a policy with an AR video prior converts additional real history into success.

Evidence already in the paper. Long-horizon evidence beyond GR-1 includes real-robot YAM tasks of more than 40 s on average at 81.7% (Sec. 4.4), and the RoboCasa365 composite tasks of 450–4,350 control steps, about 22–217 s (App. F).

Revision: Sec. 4.3 states the scope of the claim and that the range of the curve is bounded by data coverage rather than by the method; the Discussion names long-episode data at scale as an open need for the field.

W2 — Padding and coverage confound the long-context results. We agree about the 38.4 s point (80% padding): the paper states its drop only as a hypothesis, and every conclusion rests on 0–19.2 s. For the 0–19.2 s range, three pieces of evidence show that the results are not confounded.

  1. Within a window, all three video initialisations receive identical inputs and identical padding, yet the gap between the AR and the bidirectional prior grows from 3.3 points at 0 s to 17.1 points at 19.2 s (Fig. 5). Padding cannot produce a gap between models that see the same padding.
  2. Coverage: the GR-1 training set has 24,000 demonstrations averaging 12.1 s. With a 19.2 s window, 95% of training positions already see the full history of their episode; 38.4 s adds about 0.3 s of real history on average while doubling the tokens, which is consistent with the small drop at 38.4 s.
  3. New experiment, test-time history truncation. The 19.2 s GR-1 model is kept fixed; at test time every frame older than W is replaced by the oldest kept frame, which is exactly the training padding rule. Token count and padding pattern are identical in all rows; only the real history changes. All rows, including the unmodified model, were run in one batch under the formal protocol (24 tasks × 10 episodes per row); we report paired differences because 240-episode evaluations of one checkpoint vary by several points between runs.

Table R1. Test-time history truncation of the 19.2 s GR-1 model.

Visible real history WSuccess vs. unmodified 19.2 s model (same batch)
9.6 s−2.5 points
4.8 s+0.8 points
2.4 s−5.9 points
0 s (all history slots show the current frame)collapses to 2.1%

Padding costs nothing (4.8–19.2 s are within noise), removing history beyond 2.4 s costs about six points, and removing all history makes the policy fail. The gain therefore comes from observed history, and on GR-1 about 5 s of it captures the gain. The 0 s row is out-of-distribution for a model trained with history: it shows that the policy uses temporal context to track progress, and should not be read as the level of a model trained without history (63.3%, Table 4).

W3 — Training and testing on genuinely longer data. We agree that this would be the most direct test. It is unfortunately not feasible within the rebuttal period: concatenating GR-1 episodes does not produce a valid long task, because the end state of one episode does not match the start state of the next, and the simulator has no chained tasks to evaluate on. Revision: scripted multi-sub-goal data generation is added as future work in the Discussion.

AuthorsOpenReview ↗

Response to Reviewer kJXs

W4 — Controlled AR vs. bidirectional comparison (minor). Thank you for acknowledging the cost. The paper already has a pair that isolates the objective: LongLive-2.0 is Wan2.2-TI2V-5B fine-tuned into a causal model, neither has seen robot video, both go through the same adaptation. Result on GR-1: 65.7% vs. 61.7% at 2.4 s, 76.7% vs. 61.6% at 19.2 s (Fig. 5). A domain difference shifts a curve; it does not make a gap grow with the window.

Q1 — Separate the contributions; novelty vs. existing WAMs. Fig. 5 can be read as a one-factor-at-a-time roadmap on GR-1 (same adaptation recipe and protocol); we will add it to the paper as a table.

Table R2. From a standard WAM to Long-WAM on GR-1, one component at a time.

StepGR-1 success
Bidirectional Wan2.2 prior, 2.4 s history (a standard WAM)61.7%
+ AR prior (LongLive-2.0)65.7%
+ robot-domain AR pretraining (LongLive2.0-Robot)66.3%
+ history 2.4 s → 19.2 s (Long-WAM)78.7%

The same history extension for the bidirectional prior gives 61.6%. The starting point is the Wan2.2-5B base used by Fast-WAM and Motus, so the table is exactly the controlled ablation you ask for: an existing WAM plus our components, added one at a time. Future prediction: Table 1 (IDM vs. CoD) plus the intervention study in Q2. The "w/o V" row of Table 1 is an early single-frame model and will be relabelled. Deployment optimisations do not change accuracy (RoboTwin 2.0: 94.4% → 93.5% with NVFP4, Table 5).

Novelty relative to existing WAMs. Table R2 separates the accuracy gains; here we state what is new. Our contribution is deliberately not another WAM architecture. WAM architectures have largely converged on a video diffusion backbone coupled with an action expert (DreamZero, LingBot-VA, Fast-WAM, Motus), and we believe the open question for the field is no longer the architecture but how to make a WAM actually use its context. Our contributions are on that question:

  1. AR pretraining is what makes context scale. To our knowledge, Long-WAM is the first WAM that starts from an AR video foundation model and stays causal throughout: continued AR pretraining on about 10,000 hours of action-free robot and egocentric video, followed by causal video-to-action adaptation. Existing WAMs (DreamZero, LingBot-VA, AHA-WAM) instead convert a bidirectional video model into a causal one. It is also, to our knowledge, the first controlled observation of context scaling in WAMs: with the same architecture, recipe and protocol, extending history to 19.2 s lifts the AR prior to 78.7% and leaves the bidirectional prior at 61.6%.
  2. AR context models are naturally suited to asynchronous execution.* Because actions are conditioned on temporally coherent predictions from the AR foundation, plain asynchronous execution, with no action blending, prefix guidance or extra module, costs only 0.2 points on RoboTwin 2.0 (94.4% → 94.2%) and gives lower handoff RMSE and jerk than Fast-WAM and LingBot-VA (Sec. 3.1). The causal VAE also lets us encode observations as a stream while the robot is still executing, so the latest observation is used without waiting at the handoff.
  3. Real-time long-context inference on edge hardware. Shared optimisations (W4A4 NVFP4 quantisation, KV reuse, CUDA Graph replay, shared Q/K/V input quantisation, segmented attention) plus device-specific kernel tuning give roughly 3–4× speedups over the BF16 eager baseline, down to 107.4 ms on RTX 5090, 328.2 ms on DGX Spark and 378.7 ms on Jetson AGX Thor including the observation VAE (Sec. 3.2), without removing future prediction and with accuracy preserved (RoboTwin 2.0: 94.4% → 93.5% with NVFP4, Table 5).

We believe these findings and system results are new and useful to the community: they indicate where further investment pays off (AR video priors and longer-episode data) and show that a long-context predictive WAM can run in real time on edge devices. Revision: Table R2 is added to Sec. 4.3, and the contribution statement is restructured along these three points.

AuthorsOpenReview ↗

Response to Reviewer kJXs

Q2 — How much does action generation depend on video-prediction quality? Intervention / corruption study. We ran this study on both benchmarks, with the trained models fixed and only the imagined future manipulated.

Table R3. Interventions on the imagined future, GR-1 (19.2 s model, 24 tasks × 10 episodes per row; paired against the unmodified model in the same batch; 95% CI from a bootstrap over tasks).

Intervention on the imagined futureChange in success
Replace imagined latents by pure noise, keep their σ⋆ = 0.9 timestep−20.0 points (CI −30 to −10; 20/24 tasks worse)
Freeze the first forecast of the episode for all 36 s−23.3 points (CI −32 to −15; 20/24 tasks worse)
σ⋆ = 1.0 at inference (no partial denoising; noise with a consistent timestep)+1.7 points (CI −5 to +9)

LIBERO-Long (2.4 s model, 10 tasks × 20 episodes, paper protocol): noise, frozen or substituted futures (other episode, other task) all stay at 98.5–99.5% (unmodified model in the same batch: 99.0%), and sweeping σ⋆ from 1.0 to 0.5 stays within 96.5–99.5%. LIBERO-Long is saturated, so it cannot reveal the dependence that GR-1 shows.

Control depends on the forecast that the video expert forms in the future slots, and corrupting it propagates to control (−20 and −23 points on GR-1, largest on tasks that need the next sub-goal: opening a drawer, cabinet or microwave). It does not depend on the denoising depth: σ⋆ = 1.0 costs nothing. This is why a 4-step rollout without decoding suffices and why asynchronous execution, stale by a fraction of a second rather than 36 s, costs little (Tables 5–7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.