- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Long-WAM studies how real observation history affects world-action control. It continues autoregressive video pretraining on robot and egocentric videos, then adapts the foundation through asymmetric video-action coupling. Actions condition on observed history and partially denoised future visual latents. Each context window uses a separately trained policy, with forecast and action horizons fixed within each benchmark. The study finds substantial history gains for both AR initializations, with different useful windows on LIBERO-Long and RoboCasa GR-1. Streaming VAE encoding, asynchronous execution, and hardware-specific inference optimizations support deployment while retaining future prediction. Evaluation includes simulation benchmarks, dynamic manipulation on Unitree G1, and sustained tabletop manipulation on YAM.
Critical strengths
-
Useful findings on observation history. Longer history substantially improves the AR-based policies, with different useful windows across tasks and much smaller gains for the bidirectional initialization (Figure 5). This helps clarify when history benefits world-action control.
-
Strong deployment and real-robot results. Streaming VAE encoding, asynchronous execution, and hardware-specific optimizations substantially reduce inference cost while retaining future prediction and similar task success (Tables 5-7). Dynamic manipulation and sustained execution on G1 and YAM support the system's practical value (Figures 6-7).
Critical weaknesses
The future-denoising cutoff needs a clearer rationale. Section 2.2 fixes for both the action-training surrogate and inference, so actions use visual KV from a partially denoised future at this noise level. Tables 1-2 support the complete predictive path, and Appendix C.2 explains the conditioning interface. However, the paper does not explain how 0.9 was selected or why the representation at this endpoint retains the predictive information needed for control. This leaves a central choice in the predict-then-act design difficult to assess.
Questions for authors
-
How was selected, and what makes the future representation at this endpoint useful for action generation? The selection rationale or existing evidence about predictive information retained in the visual KV would help justify this choice.
-
Appendix F reasonably explains that stronger atomic execution helps Long-WAM benefit from high-level planning. Could you elaborate on which skills or execution failure modes most clearly distinguish Long-WAM + Astra from + Astra? Existing task-level results or representative rollout observations, where available, would help clarify this interaction.
-
Figure 5 suggests that useful history windows depend on the task. Could you explain the choice of history sampling interval and discuss whether multi-scale history (dense recent and sparse older observations) or selective sampling could preserve useful temporal coverage with fewer inputs? The selection rationale or existing task-level observations would help guide future work on context efficiency.
Response to Reviewer annU
We sincerely thank you for the supportive review and for questions that go to the core of the design.
W / Q1 — How was σ⋆ = 0.9 selected, and why does the representation at this endpoint carry what control needs?
Selection: σ⋆ = 0.9 is the endpoint of a 4-step video denoising schedule fixed a priori (1 → 0.994 → 0.983 → 0.963 → 0.900 under the shifted schedule); it was not tuned and we do not claim it is optimal. Reading the future at high noise is shared with related designs: DreamZero-Flash conditions actions on a noise-dominated visual context (σ ≈ 0.875), and LingBot-VA trains its action decoder on noisy video latents and decodes actions when the video is denoised only halfway (s = 0.5).
Why it carries predictive information: the action expert does not read the noisy future latents; it reads the per-layer keys and values that the video expert computes on [clean history | future at σ⋆]. To predict the denoising direction, the video expert must estimate the clean future internally, and that estimate is what the cache carries. Under causal attention the future slots aggregate this forecast from the history at every noise level; the high-noise stage fixes coarse layout and motion, which is what control needs.
Evidence, new: (a) Probes. On 1,058 LIBERO-Long windows, ridge probes from the visual cache the action expert reads predict the end-effector displacement 1.55 s ahead with held-out R² = 0.94 from the future tokens at σ⋆ = 0.9, vs. 0.87 from the current-frame cache alone (0.8 s ahead: 0.91 vs. 0.85). R² is 0.94–0.95 for future tokens at σ ∈ {1.0, 0.9, 0.7}; in the early layers the partially denoised content adds information (0.7 > 0.9 > 1.0). Decoding the one-step x̂₀ of the σ⋆ = 0.9 future gives frames close to the ground truth 0.8 s and 1.6 s ahead. (b) Endpoint sweep. LIBERO-Long (10 tasks × 20 episodes): σ⋆ = 1.0 / 0.95 / 0.9 / 0.8 / 0.7 / 0.5 gives 99.5 / 99.5 / 99.0 / 96.5 / 98.5 / 97.5%; denoising to 0.7 or 0.5 while keeping the KV timestep at 0.9 gives 97.0 / 98.5%. GR-1 (19.2 s model, 24 tasks × 10 episodes, paired against the unmodified model in the same batch): σ⋆ = 1.0 changes success by +1.7 points (95% CI −5 to +9), while replacing the future slots by noise inconsistent with their timestep costs 20 points and freezing the forecast for the whole episode costs 23 points.
Q2 — Which skills or failure modes distinguish Long-WAM + Astra from π0.5 + Astra?
Atomic execution is the difference. Alone on Atomic-Seen, Long-WAM succeeds in 67.9% of episodes, π0.5 in 39.6%; composite tasks multiply along the chain (three steps: about 0.31 vs. 0.06), so stronger atomic execution is what lets the planner's decomposition pay off. The gain is decomposition, not decision frequency: without the planner and re-querying every 15 steps, Long-WAM reaches 84.4% on Atomic-Seen but 11.2% / 5.0% on the composite settings, vs. 38.8% / 35.0% with Astra. On unseen instructions the planner grounds abstract requests into mastered atomic steps (e.g. "place the tool for peeling cucumbers" → "take the peeler from the drawer and put it on the cutting board"). Failure modes: of 101 failed composite episodes, 67 are the planner repeatedly rewriting the sub-instruction; 4 tasks fail in every setting (e.g. pressing the button in PrepareCoffee), the current skill ceiling. Caveat: for π0.5 + Astra only aggregates are available and we cannot confirm the planning interface matches ours, so this is a system-level comparison; our composite results use 5 episodes per task. Revision: failure analysis and protocol in App. F.
Q3 — Choice of the history sampling interval; multi-scale or selective history?
Control is 20 Hz; history is sampled every 4th step (5 Hz); the causal VAE compresses time 4×, so one latent covers 0.8 s and the 19.2 s window is 25 clean latents plus 2 imagined future latents. Uniform sampling was chosen because (i) a constant step between latents lets the dynamics learned in video pretraining transfer directly, (ii) the VAE compresses consecutive frames, so non-uniform sampling changes what one latent represents, and (iii) tokens and latency grow linearly and predictably with the window (App. I). Your suggestion matches what we observe: LIBERO-Long saturates at 2.4 s, and the 19.2 s GR-1 window holds on average only 7.1 s of real history, so dense recent plus sparse older context is the natural next step. Revision: multi-scale and selective sampling as future work in the Discussion.
Thank you again for the support.