- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Sparse-WAM, a training-free method for accelerating World Action Model (WAM) inference. The method uses action-to-future attention to identify important visual tokens in predicted future frames, keeping frame-specific core tokens and shared spatial anchors while pruning less important tokens. It also introduces Pilot to reduce inference overhead through token selection reuse and caching across denoising steps.
Critical strengths
- The paper is easy to follow, with clear and well-designed figures and tables.
- The proposed method makes sense. Using action attention to identify important regions in future visual predictions is a reasonable and effective approach.
- The experiments are solid, covering multiple WAM architectures, simulation benchmarks, and real-world robotic tasks.
Critical weaknesses
-
The novelty of the proposed method is quite limited. Previous works such as ToCa, SpecPrune-VLA, and WorldCache have already explored similar ideas in token pruning and caching. I don't see a fundamental difference between Sparse-WAM and these methods. I also question whether this is an attractive solution for practical WAM deployment. There are already many ways to improve inference efficiency, from kernel optimization and hardware-specific acceleration to real-time chunking (RTC), which can hide inference latency during robot execution. These approaches may achieve useful latency improvements with little performance loss. In comparison, Sparse-WAM trades task performance for speed, but the actual speedup is only 1.56× under matched execution backends (Table 6). I think the authors need to provide stronger reasons for practitioners to adopt Sparse-WAM over these alternatives.
-
The robustness results seem to contradict the motivation behind action-guided pruning. The main idea is that action attention can identify important visual regions, so the remaining tokens can be safely pruned. However, on LIBERO-Plus (Table 8), the largest performance drops occur under camera perturbations (−20.0%) and sensor noise (−13.75%), which are precisely the cases where visual information matters most. This makes me question whether action attention can reliably identify important visual tokens. I would like to see an explanation of these failures and whether they come from token selection or caching.
-
I am not sure whether Stable Anchors are really necessary. Figure 5(a) shows that Core + Stable Anchors only improves success by 1.0 percentage point over Core Only (35.5% vs. 34.5%). Considering the additional design for cross-frame attention aggregation and stable anchor selection, this improvement seems rather small. Wouldn't Core Only be a simpler and more practical solution?
Questions for authors
Please see Weaknesses.
5 comments
Robustness degradation and Benefits of stable anchors.
W2: Robustness degradation: token selection versus prediction caching.
Robustness under perturbations. The observed degradation is consistent with two interacting factors: limited future-token coverage and reuse of predictions between full refreshes. Action-to-future attention serves as a relevance proxy, but an aggressive token budget may omit useful future context under camera shifts or sensor noise. Reusing visual velocity predictions at omitted positions may also allow approximation errors to persist across denoising steps.
We added experiments that increase token retention with full refresh fixed at step 0, and increase refresh frequency at fixed 42.9% sparsity. Greater retention broadens future-token coverage and reduces reliance on cached predictions, while full refreshes update both token selections and cached predictions. The results below show that adjusting token retention and refresh frequency can improve average success rate under camera perturbations and sensor noise, while retaining meaningful speedups.
| Configuration | Sparsity | Full refresh steps | Camera SR (%) | Sensor Noise SR (%) | Avg. SR (%) | Latency (ms) | Speedup vs dense eager | Speedup vs. optimized dense |
|---|---|---|---|---|---|---|---|---|
| Dense eager | 0% | 0–9 | 51.25 | 61.25 | 70.89 | 428.90 | 1.00× | — |
| Optimized dense | 0% | 0–9 | — | — | — | 345.23 | 1.24× | 1.00× |
| Sparse-WAM (original) | 68.4% | 0 | 31.25 | 47.50 | 63.04 | 207.20 | 2.07× | 1.67× |
| Sparse-WAM | 53.1% | 0 | 45.00 | 61.25 | 68.93 | 220.47 | 1.95× | 1.57× |
| Sparse-WAM | 42.9% | 0 | 46.25 | 62.50 | 70.00 | 220.97 | 1.94× | 1.56× |
| Sparse-WAM | 30.6% | 0 | 48.75 | 68.75 | 72.14 | 233.02 | 1.84× | 1.48× |
| Sparse-WAM | 42.9% | 0, 5 | 47.50 | 62.50 | 70.54 | 248.80 | 1.72× | 1.39× |
| Sparse-WAM | 42.9% | 0, 3, 6 | 51.25 | 66.25 | 71.96 | 277.04 | 1.55× | 1.25× |
Sec. 5.4 (Table 6) reports the token-retention and refresh ablations across all seven LIBERO-Plus perturbation categories. Appendix C.1 gives the optimized dense timing reference.
W3: Benefits and overhead of stable anchors.
Value of Stable Anchors. Stable Anchors provide a lightweight way to improve task success with little additional runtime overhead. They reuse the attention statistics collected by Pilot for cross-frame aggregation and selection. All sparse variants use the same token budget and Pilot backend. In Figure 5(a), Core + Stable improves success from 34.5% to 35.5%, while speedups remain approximately 1.81×–1.82×. Our additional ablation on FastWAM-Joint shows the same pattern: Core + Stable achieves the highest success among the sparse variants at nearly identical speedups. All sparse variants below prune 42.9% of future tokens and use the same full Pilot configuration. Speedups are measured relative to dense eager inference in this ablation.
| Configuration | Spatial | Object | Goal | Long | Avg | Speedup |
|---|---|---|---|---|---|---|
| Dense eager | 99.00% | 100.00% | 98.20% | 97.80% | 98.75% | 1.000× |
| Core-only | 98.00% | 99.60% | 96.80% | 96.00% | 97.60% | 1.977× |
| Stable-only | 99.60% | 99.20% | 96.80% | 96.80% | 98.10% | 1.983× |
| Random | 98.80% | 98.80% | 95.20% | 92.40% | 96.30% | 1.978× |
| Core + Stable | 98.80% | 99.20% | 98.60% | 97.20% | 98.45% | 1.985× |
The revised manuscript extends the component ablation to FastWAM-Joint in Appendix C.2 (Table 12), with a pointer in Sec. 5.4.
Follow-up on Real-Robot Deployment and Stable Anchors
Dear Reviewer Rmef,
Thank you again for raising questions about what distinguishes Sparse-WAM from prior pruning and caching methods and how it fits into real-robot deployment. We would appreciate your feedback on how Sparse-WAM complements real-time chunking, the additional LIBERO-Plus results under camera perturbations and sensor noise, and the Stable Anchors ablation on FastWAM-Joint.
As the discussion period draws to a close, please let us know if any questions remain. We would be grateful if you could consider these clarifications and results in your reassessment.
Kind regards,
Authors
Novelty and practical value relative to existing acceleration methods.
Thank you for the thoughtful feedback and we address the concerns below with clarifications and additional experiments.
W1: Novelty and practical value relative to existing acceleration methods.
Sparse-WAM differs from ToCa, SpecPrune-VLA, and WorldCache in both the problem it solves and how it selects tokens.
Action prediction from evolving future tokens. ToCa and WorldCache reduce repeated diffusion computation while primarily preserving generation or rollout quality. VLA pruning typically targets the computationally expensive VLM backbone, which processes current observations and instructions. These inputs are known and remain fixed during action prediction, allowing their VLM representations to provide task-conditioned cues for selecting relevant visual tokens. In joint-denoising WAMs, however, the bottleneck lies in repeatedly processing a large number of future-frame tokens. These tokens are initialized as noise and evolve throughout denoising, so the future scenes are not yet available as fixed observations from which to determine action-relevant regions. The challenge is therefore to identify which regions of the evolving imagination matter for action prediction. Sparse-WAM addresses this challenge by using action-to-future attention as an online proxy for action relevance, guiding future-token selection in both MoT and unified WAM architectures without retraining.
Complementary to kernel optimization and real-time chunking (RTC). Kernel optimization and hardware-specific acceleration often require implementation and tuning tailored to particular architectures and devices. Although RTC overlaps inference with robot execution, newly generated actions still depend on observations acquired before inference completes. Reducing inference latency therefore remains important for timely responses to new observations. Sparse-WAM complements RTC by making updated action chunks available sooner: in our real-robot experiments, control frequency increases from 25 Hz with RTC to 50 Hz with RTC + Sparse-WAM. Practitioners can combine these methods to accelerate WAMs with large future-imagination workloads.
Reasons for Sparse-WAM. First, Sparse-WAM offers competitive acceleration, achieving approximately 1.5–1.6× speedups over optimized dense inference while largely preserving task success. On Cosmos3-Edge, success remains comparable to dense inference: 23.00% versus 22.90% (Table 2).
Second, compared with dynamic pruning of other competing methods like SpecPrune-VLA, Sparse-WAM retains a fixed token budget and reuses log-sum-exp (LSE) normalizers for attention scoring. These choices keep execution shapes stable and reduce scoring and scheduling overhead, facilitating compilation and CUDA Graph replay. Under matched backend settings on Cosmos3-Edge, Sparse-WAM achieves a 1.56× speedup over optimized dense inference (Table 5). ToCa, SpecPrune-VLA, and WorldCache achieved 1.06–1.29× under matched settings in Table 6 of the original submission.
Third, Sparse-WAM applies to both MoT and unified WAM architectures, enabling practitioners to accelerate existing models without the cost of model redesign or retraining.