acceptodds
Under review as a conference paper at ICLR 2027

Sparse-WAM: Accelerating World Action Models via Action-Guided Sparse Imagination

30 views

Abstract

World-action models (WAMs) leverage pretrained video models to improve generalization in robot control by jointly predicting future visual states and actions. This capability comes at a substantial inference cost, as dense future-frame tokens are repeatedly processed during denoising. Prior methods address this by token pruning that prioritizes visual fidelity to reduce denoising costs in video diffusion models. However, these methods do not use action relevance to determine which future-frame tokens to retain during joint denoising in WAMs. In this paper, we propose **Sparse-WAM**, a training-free framework for *action-guided sparse imagination* that selectively processes future-frame tokens to accelerate WAM inference. We observe substantial overlap in the spatial distribution of attention from action tokens to future-frame tokens (action-to-future attention) between consecutive denoising steps, despite continued updates to the future representations. Motivated by this, we develop *Action-Guided Token Selection* to retain frame-specific action-relevant regions together with cross-frame context. However, a naive implementation can incur attention-scoring and token-packing overhead that offsets the computational savings from pruning. We therefore introduce **Pilot**, an efficient engine that reduces sparse inference overhead through lightweight scoring and cross-step reuse of token selections. On LIBERO with FastWAM-Joint and RoboLab-120 with Cosmos 3 Edge, **Sparse-WAM** achieves inference speedups of approximately and , respectively, over dense eager inference on an NVIDIA RTX 4090, while largely preserving task performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action model, efficient inference, token sparsification, action-aware attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RmefOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sparse-WAM, a training-free method for accelerating World Action Model (WAM) inference. The method uses action-to-future attention to identify important visual tokens in predicted future frames, keeping frame-specific core tokens and shared spatial anchors while pruning less important tokens. It also introduces Pilot to reduce inference overhead through token selection reuse and caching across denoising steps.

Critical strengths
  • The paper is easy to follow, with clear and well-designed figures and tables.
  • The proposed method makes sense. Using action attention to identify important regions in future visual predictions is a reasonable and effective approach.
  • The experiments are solid, covering multiple WAM architectures, simulation benchmarks, and real-world robotic tasks.
Critical weaknesses
  • The novelty of the proposed method is quite limited. Previous works such as ToCa, SpecPrune-VLA, and WorldCache have already explored similar ideas in token pruning and caching. I don't see a fundamental difference between Sparse-WAM and these methods. I also question whether this is an attractive solution for practical WAM deployment. There are already many ways to improve inference efficiency, from kernel optimization and hardware-specific acceleration to real-time chunking (RTC), which can hide inference latency during robot execution. These approaches may achieve useful latency improvements with little performance loss. In comparison, Sparse-WAM trades task performance for speed, but the actual speedup is only 1.56× under matched execution backends (Table 6). I think the authors need to provide stronger reasons for practitioners to adopt Sparse-WAM over these alternatives.

  • The robustness results seem to contradict the motivation behind action-guided pruning. The main idea is that action attention can identify important visual regions, so the remaining tokens can be safely pruned. However, on LIBERO-Plus (Table 8), the largest performance drops occur under camera perturbations (−20.0%) and sensor noise (−13.75%), which are precisely the cases where visual information matters most. This makes me question whether action attention can reliably identify important visual tokens. I would like to see an explanation of these failures and whether they come from token selection or caching.

  • I am not sure whether Stable Anchors are really necessary. Figure 5(a) shows that Core + Stable Anchors only improves success by 1.0 percentage point over Core Only (35.5% vs. 34.5%). Considering the additional design for cross-frame attention aggregation and stable anchor selection, this improvement seems rather small. Wouldn't Core Only be a simpler and more practical solution?

Questions for authors

Please see Weaknesses.

5 comments
AuthorsOpenReview ↗

Novelty and practical value relative to existing acceleration methods.

Thank you for the thoughtful feedback and we address the concerns below with clarifications and additional experiments.

W1: Novelty and practical value relative to existing acceleration methods.

Sparse-WAM differs from ToCa, SpecPrune-VLA, and WorldCache in both the problem it solves and how it selects tokens.

Action prediction from evolving future tokens. ToCa and WorldCache reduce repeated diffusion computation while primarily preserving generation or rollout quality. VLA pruning typically targets the computationally expensive VLM backbone, which processes current observations and instructions. These inputs are known and remain fixed during action prediction, allowing their VLM representations to provide task-conditioned cues for selecting relevant visual tokens. In joint-denoising WAMs, however, the bottleneck lies in repeatedly processing a large number of future-frame tokens. These tokens are initialized as noise and evolve throughout denoising, so the future scenes are not yet available as fixed observations from which to determine action-relevant regions. The challenge is therefore to identify which regions of the evolving imagination matter for action prediction. Sparse-WAM addresses this challenge by using action-to-future attention as an online proxy for action relevance, guiding future-token selection in both MoT and unified WAM architectures without retraining.

Complementary to kernel optimization and real-time chunking (RTC). Kernel optimization and hardware-specific acceleration often require implementation and tuning tailored to particular architectures and devices. Although RTC overlaps inference with robot execution, newly generated actions still depend on observations acquired before inference completes. Reducing inference latency therefore remains important for timely responses to new observations. Sparse-WAM complements RTC by making updated action chunks available sooner: in our real-robot experiments, control frequency increases from 25 Hz with RTC to 50 Hz with RTC + Sparse-WAM. Practitioners can combine these methods to accelerate WAMs with large future-imagination workloads.

Reasons for Sparse-WAM. First, Sparse-WAM offers competitive acceleration, achieving approximately 1.5–1.6× speedups over optimized dense inference while largely preserving task success. On Cosmos3-Edge, success remains comparable to dense inference: 23.00% versus 22.90% (Table 2).

Second, compared with dynamic pruning of other competing methods like SpecPrune-VLA, Sparse-WAM retains a fixed token budget and reuses log-sum-exp (LSE) normalizers for attention scoring. These choices keep execution shapes stable and reduce scoring and scheduling overhead, facilitating compilation and CUDA Graph replay. Under matched backend settings on Cosmos3-Edge, Sparse-WAM achieves a 1.56× speedup over optimized dense inference (Table 5). ToCa, SpecPrune-VLA, and WorldCache achieved 1.06–1.29× under matched settings in Table 6 of the original submission.

Third, Sparse-WAM applies to both MoT and unified WAM architectures, enabling practitioners to accelerate existing models without the cost of model redesign or retraining.

AuthorsOpenReview ↗

Robustness degradation and Benefits of stable anchors.

W2: Robustness degradation: token selection versus prediction caching.

Robustness under perturbations. The observed degradation is consistent with two interacting factors: limited future-token coverage and reuse of predictions between full refreshes. Action-to-future attention serves as a relevance proxy, but an aggressive token budget may omit useful future context under camera shifts or sensor noise. Reusing visual velocity predictions at omitted positions may also allow approximation errors to persist across denoising steps.

We added experiments that increase token retention with full refresh fixed at step 0, and increase refresh frequency at fixed 42.9% sparsity. Greater retention broadens future-token coverage and reduces reliance on cached predictions, while full refreshes update both token selections and cached predictions. The results below show that adjusting token retention and refresh frequency can improve average success rate under camera perturbations and sensor noise, while retaining meaningful speedups.

ConfigurationSparsityFull refresh stepsCamera SR (%)Sensor Noise SR (%)Avg. SR (%)Latency (ms)Speedup vs dense eagerSpeedup vs. optimized dense
Dense eager0%0–951.2561.2570.89428.901.00×—
Optimized dense0%0–9———345.231.24×1.00×
Sparse-WAM (original)68.4%031.2547.5063.04207.202.07×1.67×
Sparse-WAM53.1%045.0061.2568.93220.471.95×1.57×
Sparse-WAM42.9%046.2562.5070.00220.971.94×1.56×
Sparse-WAM30.6%048.7568.7572.14233.021.84×1.48×
Sparse-WAM42.9%0, 547.5062.5070.54248.801.72×1.39×
Sparse-WAM42.9%0, 3, 651.2566.2571.96277.041.55×1.25×

Sec. 5.4 (Table 6) reports the token-retention and refresh ablations across all seven LIBERO-Plus perturbation categories. Appendix C.1 gives the optimized dense timing reference.

W3: Benefits and overhead of stable anchors.

Value of Stable Anchors. Stable Anchors provide a lightweight way to improve task success with little additional runtime overhead. They reuse the attention statistics collected by Pilot for cross-frame aggregation and selection. All sparse variants use the same token budget and Pilot backend. In Figure 5(a), Core + Stable improves success from 34.5% to 35.5%, while speedups remain approximately 1.81×–1.82×. Our additional ablation on FastWAM-Joint shows the same pattern: Core + Stable achieves the highest success among the sparse variants at nearly identical speedups. All sparse variants below prune 42.9% of future tokens and use the same full Pilot configuration. Speedups are measured relative to dense eager inference in this ablation.

ConfigurationSpatialObjectGoalLongAvgSpeedup
Dense eager99.00%100.00%98.20%97.80%98.75%1.000×
Core-only98.00%99.60%96.80%96.00%97.60%1.977×
Stable-only99.60%99.20%96.80%96.80%98.10%1.983×
Random98.80%98.80%95.20%92.40%96.30%1.978×
Core + Stable98.80%99.20%98.60%97.20%98.45%1.985×

The revised manuscript extends the component ablation to FastWAM-Joint in Appendix C.2 (Table 12), with a pointer in Sec. 5.4.

AuthorsOpenReview ↗

Follow-up on Real-Robot Deployment and Stable Anchors

Dear Reviewer Rmef,

Thank you again for raising questions about what distinguishes Sparse-WAM from prior pruning and caching methods and how it fits into real-robot deployment. We would appreciate your feedback on how Sparse-WAM complements real-time chunking, the additional LIBERO-Plus results under camera perturbations and sensor noise, and the Stable Anchors ablation on FastWAM-Joint.

As the discussion period draws to a close, please let us know if any questions remain. We would be grateful if you could consider these clarifications and results in your reassessment.

Kind regards,
Authors

Reviewer_7kaeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sparse-WAM accelerates world-action models without retraining by selecting future-video tokens using action-to-future attention. It combines frame-specific tokens and shared spatial anchors with an efficient engine that reuses selections and cached predictions across denoising steps. Evaluation across three models, simulation benchmarks, real-world tasks, and component ablations demonstrates substantial inference speedups with generally small task-performance losses.

Critical strengths

Practical, training-free acceleration. Action-guided selection and efficient sparse execution reduce inference cost without retraining. The matched-backend comparison shows a 1.56× speedup on Cosmos 3 Edge, supporting gains beyond compilation optimizations.

Broad evaluation with informative ablations. Tests across three WAMs, simulation benchmarks, and real-world tasks support applicability across architectures. Selection and refresh ablations substantiate the design choices and characterize the speed–performance trade-off.

Critical weaknesses

Robustness degrades materially under perturbations. On LIBERO-Plus, average success falls from 70.89% to 63.04%, including a 20-point drop under camera changes. This limits the claim that sparsification preserves control performance in conditions relevant to deployment.

Headline speedups mix algorithmic and backend gains. Main results compare optimized sparse inference against dense eager execution. Although a matched-backend comparison is provided for Cosmos 3 Edge, equivalent comparisons are missing for the other backbones, leaving the general acceleration benefit insufficiently isolated.

Questions for authors

Clarification: What speedups remain on FastWAM-Joint and Cosmos 3 Nano when dense and sparse inference use identical compilation and CUDA Graph settings?

Clarification: What causes the substantial LIBERO-Plus degradation, particularly under camera changes? Can increased token retention or more frequent refresh recover dense-model robustness while retaining a meaningful speedup?

3 comments
AuthorsOpenReview ↗

Speedups under matched compilation and CUDA Graph settings.

We thank the reviewer for recognizing the practical value of our training-free acceleration and the breadth of our evaluation. We appreciate the constructive suggestions on OOD robustness and matched-backend comparisons, which we address below with additional experiments.

W2 & Q1: Speedups under matched compilation and CUDA Graph settings.

With identical compilation and CUDA Graph settings for dense and sparse inference, Sparse-WAM achieves 1.58× and 1.59× speedups on FastWAM-Joint and Cosmos3-Nano-Policy. These results demonstrate additional acceleration beyond generic backend optimizations. In eager mode, torch.compile and CUDA Graphs are disabled. Parenthesized speedups are relative to dense inference under the corresponding backend settings.

Execution configurationFastWAM-JointCosmos3-Nano-PolicyCosmos3-Edge
Dense eager (ms)4343450859
Sparse-WAM eager (ms)287 (1.51×)2144 (1.61×)556 (1.55×)
Optimized dense (ms)3463026727
Sparse-WAM (full Pilot) (ms)219 (1.58×)1906 (1.59×)465 (1.56×)
Overall speedup vs. dense eager1.98×1.81×1.85×

Appendix B.4 (Table 9) reports the full timing breakdown across all three backbones.

AuthorsOpenReview ↗

Recovering LIBERO-Plus robustness through token retention and refresh.

W1 & Q2: Recovering LIBERO-Plus robustness through token retention and refresh.

Possible cause of the degradation. The original setting prunes 68.4% of future tokens and refreshes only at step 0. Under camera changes, a small future-token budget may miss useful context, while infrequent refreshes may allow errors in cached predictions to persist. We added experiments on LIBERO-Plus to test whether retaining more tokens or refreshing more often can recover dense-model robustness while preserving a meaningful speedup.

New experiments for LIBERO-Plus. Reducing sparsity to 30.6% raises average success from 63.04% to 72.14%, compared with 70.89% for dense inference. Camera success improves from 31.25% to 48.75%, close to the dense result of 51.25%, while retaining a 1.48× speedup over optimized dense inference. At fixed 42.9% sparsity, refreshing at steps 0, 3, and 6 raises camera success from 46.25% to 51.25%, matching dense inference. Average success reaches 71.96%, with a 1.25× speedup over optimized dense inference.

ConfigurationSparsity RateFull refresh stepsAvg. SR (%)Latency (ms)Speedup vs dense eagerSpeedup vs. optimized dense
Dense eager0%0–970.89428.901.00×—
Optimized dense0%0–9—345.231.24×1.00×
Sparse-WAM (original)68.4%063.04207.202.07×1.67×
Sparse-WAM53.1%068.93220.471.95×1.57×
Sparse-WAM42.9%070.00220.971.94×1.56×
Sparse-WAM30.6%072.14233.021.84×1.48×
Sparse-WAM42.9%0, 570.54248.801.72×1.39×
Sparse-WAM42.9%0, 3, 671.96277.041.55×1.25×

Results for all seven perturbation categories are reported below.

ConfigurationSparsityFull refresh stepsBackgroundCameraLanguageLightLayoutRobotNoiseAvg. SR (%)
Dense baseline0%0–966.2551.2587.5095.0083.7551.2561.2570.89
Sparse-WAM (original)68.4%057.5031.2585.0092.5081.2546.2547.5063.04
Sparse-WAM53.1%060.0045.0090.0093.7580.0052.5061.2568.93
Sparse-WAM42.9%062.5046.2590.0092.5083.7552.5062.5070.00
Sparse-WAM30.6%063.7548.7590.0088.7585.0060.0068.7572.14
Sparse-WAM42.9%0, 565.0047.5090.0091.2582.5055.0062.5070.54
Sparse-WAM42.9%0, 3, 661.2551.2591.2593.7582.5057.5066.2571.96

Sec. 5.4 (Table 6) reports the token-retention and refresh ablations across all seven LIBERO-Plus perturbation categories. Appendix C.1 gives the optimized dense timing reference.

AuthorsOpenReview ↗

Follow-up on LIBERO-Plus Robustness and Matched-Backend Results

Dear Reviewer 7kae,

Thank you again for your constructive feedback on robustness and backend fairness. We would appreciate your thoughts on the new comparisons using identical compilation and CUDA Graph settings, and on the LIBERO-Plus experiments testing whether retaining more tokens or refreshing more frequently can recover robustness while preserving a speedup.

As the discussion period draws to a close, please let us know if any points would benefit from further clarification. We would be grateful if you could consider these results in your reassessment.

Kind regards,

Authors

Reviewer_xURdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Sparse-WAM accelerates inference in joint-denoising World Action Models without retraining by pruning redundant future-frame tokens during denoising. The authors observe that action tokens mainly attend to two types of regions in each predicted future frame and these attended regions barely change across denoising steps. Sparse-WAM therefore performs a dense denoising step to select a fixed number of frame-specific core tokens and shared spatial anchor tokens per frame, reusing those token indices across subsequent sparse steps. A companion inference engine, Pilot, extracts attention scores online from FlashAttention log-sum-exp statistics without materializing full attention matrices and packs kept tokens into static shapes, yielding speedup on FastWAM-Joint and Cosmos 3 Policy models with very small in-distribution accuracy loss.

Critical strengths
  1. A novel attempt to migrate pruning methods targeting static history observations in VLAs to WAMs.
  2. Cross-architecture effectiveness: The combined core-plus-shared selection rule works out-of-the-box on both unified-sequence backbones and Mixture-of-Transformers backbones (FastWAM-Joint on LIBERO and real-world tasks), staying within 1% of dense task success while outperforming adapted diffusion-caching and VLA-pruning baselines.
Critical weaknesses
  1. Necessity of video denoising: Recent works such as Fast-WAM, Faster-WAM, Light-WAM, and GigaWorld-Policy have shown that omitting iterative video denoising during inference yields larger speedups while maintaining strong control performance, whereas Sparse-WAM remains slower and still sees noticeable drops on perturbation-heavy tasks. For example, action-only Fast-WAM runs in 90.1 ms (vs. 207.2 ms for Sparse-WAM) with 97.6% LIBERO success, and Light-WAM runs in 72.0 ms with 97.2% LIBERO success. Even on out-of-distribution LIBERO-Plus where future conditioning is most helpful, one-pass Faster-WAM reaches 73.57% average success without iterative video denoising, while Sparse-WAM drops to 63.04% average success. This leaves open whether pruning tokens inside multi-step video denoising offers a better trade-off than fully bypassing iterative video denoising at test time.
  2. Missing comparisons: First, in Tables 1 and 2, the dense baselines and competing methods do not use the Pilot engine, yet in Table 2 on Cosmos3-Nano-Policy, SpecPrune-VLA without Pilot is still faster than Sparse-WAM (1.88x vs. 1.81x); meanwhile, when different selection variants run on the same Pilot engine in Figure 5a, they all achieve nearly identical speedups (1.81x–1.82x), and the backend ablation in Table 6 is only studied on Cosmos 3 Edge, making it unclear how much of the runtime improvement comes from the Pilot engine versus Sparse-WAM itself. Second, the paper would be more solid if adding comparisons with several related inference-time optimization methods, including Flash-WAM, AHA-WAM, ActionCache etc.
Questions for authors

Same as critical weaknesses. Justify the performance gap in 1 and add experiments on 2 would be appreciated.

3 comments
AuthorsOpenReview ↗

Necessity of Video denoising and the speed–robustness trade-off

W1: Video denoising and the speed–robustness trade-off.

Sparse-WAM provides training-free acceleration for existing joint-denoising checkpoints, preserving their joint video–action inference formulation while reducing future-token computation.

Future conditioning matters under distribution shift. Fast-WAM [5], Light-WAM, and GigaWorld-Policy reduce inference cost by omitting future conditioning, whereas joint-denoising WAMs and Faster-WAM retain future representations for action prediction. A recent study [1] reports strong visual robustness among evaluated WAMs despite substantial denoising costs. Cosmos Policy [2], DreamZero [3], and LingBot-VA [4] also demonstrate strong control performance using future video prediction. Although omitting future conditioning reduces computation, it can come at a sharp cost to robustness: Faster-WAM’s controlled ablation reports substantially lower LIBERO-Plus success for its current-only variant, particularly under camera perturbations and sensor noise [6]. These findings motivate reducing the computational cost of WAMs while preserving the future information used for action prediction.

Sparse denoising across architectures without retraining. Faster-WAM [6] achieves robust, efficient inference through one-pass future conditioning in a specifically designed and trained MoT model. Sparse-WAM retains joint video–action denoising while reducing future-token computation through action-guided selection and cross-step reuse. It supports both MoT and unified architectures without redesign or retraining. In the LIBERO-Plus robustness sweep, Sparse-WAM achieves 70.00% mean success at 42.9% pruning, close to dense FastWAM-Joint’s 70.89%, with 1.94× and 1.56× speedups over dense eager and optimized dense inference, respectively (Table 6 and Appendix C.1). The results below show that suitable token budgets recover overall robustness comparable to dense inference while retaining meaningful speedups.

ConfigurationSparsityFull refresh stepsCamera SR (%)Sensor Noise SR (%)Avg. SR (%)Latency (ms)Speedup vs dense eagerSpeedup vs. optimized dense
Dense eager0%0–951.2561.2570.89428.901.00×—
Optimized dense0%0–9———345.231.24×1.00×
Sparse-WAM (original)68.4%031.2547.5063.04207.202.07×1.67×
Sparse-WAM53.1%045.0061.2568.93220.471.95×1.57×
Sparse-WAM42.9%046.2562.5070.00220.971.94×1.56×
Sparse-WAM30.6%048.7568.7572.14233.021.84×1.48×
Sparse-WAM42.9%0, 547.5062.5070.54248.801.72×1.39×
Sparse-WAM42.9%0, 3, 651.2566.2571.96277.041.55×1.25×

The revised manuscript compares action-only FastWAM, dense FastWAM-Joint, and Sparse-WAM on LIBERO-Plus in Sec. 5.2 (Table 3) and reports the robustness ablations in Sec. 5.4 (Table 6).

References

[1] Zhanguang Zhang et al. Do World Action Models Generalize Better than VLAs? A Robustness Study. arXiv preprint 2603.22078, 2026.

[2] Moo Jin Kim et al. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. ICLR, 2026.

[3] Seonghyeon Ye et al. World Action Models are Zero-shot Policies. CoRL, 2026 (Oral).

[4] Lin Li et al. Causal World Modeling for Robot Control. RSS, 2026.

[5] Tianyuan Yuan et al. Fast-WAM: Do World Action Models Need Test-time Future Imagination? NeurIPS, 2026 (Spotlight).

[6] Weiheng Zhao et al. Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Models. arXiv preprint 2608.04404, 2026.

AuthorsOpenReview ↗

Backend contributions and additional baseline comparisons.

W2: Backend contributions and additional baseline comparisons.

Generic backend optimizations and Pilot. We clarify the relationship between Pilot and the backend settings. In the main comparisons (Table 1 and 2), all compared acceleration methods use the same generic backend optimizations, including torch.compile and CUDA Graphs. The full Pilot configuration combines these optimizations with LSE-based attention scoring and cross-step caching for Sparse-WAM. On Cosmos3-Nano-Policy, SpecPrune-VLA is faster than Sparse-WAM (1.88× vs. 1.81×), but its average success rate is 12.92 percentage points below dense inference.

Comparisons across execution settings. We report both eager and matched-backend comparisons (Appendix B.4). Sparse-WAM already accelerates inference under eager execution. With compilation and CUDA Graphs enabled for both dense and sparse inference, it achieves speedups of 1.58×, 1.59×, and 1.56× on FastWAM-Joint, Cosmos3-Nano-Policy, and Cosmos3-Edge, respectively. In the eager configurations, torch.compile and CUDA Graphs are disabled. Parenthesized speedups compare each Sparse-WAM configuration against dense inference with the same compilation and CUDA Graph settings.

Execution configurationFastWAM-JointCosmos3-Nano-PolicyCosmos3-Edge
Dense eager (ms)4343450859
Sparse-WAM eager (ms)287 (1.51×)2144 (1.61×)556 (1.55×)
Optimized dense (ms)3463026727
Sparse-WAM (full Pilot) (ms)219 (1.58×)1906 (1.59×)465 (1.56×)
Overall speedup vs. dense eager1.98×1.81×1.85×

Sec. 5.1 clarifies the backend settings. Sec. 5.4 (Table 5) summarizes the speedups, and Appendix B.4 (Table 9) reports the full timing breakdown.

Clarification and extension of ablation studies. All sparse selection variants in Figure 5(a) use the same token budget and Pilot backend, allowing us to compare selection quality at a fixed sparsity ratio. Their nearly identical speedups indicate that combining core and stable selection introduces little additional computation and scheduling overhead with Pilot.

We extend the matched-backend evaluation from Table 6 of the original submission to FastWAM-Joint and report task success and speedup across pruning ratios below.

SparsitySpatial(%)Object(%)Goal(%)Long(%)Avg. SR(%)Speedup vs. optimized dense
0%99.00100.0098.2097.8098.751.00×
71.4%96.4098.0098.4094.6096.851.67×
51.0%98.00100.0097.6096.4098.001.57×
42.9%98.8099.2098.6097.2098.451.58×
30.6%98.80100.0097.0097.8098.401.48×

The revised manuscript reports the FastWAM-Joint pruning-ratio sweep in Appendix C.2 (Table 13), with a pointer in Sec. 5.4.

Additional comparisons. Flash-WAM and AHA-WAM use additional training or distillation, whereas our controlled acceleration comparisons keep the pretrained checkpoint fixed. Following the reviewer’s suggestion, we added ActionCache comparisons across all three backbones, with experimental settings detailed in Appendix B.3. When adapted from the VLM-based VLA models to MoT and unified WAM architectures, ActionCache exhibits low natural cache hit rates, particularly on the Cosmos3 backbones. The results are shown below.

LIBERO

MethodSpatial (%)Object (%)Goal (%)Long (%)Avg. SR (%)SpeedupFLOPs (% of dense)
FastWAM-Joint99.00100.0098.2097.8098.751.00×100.00
+ ActionCache99.2098.4096.8095.6097.501.33×74.93
+ Sparse-WAM98.8099.2098.6097.2098.451.98×49.65

RoboLab-120

MethodSimple (%)Moderate (%)Complex (%)Avg. SR (%)SpeedupFLOPs (% of dense)
Cosmos3-Edge25.6023.3011.8022.901.00×100.00
+ ActionCache24.8423.5910.5922.421.22×97.82
+ Sparse-WAM25.0025.908.8223.001.85×60.73
Cosmos3-Nano-Policy40.6335.3825.2936.751.00×100.00
+ ActionCache40.1632.0521.1834.831.13×98.75
+ Sparse-WAM40.1632.8224.1235.501.81×59.78

We added the ActionCache results to Sec. 5.2 (Tables 1 and 2) and describe its adaptation and latency estimation in Appendices B.3 and B.4, respectively.

We estimate ActionCache speedups over dense eager inference by weighting the profiled hit and forced-miss latencies using the natural hit rates measured on RoboLab-120 and LIBERO with a 10,000-entry cache.

AuthorsOpenReview ↗

Follow-up on Video Denoising and Baseline Comparisons

Dear Reviewer xURd,

Thank you again for your constructive review. We would appreciate your feedback on our clarification of why Sparse-WAM retains iterative video denoising for existing joint-denoising checkpoints, as well as our new matched-backend and ActionCache comparisons.

As the discussion period draws to a close, please let us know if any questions remain. We would be grateful if you could consider these clarifications and results in your reassessment.

Kind regards,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Author Responses for the Area Chair

Dear Area Chair,

Thank you for overseeing the discussion of Sparse-WAM. We appreciate the reviewers’ feedback and the opportunity to address their questions.

Core Contributions and Recognized Strengths

  • Action-guided sparse imagination. Sparse-WAM uses action-to-future attention to select which future video tokens to recompute, reducing denoising work while preserving future conditioning. xURd highlighted the novelty of extending token pruning to predicted future frames in WAMs, and Rmef found action attention effective for identifying relevant regions.

  • Training-free acceleration. Sparse-WAM accelerates pretrained WAMs without retraining, using Pilot to reuse token selections and cached predictions across denoising steps. 7kae recognized the practical value of this approach and cited the 1.56× speedup on Cosmos3-Edge under matched backend settings.

  • Evaluation across architectures and tasks. We evaluate Sparse-WAM on three WAM backbones spanning unified-sequence and mixture-of-transformers architectures, with experiments in simulation and on a real robot. 7kae and Rmef highlighted the breadth of the evaluation.

In our main experiments, Sparse-WAM delivers meaningful speedups under matched backend settings while generally achieving higher task success rates than competing methods.

Key Issues and Responses

  • Necessity of future denoising (xURd, Rmef). Recent studies suggest that conditioning on predicted future video can improve robustness to perturbations. On LIBERO-Plus, Sparse-WAM retains near-dense task success at lower latency by selectively recomputing future tokens.

  • Fair comparison under matched backend settings (xURd, 7kae). Extended timing comparisons and backend ablations show that Sparse-WAM reduces inference latency when dense and sparse inference use the same optimizations. Task success remains comparable to dense inference in the main experiments.

  • Robustness under perturbations (7kae, Rmef). Token-budget and refresh sweeps identify settings that recover dense-level task success while retaining speedups. In the follow-up comparisons under perturbations, Sparse-WAM reduces computation the most among the evaluated pruning and caching methods.

We also added ActionCache comparisons across all three backbones and extended the token-selection and sparsity ablations to FastWAM-Joint.

Discussion Progress

  • xURd. We addressed the questions about future denoising and expanded the inference-time baseline comparisons, but have not yet received follow-up feedback.

  • 7kae. We provided matched-backend timings across all three backbones and added robustness ablations. We are awaiting follow-up feedback.

  • Rmef. After our initial response, Rmef acknowledged the new experiments and stated that they would maintain their original rating, citing concerns about action-guided selection and deployment. We followed up with comparisons against WorldCache and SpecPrune-VLA under perturbations, clarified the additional training required by one-step distillation, and submitted real-robot demos. Supplementary material further demonstrates the real-world complementarity between RTC and Sparse-WAM.

We believe additional experiments show action-guided sparse denoising speeds up pretrained WAMs without retraining, preserving future conditioning and near-dense success under perturbations. Thank you again for your time and effort in handling our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.