acceptodds
Withdrawn from ICLR 2027

SHARP-WAM: Sparse Hierarchical Attention with Probe Tokens for Bounded-Context World Action Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World Action Models (WAMs) present a compelling framework for robotic manipulation via video prediction. However, scaling WAMs to long-horizon tasks is hindered by the dense attention complexity of foundation backbones. Existing approaches either compromise non-Markovian dependencies via truncated sliding windows, or introduce external memory plug-ins that risk complicating the pre-trained backbone. Leveraging the insight that long-horizon action modeling relies on sparse historical interactions, we propose SHARP-WAM, a parse ierarchical ttention framework with Leanable robe Tokens for bounded-context World Action Models. This framework introduces a compact set of learnable probe tokens serving as contextualized descriptors and routers. Historical key-value states are retained within a native, fixed-capacity KV Set via online utility-driven eviction, thereby enabling horizon-invariant attention inference cost. Building upon this bounded backbone-native memory, we devise a hierarchical sparse attention scheme. Temporal Top- routing is first performed via probe tokens to select relevant candidates from the Fixed KV Set, followed by camera-aware spatial partitioning. This dual-level sparsity substantially curbs long-horizon computational overhead while preventing cross-view feature corruption and preserving fine-grained semantics. Extensive experiments demonstrate that SHARP-WAM outperforms existing methods on long-horizon tasks, achieving success rates of on RMBench, on LIBERO-Mem, and in real-world trials, while remaining competitive on general-purpose benchmarks. [Project page](https://anonymous.4open.science/w/SHARP-WAM_page/)

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic Manipulation, World Action Models, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Fv12OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SHARP-WAM, a bounded-context World Action Model for long-horizon robotic manipulation. It introduces learnable probe tokens to summarize historical observations and guide sparse retrieval. A fixed-capacity KV cache retains selected raw historical representations through an eviction strategy combining probe similarity and EMA-based retrieval utility. Hierarchical sparse attention then performs temporal Top-K frame selection followed by camera-aware spatial partitioning. Training combines joint video/action flow matching with sampled dense-attention routing supervision and a two-stage transition to bounded memory, while explicit video generation is omitted during inference. Experiments report 76.0% success on RMBench, 47.0% on LIBERO-Mem, and 75.0% in real-world trials, alongside bounded inference memory and per-decision latency.

Critical strengths
  1. A practical bounded-memory architecture.

Unlike sparse attention methods that retain an ever-growing historical cache, SHARP-WAM bounds both historical storage and retrieval costs. Retaining selected raw KV states preserves fine-grained visual information that compressed summaries may lose. The complexity analysis appropriately distinguishes constant per-decision cost from total rollout cost.

  1. Promising empirical results and useful ablations.

SHARP-WAM achieves strong results on memory-dependent benchmarks while maintaining competitive general-task accuracy. FIFO eviction reduces success from 80%/82% to 67%/74% on two tasks, while removing raw KV states causes larger drops. The efficiency analysis demonstrates bounded memory usage (14.5 GiB) compared with approximately 22 GiB for unbounded variants.

Critical weaknesses
  1. The architectural novelty relative to prior work needs clearer positioning.

MemoryWAM (Yang et al., 2026) already employs learnable gist tokens, anchor/recent memory, and a dual-DiT architecture. R-KV (Cai et al., 2025) also combines importance and redundancy for KV-cache compression, although in a different setting. SHARP-WAM's distinctive contribution lies in bounded retention of raw historical KV states with hierarchical sparse retrieval. While the raw-KV ablation supports this design, a matched-memory comparison with compressed representations would better establish its advantages.

  1. Insufficient isolation of individual components.

The "w/o Probe Tokens" ablation does not clearly specify the replacement mechanism for routing and eviction. Removing spatial partitioning matches or improves accuracy (80%/84% versus 80%/82%), but the corresponding latency and memory costs are not reported. Moreover, the FIFO comparison does not separately establish the contributions of similarity-based redundancy detection and EMA utility.

  1. Limited analysis of memory-capacity trade-offs.

Although the memory-dependent benchmarks demonstrate useful long-term retention, the paper does not systematically vary the fixed KV capacity or evaluate controlled cue-to-action delays. Figure 5 examines scalability on one task with the default 36-frame capacity. These experiments would clarify when bounded retention begins to discard task-critical historical information.

Questions for authors

Can you clarify the differences from gist-based memory approaches such as MemoryWAM? Given the raw-KV ablation in Table 4, how would SHARP-WAM compare with compressed memory representations under a matched memory budget?

What replaces probes in the "w/o Probe Tokens" ablation? Can you report the runtime impact of spatial partitioning and separately ablate similarity-based and utility-based eviction?

How do accuracy, latency, and memory change with different KV-cache capacities and increasing cue-to-action delays?

Reviewer_xkWpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the cost of conditioning World Action Models (WAMs) on long observation histories for memory-dependent manipulation. It argues that dense full-history attention scales poorly with horizon length. Sliding-window caches discard non-Markovian dependencies, and external memory banks or compressed latents either add plug-in modules or lose fine-grained evidence. The proposed SHARP-WAM is built on Fast-WAM and appends learnable probe tokens to each frame, which serve as compact descriptors and routers. Historical KV states are kept in a layer-wise, fixed-capacity KV set maintained by EMA-utility and redundancy-based eviction. Attention is hierarchically sparse: probe-based temporal Top-K frame routing is followed by camera-aware spatial block selection. An auxiliary KL loss distills routing from a dense-attention teacher, and training proceeds in two stages. The paper reports the best average success on RMBench, LIBERO-Mem, and a single real-world humanoid task. It also reports competitive results on LIBERO and bounded per-decision latency and memory compared with full-history attention.

Critical strengths

1. Targets a real bottleneck with demonstrated bounded cost. Keeping per-decision cost bounded for long-horizon WAMs is a practical problem. Fig. 5 shows flat latency and memory compared with full-history attention. The appendix also states the complexity scope honestly: total rollout cost remains O(T)O(T), and replacement requires O(CKV2)O(C_{KV}^2) comparisons.

2. Clear gains on a subset of memory-heavy tasks. The most convincing result is on RMBench M(n) tasks, with 79.3 vs. 70.5 over LingBot-VA on 400 rollouts. There are also large per-task gains on LIBERO-Mem (Bt×3: 80 vs. 10, B×7: 30 vs. 0). The raw-KV ablation (Table 4) supports the point that retaining fine-grained history matters for these tasks.

Critical weaknesses

1. The added complexity is not justified by the gains, and the paper's own ablations undercut several components.

SHARP-WAM stacks many interacting mechanisms on top of Fast-WAM:

  • per-frame probe tokens with a custom causal mask
  • a layer-wise Fixed KV Set
  • EMA utility with a separate "manager" scoring pass and a probation period
  • redundancy-constrained eviction
  • per-layer temporal Top-K1K_1 routing, recomputed at every action denoising step
  • per-camera spatial Top-K2K_2 selection with a bimanual symmetry constraint
  • an auxiliary KL routing loss with a stochastically sampled dense teacher
  • a two-stage training schedule

This design also brings many hyperparameters (NpN_p, CKVC_{KV}, Manc\mathcal{M}_{anc}, etc.). Complexity of this degree requires clear gains, and the results do not provide them.

  • RMBench: The overall improvement over LingBot-VA is 2.0 pp (76.0 vs. 74.0). SHARP-WAM is worse on the M(1) average (73.4 vs. 76.8), including Swap T (80 vs. 94) and Observe & Pick Up (6 vs. 14, and 28 for HiMem-WAM).
  • LIBERO-Mem: The overall gain over MemoryVAM is 4.5 pp, but SHARP-WAM is lower on 5 of the 10 tasks (e.g., Bt→P 30 vs. 65, B×3 70 vs. 90, Cs+E 55 vs. 70).
  • LIBERO: SHARP-WAM (97.3) is below its own initialization, Fast-WAM (97.6). On LIBERO-Long, it is also lower than Fast-WAM (94.0 vs. 95.2).
  • Own ablations: Removing bounded capacity (82/82) or spatial partitioning (80/84) matches or exceeds the full model (80/82). In Fig. 5(a), the variant without the Fixed KV Set appears to have latency comparable to the full model. The latency benefit over Full History therefore seems to come largely from sparse attention alone. The Fixed KV Set and its eviction machinery mainly reduce peak memory (22 → 14.5 GiB at around 800 frames) without any accuracy gain. No ablation isolates the camera-aware aspect of partitioning, so the claim that it prevents cross-view feature corruption is untested.
  • Motivation and novelty: The introduction criticizes memory banks as plug-ins that complicate the pretrained backbone. SHARP-WAM, however, adds tokens, modifies attention masks, introduces per-layer caches and routing losses, and appears to fine-tune the full backbone, which is arguably more invasive. Individually, the components closely follow known techniques: anchor plus recent-window retention resembles attention sinks (StreamingLLM), utility-based eviction resembles heavy-hitter eviction (H2O), block routing follows MoBA/NSA, and probes follow gist/summary tokens. The LLM KV-cache eviction literature is not discussed. The contribution is mainly the combination of these techniques for WAMs, which raises the bar for empirical evidence.

This is consequential because the paper's case rests on this specific design being necessary. The gains are marginal and mixed, and the ablations show that several components are not needed for accuracy. A much simpler alternative, such as Fast-WAM with full-history or longer-window KV caching plus plain sparse attention, may suffice.

2. The evaluation cannot statistically support the headline claims.

  • No seeds, standard deviations, or confidence intervals are reported for any result.

  • Several evaluations are small. LIBERO-Mem uses 20 rollouts per task (5 pp granularity). The real-world evaluation is a single task with 20 trials. Ablations use two RMBench tasks, and the efficiency analysis uses one.

  • As a rough check, I ran a pooled two-proportion z-test that ignores task structure:

    • RMBench overall vs. LingBot-VA (684 vs. 666 successes out of 900): p≈0.33p \approx 0.33
    • LIBERO-Mem vs. MemoryVAM (94 vs. 85 out of 200, assuming the same protocol): p≈0.37p \approx 0.37
    • Real world (15/20 vs. 10/20): p≈0.10p \approx 0.10

    None of these overall improvements is significant at the 5% level. The RMBench M(n) subset is the exception. Many ablation differences are also 2–4 points on 100 trials (e.g., K1=8K_1=8 vs. 1616: 77/81 vs. 80/82), which is within plausible noise.

  • The baselines are inconsistent across benchmarks. The strongest baseline on each benchmark is not evaluated on the others: LingBot-VA is absent from LIBERO-Mem, and MemoryVAM is absent from RMBench. Fast-WAM's M(1) results on RMBench are reproduced, while its M(n) results are copied from WLA-0, which mixes protocols. Most importantly, there is no same-backbone control on all benchmarks (e.g., Fast-WAM with full-history or matched-window caching), and this is the comparison that would isolate the contribution.

This is consequential because the claimed state-of-the-art results rest on differences that cannot currently be distinguished from evaluation noise.

Questions for authors

Q1. Which parameters are trained, and is the backbone preserved? Which parameters are updated in Stage 1 and Stage 2?

Fig. 2(a) and the video-dynamics loss in Eq. (10) appear to indicate that the video DiT and action DiT are fully fine-tuned together with the probes. If so, two follow-ups:

  • (a) In what sense is the memory "native" and less intrusive than the memory-bank plug-ins criticized in the introduction?
  • (b) Is the pretrained backbone's capability preserved after fine-tuning? Please report video-prediction quality and general manipulation performance relative to Fast-WAM trained on the same data. The LIBERO results (97.3 vs. 97.6 overall, 94.0 vs. 95.2 on LIBERO-Long) may hint at some degradation.

If the backbone is instead frozen, please state this explicitly and specify which modules are trained.

Q2. Does eviction actually fire in the reported benchmarks? What are the episode lengths, in decision steps or frames, for RMBench, LIBERO-Mem, and the real-world task? The relevant reference point is the total retained context of ∣Manc∣+∣Rt∣+CKV=42|\mathcal{M}_{anc}| + |\mathcal{R}_t| + C_{KV} = 42 frames. If most episodes are shorter than this, the Fixed KV Set and eviction policy are never exercised in the main results. That would be consistent with "w/o Bounded Capacity" matching the full model in Table 4. Please report how often eviction is triggered per benchmark.

Q3. Statistical reliability. Please report the number of seeds and add std/CI to all tables. Please also increase the rollouts for LIBERO-Mem and add more real-world tasks or trials. Are the baseline numbers in Tables 2 and 3 reproduced by the authors or copied from prior work? If copied, under what protocol were they obtained?

Q4. Same-backbone controls. Please evaluate Fast-WAM with (i) a full-history KV cache, (ii) a sliding window of matched size (42 frames), and (iii) mean-pooled MoBA/NSA-style routing in place of probes, all on the three benchmarks. What router does the "w/o Probe Tokens" variant use?

Q5. Component necessity and cost. "w/o Bounded Capacity" and "w/o Spatial Partitioning" match or exceed the full model. Which components are needed for accuracy, and which only for efficiency? Please ablate camera-awareness specifically, for example by allowing blocks to cross view boundaries, to support the cross-view corruption claim. Please also report the training cost of the dense-teacher supervision and the two-stage schedule relative to Fast-WAM.

S1. Evaluating on horizons well beyond the cache capacity, where bounded memory is actually necessary, would better showcase the method. A plot of success vs. horizon length against full-history and FIFO baselines would make this clear. A discussion of, and comparison with, LLM KV-cache eviction methods such as H2O and StreamingLLM would also clarify what is new.

Reviewer_B1akOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on resolving the complexity of dense attention of models on long-horizon tasks. Inspired by the previous sparse historical interaction approaches, the authors propose SHAPR-WAM, which utilizes a learnable tokens for contextualized descriptors and router. Experiments on RMBench and LIBERO-Mem shows that proposed method exceeds other methods' performance.

Critical strengths
  • The proposed combination of bounded memory and sparse access over native backbone KV states is effective
Critical weaknesses
  • Some important relevant papers are missing [1,2,3,4,5,6,7]
  • Some papers share similar ideas and details
    • Wonder [1] also retrieves relevant chunks using context-aware sparse attention without compressing or removing historical KV
    • The authors should clarify difference and quantitatively compare.
  • There's no explicit analyses on the importance of sparse routing
  • The scores in the main table (table1, 2) seem to have low reliability since they did not perform sufficient demonstrations to get reliable results (since these results for each tasks are integers, not decimals) and they also don't have std or error ranges.

[1] Xu et al., Wonder: Video World Model Done Better [2] Yi et al., WorldKV: Efficient World Memory with World Retrieval and Compression [3] Wu et al., Addressable Memory for Video World Models [4] Chen et al., ReWorld: An Interactive World Model with Long-Horizon Memory [5] Samuel et al., Fast autoregressive video diffusion and world models with temporal cache compression and sparse attention [6] Sun et al., TempoFit: Plug-and-Play Layer-Wise Temporal KV Memory for Long-Horizon Vision-Language-Action Manipulation [7] Ye et al., D²-VLA: Dual-Memory Dual-Frequency Vision-Language-Action Model For Long Dynamic Manipulation

Questions for authors
  • In Table 1, some scores are reproduced and some are not. Is there a reason that do not reproduce all?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.