acceptodds
Under review as a conference paper at ICLR 2027

SimpleWAM: Training-Free Sink-Aware Compression for Efficient World Action Modeling

15 views

Abstract

World-action models (WAMs) jointly model future visual dynamics and robot actions, but long visual contexts and growing historical key-value (KV) caches can make inference expensive. We ask which parts of a pretrained WAM's visual context are actually necessary for action generation. Profiling action-to-visual attention in Fast-WAM and LingBot-VA reveals recurrent sink tokens with persistent action dependencies, alongside visual evidence whose relevance varies with the current action query. Based on this observation, we introduce SimpleWAM, a training-free sink-aware compression framework with two strategies: KeepSink protects profiled sinks while selecting the remaining action-visible tokens, and SinkAge extends sink preservation to historical KV through time-to-live scheduling and global budget allocation. On Fast-WAM, KeepSink remains competitive under a reduced token budget, whereas removing the profiled sinks drops RoboTwin Suite-10 success from 75% to 3%. SinkAge retains performance close to the Full KV while reducing the active visual-KV context by 55.1–69.4%. Experiments across WAM backbones further show that sink sensitivity varies with backbone and compression budget. In real-world experiments, KeepSink and SinkAge retain performance comparable to or better than Full KV under reduced visual-context budgets. Overall, SimpleWAM reduces active visual-KV cost without retraining while largely retaining task success, including under growing historical context.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Efficiency, Training-Free, World Action Modeling, Sink-Aware Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w8PPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper profiles action-to-visual attention sinks in pretrained WAMs. KeepSink preserves these positions during spatial token selection, while SinkAge combines sink protection, age weighting, and a bounded historical attention budget. The objective is to reduce active visual context without retraining the policy.

Critical strengths
  • The sink-removal experiment is informative. Removing the dominant sink reduces Fast-WAM success from 73% to 3%, while LingBot-VA shows a different sensitivity pattern. This provides useful evidence that compression behavior depends on the backbone.

  • The historical comparisons use a shared physical packing interface and include established cache-selection baselines. This is a reasonable foundation for comparing active attention budgets.

Critical weaknesses
  • The proposed selection rules do not demonstrate a convincing advantage over simpler alternatives. On LIBERO-Long, H2O achieves 98.2% success versus SinkAge’s 96.6% at the same reported KV reduction. Removing historical sink privileges barely changes performance, and random filling outperforms the default SinkAge configuration on Suite-10. These results weaken the justification for the method’s additional design choices.

  • The efficiency and writing claims need tighter alignment with the evidence. Stored caches remain unpruned, so active-KV reduction is not equivalent to actual cache-memory savings. The real-world study also has only ten trials per task. Particularly concerning is the description of 85%, 84%, and 66% success as “comparable”: this is defensive wording that downplays a 19-point drop from Full KV.

Questions for authors
  • Can SinkAge show a consistent advantage over H2O or a simple age-based allocator at matched budgets?

  • What are the actual GPU-memory and policy-inference latency savings, separately from episode wall time?

3 comments
AuthorsOpenReview ↗

Response to Reviewer w8PP: Simple Allocation Baselines, Physical KV Savings, and Experimental Claim Clarification

We thank the reviewer for the constructive feedback. We address the concerns below.

W1 / Q1. SinkAge versus H2O and Simpler Allocation Strategies

  • Clarification of our claim. SimpleWAM investigates structural sink dependencies and historical visual-context redundancy in pretrained WAMs, rather than establishing a universally superior KV-ranking algorithm. We include different retention strategies to examine whether established KV-compression principles also apply to WAM action generation.
  • **Comparison with H2O. **On LIBERO-Long, H2O achieves 98.2% SR versus 96.6% for SinkAge, both with 69.4% active-KV reduction. We acknowledge H2O's stronger performance. This supports our broader finding that different KV-retention strategies can maintain competitive closed-loop performance under substantial visual-context compression.
  • Structural sink importance. Controlled spatial removal on Fast-WAM reduces SR from 74% to 3% at the same budget. This establishes the functional sensitivity to identified structural sinks, independently of historical KV-ranking comparisons.
  • Random Fill and DropSink-age. Random Fill is not unconstrained random pruning: it preserves the offline-profiled sink anchors, recent-sink protection, TTL eligibility, and global KV budget, randomizing only the remaining eligible-token selection. In contrast, DropSink-age removes explicit sink privileges but does not forcibly exclude sink tokens, which can still be naturally selected through dependency–age ranking. Its comparable performance (78.5% vs. 79.0% for SinkAge) therefore does not contradict the controlled spatial sink-removal finding.
  • Additional simple-baseline comparisons. At B=128, SinkAge / StreamingLLM / H2O / Random Fill achieve 78% / 80% / 81% / 83% SR, respectively (Full KV: 83%). We additionally evaluate Pure Age-only, which retains the most recent eligible tokens without explicit sink protection or dependency scoring. At B=384, Pure Age-only achieves 73% versus 79% for SinkAge under the same evaluation protocol. These results do not establish a consistent advantage for dependency–age ranking, but further support historical visual-context compressibility under bounded KV budgets.

W2. Efficiency Evidence and Accuracy of Experimental Claims

  • Logical versus physical compression. We agree that reducing action-visible KV does not necessarily reduce persistent GPU-memory usage. The original Suite-10 evaluation reports 65.4% active visual-KV reduction (2,326→806) while maintaining 79.0% SR, but this should not be interpreted as equivalent physical memory savings.
  • Additional physical pruning. We implement persistent KV eviction and compaction and measure actual GPU memory under matched inference settings. Physical KeepSink / SinkAge reduce resident KV from 7.22 GB to 0.78 / 0.61 GB (89.2% / 91.6% reduction), respectively. Peak allocated GPU memory decreases from 32.73 GB to 26.28 / 26.11 GB (19.7% / 20.2% reduction).
  • Policy-inference latency. Full KV and Physical KeepSink require 3.75 and 3.80 s/chunk, respectively, while Physical SinkAge is at 8.32 s/chunk (K=2). KeepSink incurs only 1.3% latency overhead, whereas SinkAge remains limited by online dependency probing. We leave reducing this overhead for future work.
  • Correction of the “comparable” description. We agree and will revise Sec. 4.2 to explicitly distinguish KeepSink's near-Full performance (84% vs. 85%) from DropSink's substantial 19 pp decrease (66% vs. 85%). This demonstrates significant but backbone-dependent sink-removal sensitivity.
  • Real-world evaluation scope. We acknowledge that the real-robot evaluation includes only 10 trials per method per task. On color sorting, Full KV / KeepSink / SinkAge achieve 2/10, 4/10, and 3/10 successes, respectively; on fruit placement, the corresponding results are 4/10, 6/10, and 4/10. These results provide preliminary evidence of deployment feasibility and establish gains in closed-loop evaluation. We will extend to 20 trials for more convincing results.

In summary, we provide (1) additional matched-budget comparisons with H2O, StreamingLLM, Random Fill, and Pure Age-only shows no substantial performance degradation, suggesting action-irrelevant redundancy as a key source of historical visual-context compressibility. and (2) physical KV pruning experiments, confirming actual GPU-memory savings while identifying online dependency probing as a remaining latency bottleneck. Together, these results strengthen our central findings: structural action-to-visual sinks exhibit backbone-dependent functional importance, and pretrained WAMs can maintain competitive closed-loop performance under substantially reduced visual-KV budgets.

AuthorsOpenReview ↗

Follow-up on KV Compression Baselines and GPU-Memory Efficiency

We sincerely appreciate your constructive feedback, which has been particularly helpful in clarifying the scope of our efficiency claims and the significance of historical visual-context redundancy.

In response, we have added simpler allocation baselines, provided physical KV-pruning results to verify actual GPU-memory savings, and explicitly acknowledged the limitations regarding inference latency and baseline superiority.

As the discussion period approaches its end, please let us know if any concerns remain. If our clarifications and additional evidence have addressed your main concerns, we would be grateful if you would consider reflecting this in your final rating.

Thank you again for your valuable feedback and consideration.

AuthorsOpenReview ↗

Follow-up: Probe-Free KV Compression Results

We conduct an additional experiment to validate the practical efficiency of historical KV compression.

Our probe-free SinkAge variant eliminates expensive online dependency probing while retaining structural sink protection and bounded KV allocation. It achieves 82% SR versus 83% for Full KV on RoboTwin Suite-10 (B=384), with 90.8% lower resident KV and 20.1% lower peak GPU memory. Policy latency decreases from 8.284 to 3.718 s/chunk (−55.1%), reaching Full-KV-comparable latency (3.752 s/chunk).

These results further demonstrate that competitive closed-loop performance and actual GPU-memory savings can be achieved without costly online dependency probing.

Reviewer_5rcqOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an inference-time context optimization technique for World-action models. The authors challenge the necessity of accessing all visual context in the video backbone for action generation. Therefore, they propose to compress such access by using top-K attention in the action-video attention, with sink preservation: KeepSink for observed video frames and SinkAge for future predicted video frames. The experiments on Fast-WAM and LingBot-VA demonstrate that both KeepSink and SinkAge can retain comparable performance while reducing the visual-KV context by 25% and 55.1–69.4%, respectively.

Critical strengths
  1. The SimpleWAM reduces active visual-KV cost without retraining while largely retaining task success, which is meaningful to the community.

  2. The authors conducted sufficient experiments to benchmark the performance of the proposed techniques.

Critical weaknesses
  1. The proposal to reduce video-KV access is meaningful. However, the proposed KeepSink and SinkAge do not show remarkable advantage compared to other baselines.
    • (Table 1.) On Fast-WAM, KeepSink is not better than FastV and Saliency. On LingBot-VA, the performance of KeepSink is even worse than Random.
    • (Table 2.) Sinkage is slightly better than H2O, with a slightly lower percentage of visual-KV reduction on RoboTwin ALL-50 Clean. On RoboTwin ALL-50 Randomized, it is worse than three other baselines. On LIBERO-Long, it is worse than H2O and SnapKV.

Taken together, while KeepSink and SinkAge substantially reduce active visual KV while maintaining performance, it is hard to say they beat other competitors.

  1. There is no evidence provided for efficiency improvement during inference, such as GPU memory or end-to-end inference time. It remains unknown whether the reduced visual-KV access with KeepSink and SinkAge can really reduce GPU memory usage or improve inference latency. This evidence is crucial for inference-time optimization techniques.

  2. It is unclear about the sink-token prefiling. For example,

    • Where do the 100 task-episode profiles come from? From the training set, test set, or arbitrary data?
    • Do the task-episode profiles need to exactly match the task-type in evaluation? Do they generalize across tasks?
    • How many profiles do we generally need?
    • Does it support online inference?
    • What is the cost (in terms of time and compute) of profiling compared to the full evaluation? (I assume both techniques require at least the prefilling of WAMs)
Questions for authors

Please read the weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5rcq: Clarification and Structural Sink Evidence, Practical KV Efficiency, and Profiling Transparency

We thank the reviewer for the constructive feedback. We clarify that SimpleWAM investigates structural sink dependencies and historical visual-context redundancy in pretrained WAMs, rather than proposing a universally superior KV-ranking algorithm. We address the concerns below.

W1. Effectiveness Compared with Existing Baselines

  • Clarification of our objective. We investigate whether the attention-sink phenomena and visual-context redundancy observed in LLMs also emerge in pretrained WAM action-to-visual attention. StreamingLLM, H2O, and SnapKV serve as representative retention strategies to examine how established KV-compression principles transfer to WAMs.
  • Structural sink importance. On Fast-WAM at κ=0.75, KeepSink achieves 74% SR (Full KV: 73%), whereas removing the profiled sink reduces SR to 3%. LingBot-VA achieves 84% / 66% with KeepSink / DropSink (Full KV: 85%). These matched-budget interventions demonstrate backbone-dependent sensitivity to structural sinks.
  • Tighter-budget comparison. At κ=0.50 (60/120 tokens), Fast-WAM achieves 70.3% SR with KeepSink, versus 72.0% FastV, 71.7% Saliency, and 36.5% Random. Our sink audit shows that FastV and Saliency naturally rank token 0 first. On LingBot-VA, Random outperforms KeepSink at κ=0.75 (85% vs. 84%) and κ=0.50 (69% vs. 65%); FastV (80%) and Saliency (74%) also outperform KeepSink (65%) at κ=0.50. This reflects LingBot-VA's more distributed dependencies, where protecting a single sink does not guarantee optimal token retention.
  • Historical compressibility. At H=72, SinkAge reduces active visual KV by 65.4% (2,326→806) while matching Full KV (79.0% SR). At B=128, SinkAge / StreamingLLM / H2O achieve 78% / 80% / 81% SR, while sink-protected Random Fill achieves 83% (Full KV: 83%). These comparisons validate historical visual-context redundancy without establishing SinkAge's ranking superiority.

W2. GPU Memory and Inference Latency

  • Logical versus physical compression. On Suite-10, SinkAge reduces active KV by 65.4% while maintaining 79.0% SR and reducing episode wall time from 204 to 128 s/episode. We agree that active-KV reduction alone does not establish physical memory savings.
  • Physical KV pruning. We implement persistent KV eviction and compaction. Physical KeepSink / SinkAge reduce resident KV from 7.22 GB to 0.78 / 0.61 GB (−89.2% / −91.6%) and peak allocated GPU memory from 32.73 GB to 26.28 / 26.11 GB (−19.7% / −20.2%).
  • Policy-inference latency. Full KV and Physical KeepSink require 3.75 and 3.80 s/chunk, respectively, while Physical SinkAge is at 8.32 s/chunk (K=2). KeepSink incurs only 1.3% latency overhead, whereas online dependency probing remains SinkAge's main latency bottleneck. We will discuss this limitation and leave reducing the overhead to future work.

W3. Sink-Token Profiling and Deployment

  • Profiling and cross-benchmark reuse. We profile 100 RoboTwin 2.0 Suite-10 evaluation episodes (10 tasks × 10 episodes). Despite overlap with Suite-10 evaluation, the frozen sink set transfers directly to LIBERO-Long without re-profiling or tuning.
  • Online inference. SimpleWAM runs closed-loop without retraining, reusing offline-profiled sinks for budgeted visual-KV selection. Online dependency probing incurs additional overhead; real-time acceleration is not guaranteed.
  • Profiling cost. Each sample includes vision encoding, KV prefill, and 20 action-denoising forwards across 30 layers. On NVIDIA RTX PRO 6000, denoising takes 0.44 s and other operations 0.09 s, totaling 0.53 s/sample.

In summary, our additional experiments address three concerns: (1) baseline effectiveness, showing that different KV-retention strategies can preserve competitive performance without establishing SinkAge's ranking superiority; (2) practical efficiency, demonstrating substantial physical GPU-memory savings while identifying online dependency probing as a latency bottleneck; and (3) profiling and deployment, clarifying offline sink identification, cross-benchmark reuse, and profiling cost. Together, these results validate backbone-dependent structural sink importance and substantial historical visual-context compressibility in pretrained WAMs.

Thanks for the detailed rebuttal.

I appreciate the authors' findings on attention sinks and visual-context redundancy in WAMs, and I believe this is a meaningful research direction.

The additional experiments demonstrate substantial KV-memory savings and approximately 20% reduction in peak GPU memory usage, which addresses one of my major concerns.

However, the proposed methods do not demonstrate inference acceleration, which is a bit out of my expectations. KeepSink introduces negligible latency overhead, and SinkAge is substantially slower than Full KV (8.32 vs. 3.75 s/chunk), despite using fewer KV tokens.

Moreover, KeepSink and SinkAge still do not demonstrate convincing advantages over existing methods such as FastV and H2O, respectively.

Overall, I acknowledge the value of the findings and additional experiments, but remain not fully convinced from the perspective of technical contribution. Therefore, I will increase my rating from 1 to 2, but maintain my negative recommendation.

AuthorsOpenReview ↗

Follow-up: Probe-Free KV Compression and Performance

Thank you for your prompt and thoughtful feedback. We sincerely appreciate your recognition of our findings on structural attention sinks and visual-context redundancy in WAMs, as well as the additional experimental evidence of GPU-memory savings.

We acknowledge your concern regarding inference latency. We would like to clarify that the efficiency contribution of SimpleWAM primarily lies in reducing action-visible KV under constrained budgets and achieving substantial GPU-memory savings through physical KV pruning, rather than accelerating end-to-end inference.

Additional Latency Experiments. Taking your concern seriously, we further evaluated a probe-free Physical Sink+Age variant that eliminates additional action-denoising probes while retaining structural sink protection and bounded historical KV allocation. Under matched inference settings, this reduces policy latency from 8.284 to 3.718 s/chunk (−55.1%), achieving Full-KV-comparable latency (3.718 vs. 3.752 s/chunk). The probe-free variant maintains competitive closed-loop performance (82% vs. 83% SR for Full KV) while reducing resident KV by 90.8% and peak GPU memory by 20.1%, with Full-KV-comparable inference latency (3.718 vs. 3.752 s/chunk).

These additional results demonstrate that substantial historical visual-KV compression and actual GPU-memory savings can be achieved with negligible inference-latency overhead, without requiring expensive online dependency probing. While this simpler deployment variant does not establish the superiority of dependency–age ranking, it further supports our central finding of substantial action-irrelevant redundancy in accumulated visual history. Further improving action-aware selection without additional probing remains an important direction for future work.

We sincerely appreciate your constructive feedback and reconsideration of our work.

Reviewer_hQjnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper profiles action-to-visual attention in two world–action models (Fast-WAM, LingBot-VA) and reports a recurrent attention sink at visual token 0. It proposes two training-free compressors. KeepSink protects the sink and fills a reduced token budget by action attention. SinkAge applies TTL and age-weighted allocation to the historical KV cache while protecting sinks. On RoboTwin and LIBERO-Long, the methods stay within about 2 points of Full KV while reducing "active" visual KV by 55–69%.

Critical strengths
  1. A practical, training-free setting. Inference-time context reduction for WAMs, especially a bounded KV history for autoregressive WAMs, is a relevant problem.

  2. A clear sink-removal intervention. Removing token 0 collapses Fast-WAM from 0.73 to 0.03 success (Tab. 1), a striking demonstration of how much the action pathway relies on one position. The supplement adds cross-task and cross-benchmark sink diagnostics (App. B.1, Fig. 10).

  3. Reasonable baselines. StreamingLLM, H2O and SnapKV are evaluated through a shared physical-gathering interface, so active-KV counts are comparable.

Critical weaknesses

W1 [Novelty] Closely similar methods already exist for LLMs, VLMs and VLAs, and nothing is designed specifically for WAMs.

LLMs: KeepSink ("protected sink + top-k fill") and SinkAge ("recent + heavy hitters + TTL") are direct adaptations of StreamingLLM, H2O and SnapKV, as Sec. 3.3–3.4 acknowledge. VLMs: Visual attention sinks have already been characterized (e.g., See What You Are Told, ICLR 2025), and VLM token pruning has already argued that some sinks are structural anchors that must be kept (e.g., CAPA). Attention-based visual pruning (FastV, SparseVLM) is standard. VLAs: Action-aware visual token pruning and caching already exist (VLA-Cache, ADP, VLA-Pruner, VLA-IAP, all cited). The discovered sink is token 0, the classic first-position sink. Most importantly, the method uses nothing specific to WAMs. It does not exploit the predicted future frames, the video–action coupling, or the denoising structure (Fig. 4 observes the dependency shift across denoising steps but does not use it). The same procedure could be applied unchanged to any transformer policy. The contribution is therefore an application of existing techniques to two new backbones, not a new method for world–action modeling.

W2 [Soundness] The results, including those in the supplement, do not support the "sink-aware" design.

Spatial, Fast-WAM: at κ = 0.75, KeepSink (0.740) is no better than FastV (0.750) or generic Saliency (0.740) (Tab. 1). Only κ = 0.75 and 0.70 are tested (App. Tab. 8), and no baseline is reported at 0.70. Spatial, LingBot-VA (App. Tab. 6): KeepSink does not beat Random selection at two of three budgets: 0.840 vs 0.850 at κ = 0.75, and 0.650 vs 0.690 at κ = 0.50. Temporal: SinkAge is within ±2 points of StreamingLLM, H2O and SnapKV and is not best on any suite (Tab. 2). On Suite-10, DropSink-age, which removes the sink privilege, matches SinkAge (0.785 vs 0.790). StreamingLLM is equally accurate with fewer active tokens and lower wall time (Tab. 3). In App. Tab. 5, random filling (0.845) is the best configuration, and τ = 6 (0.820) also beats the frozen default (0.790). So the sink-aware ranking contributes nothing measurable. The historical compressor effectively reduces to a TTL / recency window, and the spatial one is not better than random selection on LingBot-VA.

Questions for authors

What do KeepSink and SinkAge add over FastV / saliency pruning, Random selection, and StreamingLLM / H2O? Please report budgets where the methods differ (κ ≤ 0.5) for both backbones, with baselines.

Why does random filling (0.845) outperform SinkAge (0.790), and why does DropSink-age match it?

Why is KeepSink slower than Full KV on LingBot-VA? What are the actual memory and latency savings with physical cache pruning?

4 comments
AuthorsOpenReview ↗

Response to Reviewer hQjn: Novelty and Soundness, Validation and Baseline Comparisons, Physical Pruning Performance

We thank the reviewer for the constructive feedback. We address the concerns below.

W1. Novelty and WAM-Specific Contributions

  • Research question. We investigate whether pretrained WAMs contain intrinsic action-to-visual structural sinks and redundant visual history exploitable at inference time without weight updates. Distinct dependency shifts in Fast-WAM and LingBot-VA motivate this backbone-specific analysis, rather than another ranking criterion.
  • Structural sink discovery. Profiling identifies recurrent sinks; KeepSink/DropSink interventions test their functional importance in closed-loop control. Fast-WAM exhibits a concentrated, highly sensitive sink, whereas LingBot-VA has distributed dependencies and weaker single-sink reliance.
  • Historical compressibility and efficiency. At H=72, SinkAge reduces active visual KV by 65.4% (2,326→806) while matching Full KV (79.0% SR). Physical KV eviction further reduces resident KV by 89.2% / 91.6% and peak GPU memory by approximately 20% for KeepSink / SinkAge. Our contribution is the discovery and exploitation of WAM-native context structures, not superior KV ranking.

W2. Soundness of Sink-Aware Design

  • Backbone-dependent sink sensitivity. At κ=0.75, Fast-WAM achieves 74% SR with KeepSink versus 3% with DropSink (Full KV: 73%); LingBot-VA achieves 84% versus 66% (Full KV: 85%). These interventions demonstrate functional sink importance, with stronger single-sink dependence in Fast-WAM. LingBot-VA's distributed dependencies explain why preserving one sink is insufficient under tight budgets.
  • Historical versus forced sink removal. DropSink-age is not spatial DropSink: it disables mandatory recent-sink retention and the mid-age sink bonus but retains dependency–age ranking, TTL, and budget. Sink tokens can still be selected, explaining its 78.5% SR versus SinkAge's 79.0%.

Q1. Comparison with FastV, Saliency, Random, StreamingLLM, and H2O

  • Matched spatial budget (κ=0.50). Fast-WAM KeepSink / FastV / Saliency / Random achieve 70.3% / 72.0% / 71.7% / 36.5% SR; LingBot-VA achieves 65.0% / 80.0% / 74.0% / 69.0%. Our sink audit shows FastV and Saliency naturally rank token 0 first and preserve all five inspected sink candidates. LingBot-VA benefits from preserving additional action-relevant tokens beyond a single sink.
  • Historical budget (B=128). SinkAge / StreamingLLM / H2O / sink-protected Random Fill achieve 78% / 80% / 81% / 83% SR (Full KV: 83%). Random Fill preserves profiled sinks, recent protection, and TTL, randomizing only remaining eligible tokens. These baselines validate historical compressibility, not SinkAge's ranking superiority.

Q2. Random Fill and DropSink-age Ablations

  • Random Fill mechanism. The Suite-10 ablation yields 84.5% SR versus 79.0% for SinkAge. Random Fill preserves profiled sinks, recent protection, TTL eligibility, and the global budget, randomizing only remaining eligible tokens. Thus, fine-grained dependency–age ranking provides no consistent additional benefit under these constraints.
  • DropSink-age interpretation. It removes explicit sink privileges without excluding sink tokens, which can still rank highly through dependency–age scoring. Its competitive performance therefore does not establish that structural sinks are dispensable.

Q3. Why Introduce Age-Aware Historical KV Allocation?

  • Motivation and design. Autoregressive WAM history grows with interaction length. TTL limits token age, B bounds action-visible KV, sink preservation retains structural anchors, and dependency–age scoring allocates the remaining budget. We do not claim optimal age weighting or boundaries.
  • Evidence and simpler baseline. From H=12 to H=72, active-KV growth is 1.203× for SinkAge versus 1.931× for Full KV. At H=72, SinkAge maintains 79.0% SR with 65.4% fewer active KV. Pure Age-only achieves 73% SR, highlighting action-irrelevant redundancy rather than dependence on specific ranking.

Q4. GPU Memory and Inference Latency

  • Physical memory savings. Persistent KV eviction and compaction reduce resident KV from 7.22 GB to 0.78 / 0.61 GB (−89.2% / −91.6%) and peak GPU memory from 32.73 GB to 26.28 / 26.11 GB (−19.7% / −20.2%) for Physical KeepSink / SinkAge.
  • Policy latency. Full KV / Physical KeepSink require 3.75 / 3.80 s/chunk (1.3% KeepSink overhead). Physical SinkAge is at 8.32 s/chunk with K=2 probes (2.22× Full KV). Online dependency probing remains a latency bottleneck; reducing its overhead is future work.

In summary, our evidence supports: (1) backbone-dependent structural sink importance through controlled removal; (2) historical visual-context compressibility through matched-budget comparisons, with GPU-memory savings verified by physical KV pruning; and (3) action-irrelevant redundancy in accumulated visual history, as competitive age-only performance suggests much of this context is unnecessary for action generation.

AuthorsOpenReview ↗

Follow-up on WAM-Specific Contributions and Additional Experimental Evidence

We sincerely appreciate your constructive feedback, which has been particularly valuable in helping us clarify the WAM-specific contributions and functional importance of structural attention sinks.

In response to your concerns, we have provided additional matched-budget comparisons, clarified the Random Fill and DropSink-age ablations, and validated actual GPU-memory savings through physical KV pruning.

As the discussion period approaches its end, please let us know if any concerns remain. If our responses and additional experiments have addressed your main concerns, we would be grateful if you would consider reflecting this in your final rating.

Thank you again for your valuable feedback and consideration.

Reviewer_hQjnOpenReview ↗

response

The new reply came on Nov 18, and there wasn’t time for further discussion. I still hold a negative view of the outcome.

AuthorsOpenReview ↗

Follow-up: Additional Probe-Free Latency Results

We conduct an additional experiment to address the inference-latency concern.

Our probe-free SinkAge variant eliminates additional action-denoising probes while preserving structural sink protection and bounded historical KV allocation. It reduces policy latency from 8.284 to 3.718 s/chunk (−55.1%), achieving Full-KV-comparable latency (3.718 vs. 3.752 s/chunk). Meanwhile, it maintains competitive closed-loop performance (82% vs. 83% SR for Full KV), with 90.8% lower resident KV and 20.1% lower peak GPU memory.

These results further strengthen the practical efficiency evidence for training-free visual-context compression in pretrained WAMs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.