acceptodds
Under review as a conference paper at ICLR 2027

Deconfounding and Compressing History Memory in Streaming Vision-Language-Action Policies

Abstract

How much explicit history does a streaming vision-language-action (VLA) policy need? A common test shortens its attention window, but this intervention can also shrink the physical key/value (KV) pool and change which history remains visible during denoising. We separate these quantities in LingBot-VA and show that an apparent collapse from 97.5% to 0% success on LIBERO-Long is a cache-capacity confound: prediction slots displace the previous chunk’s video tokens. With the window and pool controlled, retaining the immediately preceding chunk achieves 99% success versus 97% with full history; a three-task RoboTwin replication shows the same zero-to-one retention transition. Content interventions test what the retained chunk contributes. We then combine short retention with KV quantization. On LIBERO-Long, two chunks at K4/V2 achieve 99% success, with a paired difference of 0 percentage points (95% CI: [−3, 3]) from the native W = 30 FP reference. Their history-KV storage equivalent under simulated quantization is 28.5 MiB versus 714.4 MiB of native history-slot capacity, approximately 25× smaller. On the evaluated tasks, recent cached content supports high success without retaining the full past; measuring that dependence requires controlling pool capacity separately from the attention window.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Learning, History Dependence, KV Cache Compression, Low-Bit Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eKWTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the historical KV memory requirements of streaming VLA policies and opportunities for compression. It identifies a confound in LingBot-VA: reducing the attention window also shrinks cache capacity, allowing current predictions to evict useful history. The authors independently control window size, cache capacity, and history retention, supported by token-level tracing and ablations on historical content and recency. Experiments on LIBERO-Long and three RoboTwin tasks show that retaining one or two recent chunks can preserve task success. Combining short retention with simulated asymmetric KV quantization yields approximately 25× storage-equivalent compression of historical KV memory, although actual packed-cache memory savings are not measured.

Critical strengths
  1. The paper shows that reducing the attention window can also shrink cache capacity, causing current predictions to evict historical tokens. Controlled experiments and token-level traces substantiate this explanation. This contribution improves the validity of memory evaluations and cautions against interpreting small-window performance collapse as evidence that long histories are necessary.
  2. Retention, recency, content, and quantization ablations show that a small recent cache can preserve performance on the evaluated tasks. Matched warm-up controls and paired comparisons strengthen these findings. The results provide concrete guidance for reducing historical KV storage while distinguishing explicit cache requirements from the information already encoded in retained representations.
Critical weaknesses
  1. The central retention results are established on LingBot-VA using LIBERO-Long and three RoboTwin tasks. The additional FastWAM experiments address quantization rather than history retention. The evaluation also lacks a setting where longer explicit history demonstrably improves performance. Consequently, the conditions under which short retention becomes insufficient remain unclear, limiting the broader applicability of the conclusions.
  2. The approximately 25× historical KV compression is a storage-equivalent calculation: the quantization experiments dequantize values back into a BF16 cache rather than implementing packed low-bit storage. Although this limitation is clearly disclosed, the experiments do not establish the actual memory savings or runtime trade-offs of the proposed cache configuration. Since the quantization strategy largely follows existing methods, this implementation gap limits the demonstrated practical contribution beyond the paper’s diagnostic findings.
Questions for authors
  1. The native implementation evicts individual slots, whereas the controlled implementation retains whole chunks; numerical equivalence is established only before eviction. What evidence separates the effect of cache capacity from the change in eviction behavior after eviction begins? Does any existing comparison hold eviction behavior fixed while varying capacity?
  2. How is the nominal window parameter WW translated into the attention mask in the controlled implementation? In the fixed recent-2, 1,152-slot sweep, which historical tokens are actually accessible at W=2W=2 versus W=130W=130? If the retention rule already determines the effective read set, what independent role does WW play in this comparison?
Reviewer_za9kOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the management and compression of historical KV caches in streaming VLA policies, primarily using LingBot-VA as the experimental backbone. The authors first identify an issue in the native implementation: reducing the attention window size also decreases the physical KV-cache capacity, which may prevent the complete retention of historical information. To address this issue, they modify the evaluation protocol to independently control the attention window and physical KV-cache capacity. Based on this protocol, the authors investigate the effects of retaining different numbers of historical KV-cache chunks, varying the recency and content of cached values, and applying KV-cache quantization. Experiments show that retaining a small number of recent chunks can maintain high success rates for the evaluated LingBot-VA policy on LIBERO-Long and three RoboTwin tasks. Furthermore, combining KV-cache quantization with the retention of two historical chunks achieves the same success rate as the original baseline, while reducing the estimated KV-cache storage requirement by approximately 25× under simulated quantization.

Critical strengths
  • Well-motivated problem formulation and systematic analysis of history dependence. This paper investigates the management of historical KV caches during streaming VLA inference. By decoupling the attention window, physical KV-pool capacity, and history retention policy, the authors identify a confounding factor in the native LingBot-VA implementation and provide a more controlled evaluation protocol. Furthermore, the experiments on history retention, recency, and cached content offer useful empirical insights into how the policy utilizes historical KV information.

  • Promising results for training-free KV-cache compression. The paper combines short-history retention with low-bit KV quantization and demonstrates that historical KV storage requirements can be substantially reduced while maintaining high task success rates in the evaluated settings. Specifically, on LIBERO-Long, retaining the two most recent chunks with K4/V2 quantization achieves the same observed success rate as the native floating-point reference. These results suggest that the memory efficiency of existing streaming VLA policies can be improved without retraining the backbone network.

Critical weaknesses
  • The paper is framed as a study of historical KV caches in streaming VLA policies, but its core experiments are limited to LingBot-VA. Although the authors conduct a source-code audit of other streaming video diffusion systems and provide additional quantization experiments on FastWAM, the main experiments concerning historical KV-cache dependence and the window-capacity confound are still restricted to LingBot-VA. These findings have not been directly validated on other streaming VLA architectures.

  • The analysis of history dependence remains limited. Although the experiments show that retaining only the most recent chunk's KV cache can maintain high success rates, the representation of this chunk may already contain information from earlier chunks. Furthermore, the experiments lack evaluations on tasks that genuinely require long-term memory. Therefore, the current results only demonstrate that the evaluated model and tasks do not require retaining a large amount of explicit historical KV cache, but do not fully establish the actual long-term history requirements of streaming VLA policies.

  • Insufficient RoboTwin evaluation. The RoboTwin experiments cover only three selected tasks, and the paper does not provide a sufficiently clear justification for their selection. This limited evaluation makes it difficult to assess the robustness and generalizability of the findings across different manipulation scenarios. In particular, different robotic tasks may require substantially different amounts of historical information. A broader evaluation covering more RoboTwin tasks, especially those with different temporal dependencies, would make the experimental conclusions more convincing.

  • Lack of validation of actual GPU memory savings from KV-cache compression. The paper reports that combining history retention with K4/V2 quantization can reduce historical KV-cache storage requirements by approximately 25×. However, the experiments use simulated quantization, in which the quantized KV cache is still stored in BF16 format after dequantization. Therefore, the reported 25× compression ratio is based on theoretical storage estimates rather than actual GPU memory measurements. The authors are encouraged to implement a real low-bit KV cache and report GPU memory consumption and end-to-end inference latency to validate the practical deployment benefits of the proposed approach.

Questions for authors
  • Main figure and presentation. Could the authors improve the main figure to more clearly illustrate the relationships among the attention window, physical KV-cache capacity, and history retention policy? In particular, it would be helpful to more explicitly distinguish the native implementation from the controlled implementation.

  • Task coverage. Could the authors expand the evaluation to cover a wider variety of tasks, particularly on RoboTwin, and include tasks that explicitly require long-term historical information to demonstrate the robustness of the findings?

  • Experiments on other VLA models. Could the authors validate the proposed analysis on other streaming VLA architectures to demonstrate that the identified KV-cache issue and the findings regarding history retention are not specific to LingBot-VA?

  • Actual GPU memory usage and runtime performance. Could the authors conduct experiments using an actual packed low-bit KV-cache implementation and report real GPU memory consumption, peak GPU memory usage, and end-to-end inference latency, rather than relying solely on simulated quantization and storage-capacity estimates?

Reviewer_qQQNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

They study history-KV requirements in streaming VLAs. It identifies a confound in LingBot-VA: reducing the attention window also shrinks the cache, allowing prediction tokens to displace historical information. After decoupling these factors, retaining one recent chunk maintains high success. Combining short retention with KV quantization yields an estimated 25× reduction in history-storage equivalents.

Critical strengths
  • The paper identifies a concrete evaluation confound and supports its diagnosis with cache traces, window sweeps at fixed retention, and replication across tasks.
  • The evaluation includes closed-loop success, paired comparisons, and content interventions, while explicitly distinguishing simulated quantization from measured deployment benefits.
Critical weaknesses
  1. Insufficient evaluation on memory-dependent benchmarks. LIBERO-Long and the three selected RoboTwin tasks do not establish a clear need for long-term memory. Evaluation on explicitly memory-dependent tasks, such as RoboDojo’s memory subset, is needed to determine whether short retention remains sufficient when historical information is essential.

  2. Missing real-world validation. Leave the practical benefits and reliability of the compressed history unverified.

  3. Unverified deployment benefits. The 25× reduction is a storage estimate under simulated quantization; KV remains stored in bf16. Actual packed-cache memory, latency, and throughput are not measured.

  4. Limited novelty and generality. The main finding concerns LingBot-VA's cache implementation, while compression combines short retention with existing quantization. The core history experiments are not replicated on another backbone.

  5. Unclear value of auxiliary memory. The teacher fails on the corresponding offline tasks, making improved action fidelity difficult to interpret. QSDM also shows no statistically closed-loop gain (Both closed-loop success differences have confidence intervals containing zero).

  6. Weak presentation. Lack of figures for human reading. Beyond a small cache-budget schematic, there is no clear teaser or method overview explaining the native failure, controlled cache design, and compression pipeline, making the paper unnecessarily difficult to follow.

Questions for authors
  1. Additional validation: Does short retention remain sufficient on explicitly memory-dependent tasks, such as RoboDojo’s memory subset (see Weakness #1)?

  2. Real-world validation: Can real-world closed-loop results establish the reliability of compressed history (see Weakness #2)?

  3. What memory, latency, and throughput gains does an actual packed-KV implementation achieve, beyond simulated storage estimates?

  4. What contribution extends beyond diagnosing LingBot-VA’s cache implementation and applying existing quantization? Is there evidence that the core finding generalizes across backbones?

  5. What supports QSDM’s control value, given the unsuccessful teacher rollouts and closed-loop difference CIs containing zero?

  6. Repolish writing and presentation. See weakness #6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.