acceptodds
Under review as a conference paper at ICLR 2027

What to Forget: Contribution-Aware Memory for Streaming Video Understanding

Abstract

Multimodal Large Language Models (MLLMs) have made impressive strides in offline video understanding tasks. However, adapting them to streaming settings is far from straightforward, as the ever-growing visual context quickly exhausts GPU memory. While prior work mitigates this by compressing the cache and retaining tokens with high attention scores, we find that these scores do not always reflect a token's contribution to the model output. In this paper, we formalize token contribution through the Leave-one-out (LOO) reference, defined as the shift in model output distribution caused by removing a single token. Utilizing the self-prediction objective, we derive the first-order KV Taylor score that shows stronger agreement with LOO than attention-based scores. Building upon this, we thus propose CALM, i.e., Contribution-Aware Layer-wise Memory, a training-free caching strategy that allocates retention budgets based on KV Taylor scores and applies conditional novelty reranking to suppress redundant tokens. To retain long-range context, we further maintain a lightweight text memory that stores compact summaries of past video chunks. Experiments on StreamingBench and OVO-Bench demonstrate that CALM achieves state-of-the-art streaming video understanding results, with gains of 6.25% on StreamingBench and 11.82% on OVO-Bench (12.77% on the Real-Time and 10.87% on the Backward Tracing).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
KV cache, Token Compression, Streaming Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_M5u5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CALM, a training-free KV-cache management framework for streaming video understanding. The method uses leave-one-out (LOO) output divergence as a diagnostic reference for token importance and introduces a gradient-based KV Taylor score computed through a single probe forward–backward pass. CALM combines this score with depth-dependent recency weighting, a center-weighted allocation of cache capacity, conditional novelty reranking, and a bounded text memory containing generated chunk summaries.

Critical strengths
  1. The paper investigates the scoring signal rather than relying only on end-to-end accuracy.
  2. Experiments demonstrate consistent improvements across multiple backbones.
  3. Bounded-memory design and update-cost breakdown address practical concerns.
Critical weaknesses
  1. The relationship between KV Taylor and actual token deletion remains indirect.
  2. Depth-profile analysis does not directly justify the proposed allocation of capacity.
  3. The magnitude and robustness of the improvements need more precise presentation.
Questions for authors
  1. How sensitive are the rankings and downstream results to guidance-prompt wording and the use of a single self-predicted token?
  2. Can the authors demonstrate that layers with stronger KV Taylor–LOO agreement also benefit more from additional cache capacity?
  3. Can the authors provide paired uncertainty estimates for comparisons with HERMES and the attention-based CALM variant? Do the conclusions remain consistent across cache budgets other than 4K?
Reviewer_AaYEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies KV cache compression for streaming video understanding with frozen MLLMs. The authors argue that attention scores, which most existing eviction methods rely on, do not reflect how much a visual token actually contributes to the model's output. They define token contribution with a leave-one-out (LOO) reference: the KL divergence between the model's output distributions with and without a given token. Because LOO is too expensive to compute online, they propose a first-order "KV Taylor" score, obtained from a single forward–backward pass on a self-prediction objective, and show it agrees with LOO better than attention does. On top of this score they build CALM, a training-free caching policy. It uses layer-dependent budgets and scoring rules, a novelty-based reranking step to reduce redundancy, and a lightweight text memory of self-generated chunk summaries. Experiments on streaming and offline video benchmarks across several backbones show improvements over prior training-free methods.

Critical strengths
  1. The paper questions the widely adopted assumption that attention reflects token importance, and tries to ground eviction in a more direct notion of "what happens to the output if this token is removed". This is a sensible and timely direction.
  2. KV Taylor is cheap and easy to plug in. It needs only one extra forward–backward pass per chunk, the cost sits outside the query path, and response latency stays low.
  3. The method is training-free, works across several model families, and consistently improves over strong training-free baselines on streaming benchmarks.
Critical weaknesses
  1. LOO measures fidelity to the full-cache model, not task correctness, yet the paper treats it as ground-truth importance. LOO is the magnitude of the change in the output distribution and is blind to its direction. A token whose removal flips a correct answer to a wrong one and a distractor whose removal flips a wrong answer to a correct one both score high and are both preferentially kept. Optimizing toward LOO therefore means imitating the uncompressed model, including its mistakes. The paper offers only limited evidence that agreement with LOO translates into benchmark accuracy, and it never directly links proxy–LOO agreement to accuracy retention.

  2. The reported gains contradict the LOO framing and are not explained by it. If the objective is to preserve the full-cache model's behavior, full-cache accuracy should be an upper bound. Yet CALM substantially outperforms the uncompressed base model. Those gains must come from mechanisms outside the "contribution-aware" principle, such as removing distracting tokens, recency bias, or the text-summary memory. The ablations also suggest that replacing attention with KV Taylor contributes only a small fraction of the total improvement. The central claim, that contribution-aware scoring is what makes CALM work, is therefore not well supported.

  3. The design rests on a chain of proxies, and each link is weakly justified. LOO stands in for usefulness; KV Taylor stands in for LOO; and the layer-wise agreement between KV Taylor and LOO is used to set per-layer cache budgets. The last step is a logical leap in particular: how reliable a scoring proxy is at a layer says nothing about how much cache that layer needs. If anything, low reliability could argue for a larger budget. Errors can compound along the chain, and the final design choices are only loosely tied to the stated motivation.

  4. The validation setting differs from deployment. The LOO analysis is computed with the actual benchmark questions, effectively an oracle that knows the query. At deployment, CALM scores tokens with a generic guidance prompt before any question arrives. The paper does not show that tokens important for the guidance prompt are the ones future questions will need, and this is precisely the core difficulty of streaming settings.

Questions for authors
  1. Is LOO a valid reference for token usefulness? LOO measures the magnitude of the output change, not whether the change helps or hurts, and single-token removal cannot capture redundant evidence. How often do high-LOO tokens turn out to be harmful (their removal improves the answer)? Would a correctness-aware variant (e.g., the signed change in the correct answer's log-probability) or group-level removal produce a similar ranking, and does KV Taylor still outperform attention under such references?
  2. Does agreement with LOO actually translate into accuracy? Is closer agreement between KV Taylor and LOO predictive of less accuracy loss after compression? Relatedly, why should layers where KV Taylor agrees better with LOO receive larger budgets, rather than budgets set by each layer's measured sensitivity to eviction?
  3. Where do the gains come from? If LOO targets fidelity to the full-cache model, full-cache accuracy should be an upper bound, yet CALM clearly exceeds it. Can you attribute the gains separately to contribution-aware scoring, distractor removal, recency, and the text memory? How much advantage over prior methods remains without the text memory, and how much would prior methods gain by adding it?
Reviewer_Ak8SOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at KV cache compression for streaming video understanding with MLLMs. The authors argue that attention scores don't reflect how much a token actually contributes to the model output. They define a leave-one-out (LOO) reference and propose a cheap first-order approximation, the KV Taylor score. It comes from one forward-backward pass on a self-prediction loss under a generic guidance prompt. On top of this, they build CALM, a training-free caching framework.

Critical strengths
  1. The motivation is clear, the point that attention isn't the same as contribution is easy to follow, and the LOO reference gives a concrete way to check this. KV Taylor shows better agreement with LOO than with attention on several benchmarks.

  2. The method is training-free and practical, keeps memory bounded, and has low answer latency.

Critical weaknesses
  1. The primary concern stems from the method's performance evaluation. Firstly, the improvement compared to the comparison method is very limited. Against HERMES, CALM improves StreamingBench by only +0.12 on Qwen2.5-VL and +0.8 on LLaVA-OV. Secondly, the source of the contribution is unclear; the contribution from text memory appears to be greater. Without text memory, CALM is actually below HERMES (which has no text memory) on StreamingBench, yet the paper neither compares to nor mentions previous work related to text memory, and text memory is not a new idea.

  2. The contribution signal itself seems weak. First, most validations use the actual benchmark questions as probes, while CALM scores tokens using only the first-token loss of a generic summary prompt, so it's unclear how well the score reflects relevance to future questions. Second, the agreement between KV Taylor and LOO is low in absolute terms (Spearman 0.17–0.29 on video, vs. 0.13–0.15 for attention). Third, even LOO itself is only clearly better than attention at 95% removal (Table 7), so a perfect approximation would still bring limited gains. This matches Table 4, where replacing attention with KV Taylor in the full pipeline adds only 1–2%.

Questions for authors
  1. Did the base-model rows in Table 1 run with the same inputs (frame rate, maximum frames) as CALM? Did they use full caching? How much of the performance gain comes from retaining right tokens, and how much from recency or text memory?

  2. Were the numbers for the baselines reproduced under the same settings (frame rate, chunk size, 4K budget), or directly taken from their papers?

  3. How does CALM compare with HERMES without the text memory, or with the same text memory added to HERMES?

  4. KV Taylor uses only the first-token loss under a generic prompt. Could the authors provide results using more tokens or several prompts? How well does the score correlate with relevance to the actual future questions?

  5. Table 6 only reports CALM's own figures. Could the authors provide results comparing memory and latency with HERMES?

  6. TimeChat-Online's 75.36 on StreamingBench in Table 1 is the same as on MVBench in Table 2. Is this a coincidence or a copy error?

Reviewer_UVjcOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Clear motivation, good insight on token compression methods (see strength 1). Solid experiments (See strength 1). The current method has some compatibility issues with modern MLLM inference infrastructure, which may limit its impact on real world streaming video understanding systems. But I think the contribution so far is enought for acceptance.

It would be better if the authors discuss more about the related works. It will better explain why there is a common assumption in token compression that attention scores accurately reflect token importance.

Critical strengths

Clear motivation and meaningful insight. The paper challenges the common assumption in token compression that attention scores accurately reflect token importance. The proposed LOO reference and KV Taylor approximation provide a new direction for contribution-aware KV cache compression.

Solid empirical validation. CALM achieves consistent improvements across multiple MLLM backbones and streaming video benchmarks. The authors also provide extensive ablations and additional statistical evidence, demonstrating the effectiveness of KV Taylor over attention-based scoring.

Critical weaknesses
  1. Proactive latency at the worst situation is still very long

The amourtized latency looks good but the worst latency could be long and does not support steaming understanding. Specifically, the memory is updated chunk-wise. That is to say, if a question comes at the end of a chunk, the total latency = memory update latency + query latency.

  1. Compatibility issues with modern MLLM inference infrastructure

CALM requires a forward-backward pass for every streaming chunk, introducing substantial deployment challenges. Modern MLLM inference systems rely heavily on INT8/INT4 quantization, specialized forward-only attention kernels, CUDA Graphs, and prefill-decode disaggregation, which may not directly support gradient computation over cached KV states.

Although the authors report manageable overhead under their experimental setup, they do not evaluate compatibility with these inference optimizations or quantify the associated implementation, memory, and latency costs.

Given that KV Taylor improves accuracy over attention-based scoring by only 1.28 percentage points on StreamingBench and 1.85 points on OVO-Bench, it remains unclear whether these gains justify the additional complexity in practical serving systems.

  1. There could be more Discussion on Related KV Cache and Visual Token Compression Methods.

The paper would benefit from a more comprehensive discussion of related approaches, particularly ReTaKe (Wang et al., 2024), AdaReTaKe (Wang et al., 2025), and LOOK-M (Wan et al., 2024).

These works explore related techniques such as attention-guided KV cache compression, adaptive token retention, token merging, and multi-stage visual token reduction. Although their compression targets and deployment settings differ, a clearer comparison would help position CALM's technical contributions and highlight its distinctive advantages over existing approaches.

Questions for authors

My major concern is Weakness 2. But I think the current contribution is solid and could inspire this field to some extent. The infra-friendly version could be explored in the future.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.