acceptodds
Withdrawn from ICLR 2027

Where Does the Video Go? Probing Persistent Temporal Memory after Visual KV Eviction in Video LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video large language models (video LLMs) represent a video as a long multimodal prefix, producing large key–value (KV) caches that can remain resident when the same video is reused for multiple temporal queries. Existing efficiency methods primarily reduce visual computation before or during multimodal processing, or compress the resulting cache by retaining entries selected using importance or relevance scores. What remains unclear is which KV states must stay resident once the complete video prefix has already participated in multimodal prefill. We study this post-prefill setting, in which the complete video prefix is processed once and its KV cache is physically compacted before any downstream query is observed. Across three video LLM families, we find that substantial temporal-grounding ability remains after most visual-payload KV states are removed when the compact cache retains sparse temporal-carrier positions aligned with each model's video serialization. Under identical physical KV budgets, retaining these carrier positions consistently outperforms replacing them with additional visual-payload states, showing that post-prefill memory is functionally organized rather than uniformly distributed across the video prefix. Controlled carrier-content interventions further show that the retained states carry information tied to the processed video, rather than acting only as generic positional scaffolds. Motivated by these findings, we introduce TC-KV, a simple, training-free, and query-independent cache policy that preserves the temporal-carrier structure together with a small visual residual after complete prefill. TC-KV thereby substantially reduces persistent video-prefix KV and enables the same compact memory to serve multiple temporal queries, without retraining or modifying the complete multimodal prefill.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Models, KV Cache Compression, Temporal Grounding, Persistent Video Memory, Mechanistic Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EDW5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies which KV states need to remain after a Video LLM has fully processed a video. It hypothesizes that sparse temporal-carrier positions associated with each model’s temporal serialization are particularly important for subsequent temporal queries, and proposes TC-KV, a query-independent post-prefill compression method that preserves these carriers together with a small visual residual. The method is evaluated on three Video LLMs across temporal grounding, moment retrieval, and highlight detection, with additional interventions analyzing carrier topology and content.

Critical strengths
  1. The paper studies a practically relevant post-prefill KV compression setting, where one compact video cache can be reused across multiple queries without re-encoding the video. This is a useful and underexplored efficiency setting.

  2. The paper is generally well written and clearly organized. The method and experimental setup are presented in a way that makes the overall argument easy to follow.

Critical weaknesses
  1. The central temporal-carrier hypothesis is not strongly supported. Carrier positions are defined a priori from model-specific temporal serialization, and the experiments mainly show that these hand-selected positions are useful retention targets. This does not fully establish the stronger claim that they form a privileged persistent-memory mechanism, especially since other contextualized structural positions are not adequately controlled for.

  2. The empirical scope is too limited for the claimed generality. The strongest mechanistic evidence is concentrated on Qwen3-VL, while effects are substantially weaker on the other models. The evaluation also covers a narrow set of temporal tasks and limited baselines, so I am not convinced that the proposed carrier-centric mechanism generalizes broadly across Video LLMs or beyond the specific retention heuristics studied.

Questions for authors
  1. What evidence supports treating the designated temporal-carrier positions as a privileged memory mechanism rather than simply useful contextualized structural tokens?

  2. How do you reconcile the general carrier-memory claim with the much weaker carrier-content effects observed for LLaVA-OneVision-1.5 and InternVL3.5 compared with Qwen3-VL?

Reviewer_HcLvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes TC-KV, which retains temporal carriers and a small visual residual after the video has been fully processed. Experiments show that retaining approximately 8% of the video-associated KV cache preserves substantial temporal-grounding and highlight-detection performance while allowing the same cache to be reused across multiple queries.

Critical strengths
  1. The paper addresses a worthwhile question. At the same cache budget, protecting temporal carriers improves mIoU across all nine model–selector combinations on Charades-STA, providing useful evidence for cache-retention strategies.
  2. The memory savings are substantial. For example, Qwen3-VL retains only 8.14% of the video-associated KV cache while mIoU decreases from 45.11 to 42.77. The cache does not need to be selected again for each query.
Critical weaknesses
  1. Figure 1 is difficult to follow and does not communicate the framework clearly. As the main overview figure, it should make the method easier to understand.
  2. The scope of the mechanistic explanation remains unclear. Replacing carrier content substantially reduces Qwen3-VL's performance but has little effect on LLaVA, suggesting that different models may rely on different ways of storing information. The state-restoration experiment also does not demonstrate recovery in task accuracy, leaving the link between carrier function and task performance insufficiently established.
  3. The baseline comparison is not yet sufficient. The experiments require shared retained positions across layers and attention heads, rather than preserving the layer-wise or per-head selection used by some original baselines. This helps control variables but may change baseline performance. The advantage over the original implementations therefore needs further validation.
Questions for authors
  1. How do you explain LLaVA's nearly unchanged performance after replacing the carrier K/V states? Could the retained visual residual or boundary states account for most of the preserved video information?
  2. Does restoring carrier states improve temporal-grounding accuracy after visual perturbation?
  3. Do the main conclusions still hold when the baselines retain their original layer-wise or per-head implementations at the same KV byte budget?
Reviewer_oaWcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how temporal information is preserved in video LLMs after most visual KV-cache states are removed following complete video prefill. The authors identify sparse temporal-carrier states that retain video-dependent information and propose TC-KV, a training-free, query-independent cache compression strategy that preserves these carriers alongside a small visual residual. Experiments across three video LLM families demonstrate that TC-KV retains substantial temporal-grounding and highlight-detection performance while preserving only approximately 8% of video-associated KV states. The paper further provides controlled interventions to analyze the role of temporal-carrier topology and content in persistent video memory.

Critical strengths
  1. Systematic Mechanistic Analysis of Temporal Memory in Video LLMs

The paper provides an interesting perspective on how temporal information is preserved in video LLMs after visual KV-cache eviction. Unlike prior approaches that primarily focus on identifying important tokens for cache compression, this work investigates the functional roles of temporal-carrier states through controlled interventions. In particular, the matched-budget topology comparisons, cross-video carrier transplantation, and layer-wise analyses provide meaningful evidence that temporal-carrier states play an important role in preserving video-dependent information. These analyses offer useful insights into how video LLMs organize and retain temporal information.

  1. Practical and Effective Query-Independent KV-Cache Compression

The proposed TC-KV provides a simple, training-free, and query-independent approach to post-prefill KV-cache compression. The method preserves substantial temporal-understanding performance while retaining only approximately 8% of video-associated KV states. The evaluations across three video LLM families and multiple temporal-understanding tasks further demonstrate the practical applicability of the approach. Importantly, the compact cache can be reused across multiple queries without re-encoding the video, making it a promising direction for memory-efficient video understanding.

Critical weaknesses
  1. Limited Generalizability of the Mechanistic Findings Across Video LLM Architectures: This is a minor concern. It doesn't cover a lot of architectures available in the community.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.