acceptodds
Under review as a conference paper at ICLR 2027

LifeKV: Full-Lifecycle KV Cache Retrieval for Streaming Video Question-Answering

Abstract

Streaming VideoQA suffers from increasing GPU memory consumption and inference latency as the visual context grows. Existing KV retrieval methods reduce this overhead, but typically target only specific inference stages and lack a unified mechanism for controlling memory and latency throughout streaming inference. We propose LifeKV, a training-free full-lifecycle KV retrieval framework that keeps historical visual KV states revisitable during visual prefilling, question prefilling, and autoregressive decoding. The complete historical visual KV memory is stored in RAM, while only a compact query-relevant context is maintained on the GPU. LifeKV further introduces layer-ahead asynchronous KV prefetching to overlap descriptor-based retrieval and RAM-to-GPU KV recall with ongoing Transformer computation. Experiments show that LifeKV maintains competitive VideoQA accuracy while substantially reducing the growth of GPU memory usage and inference latency as the visual context length increases, providing an efficient and scalable solution for streaming video understanding over existing VideoQA models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Multimodal Large Language Models, Streaming Video Question-answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C88FOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets KV cache management for streaming VideoQA. The authors observe that existing retrieval methods retrieve only during question prefill or decoding, while visual prefill still suffers from unbounded GPU memory growth or lossy compression. They propose LifeKV, a training-free framework that keeps the full historical visual KV in RAM and recalls query-relevant blocks to the GPU across visual prefill, question prefill, and decoding. Experiments on OVO-Bench and StreamingBench with four backbones show competitive accuracy and lower peak GPU memory.

Critical strengths

-The problem is well motivated and practically important. Extending retrieval to the visual-prefill stage is a sensible fix to a real limitation of prior streaming pipelines.

-The efficiency results, especially the flattened memory growth and latency at long contexts, are encouraging. The training-free design is appealing.

Critical weaknesses

-My main concern is novelty and depth. Each component—KV offloading, top-k retrieval, layer-wise reuse, and prefetching—is familiar from long-context LLM systems; the contribution is mostly a careful combination for VideoQA.

-The evaluation is also narrow: two streaming benchmarks, mostly real-time subsets, and a limited set of baselines. Stronger KV offload/retrieval systems are not compared. Accuracy gains are small and inconsistent across backbones. In some settings LifeKV is close to or below Full Attention, so the claim of competitive accuracy needs more support.

Questions for authors

Strenthen the experimental evaluation. Please inlcude more benchmarks and more recent KV offloading/retrieval baselines, and report accuracy and efficiency under the same controlled setting with more backbone scales.

3 comments

W1. Novelty and Technical Contributions

We thank the reviewer for the assessment of novelty and technical depth. We acknowledge that KV offloading, Top-K retrieval, layer-wise reuse, and prefetching have been explored in long-context LLM systems. Our contribution does not lie in these individual components, but in designing a unified full-lifecycle KV retrieval framework tailored to streaming VideoQA.

Unlike conventional long-context inference, streaming VideoQA continuously receives visual inputs without knowing future queries or their required historical evidence. LifeKV addresses this challenge by enabling retrieval across visual prefilling, question prefilling, and decoding, allowing historical visual information to be revisited throughout inference. It further integrates full-fidelity historical KV storage with layer-ahead asynchronous prefetching to reduce GPU memory consumption and retrieval overhead.

We will clarify that the primary contribution is the coordinated design of these mechanisms for scalable streaming inference, rather than the individual techniques themselves. The existing module ablations and system analysis further support their respective contributions.

W2. Evaluation Scope and Accuracy Tradeoff

We acknowledge that the evaluation can be broadened. Our focus is streaming rather than offline VideoQA, where video contexts grow continuously and future queries are unknown. This setting makes persistent historical KV management particularly important. The main paper emphasizes real-time tasks, while the complete OVO-Bench evaluation is already reported in the appendix.

Regarding accuracy, our objective is not to consistently outperform Full Attention, but to preserve its accuracy while substantially reducing the growing GPU memory and inference costs. Performance differences across backbones and tasks remain, and we do not claim uniform improvements. We therefore position competitive accuracy in terms of the tradeoff against efficient streaming KV retrieval methods, rather than absolute superiority over Full Attention.

To strengthen this evidence, we have expanded the evaluation with longer videos, additional backbones, and a recent retrieval baseline, as detailed in Q1.

Q1. Expanded Benchmarks, Baselines, and Controlled Evaluation

We have expanded our evaluation with an additional long-video benchmark, more backbone scales, a recent streaming KV baseline, and controlled efficiency comparisons. The additional experiments are summarized below.

(1) Additional Long-Video Benchmark

We evaluate LifeKV, Full Attention, ReKV, and MuKV on RVS-Movie, which contains 22 movie videos averaging approximately one hour. All methods use the same evaluation judge (GLM-5.3-Flash) to ensure consistent scoring.

BenchmarkMethodAccuracy (%)
RVS-MovieFull Attention26.51
LifeKV25.40
ReKV25.11
MuKV22.53

LifeKV achieves accuracy close to Full Attention and outperforms both ReKV and MuKV, supporting its applicability to longer video streams.

(2) Additional Backbone Families and Scales

We extend the evaluation to InternVL2.5-1B and InternVL2.5-4B, supplementing the original LLaVA-OV and Qwen3-VL model families.

BackboneOVO-Bench (%)StreamingBench (%)
InternVL2.5-1B46.9168.14
InternVL2.5-4B55.1072.51

These results further demonstrate that LifeKV can be applied across different backbone architectures and parameter scales without additional training.

(3) Comparison with a Recent Streaming KV Baseline

We additionally compare LifeKV with StreamKV (AAAI 2026) on OVO-Bench and StreamingBench using LLaVA-OV-0.5B and Qwen3-VL-2B.

BackboneBenchmarkFull AttentionStreamKVLifeKV
LLaVA-OV-0.5BOVO-Bench44.3543.1443.58
StreamingBench55.1152.5154.67
Qwen3-VL-2BOVO-Bench53.5553.5254.63
StreamingBench71.4271.6870.58

LifeKV outperforms StreamKV in three of the four settings, while StreamKV achieves higher accuracy on Qwen3-VL-2B with StreamingBench. Overall, LifeKV maintains accuracy close to Full Attention across the evaluated configurations.

(4) Controlled System Efficiency Comparison

We further compare Full Attention, MuKV, ReKV, StreamKV, and LifeKV on Qwen3-VL-2B using a single NVIDIA A800 GPU. We evaluate video contexts of 128, 512, and 1024 frames and report end-to-end inference latency, peak GPU memory, and host RAM usage.

FramesMethodE2E Latency (s) ↓Peak VRAM (GiB) ↓Host RAM (GiB) ↓
128Full Attention2.537.972.10
MuKV4.219.503.70
ReKV3.979.482.17
StreamKV5.475.912.14
LifeKV4.744.985.73
512Full Attention14.8024.612.60
MuKV22.9273.299.43
ReKV37.0520.668.04
StreamKV33.2516.022.71
LifeKV17.185.0717.53
1024Full Attention47.1253.643.31
MuKV67.2078.5017.05
ReKV156.4348.6215.94
StreamKV103.0439.163.35
LifeKV36.105.1432.74

LifeKV exhibits the strongest efficiency advantages at longer context lengths. At 1024 frames, it achieves approximately 23.4% lower E2E latency and 90.4% lower peak VRAM than Full Attention, while also being faster than all evaluated retrieval baselines.

Together, these experiments provide broader evidence that LifeKV maintains competitive accuracy across different streaming VideoQA benchmarks and backbones while offering a favorable latency and GPU memory tradeoff for long video streams.

Thanks for your response. I keep my score, due to the limited technical novelty.

Reviewer_PyYsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LifeKV, a training-free KV-retrieval wrapper for streaming VideoQA. It keeps all historical per-frame visual KV blocks in pinned CPU RAM and mean-pooled key descriptors on the GPU. A mean-pooled query selects the top-k frames during visual prefilling, question prefilling, and decoding. Each selection is shared by a group of G=4 layers and computed δ=2 layers early, so the RAM-to-GPU copy overlaps computation. Across four backbones, it matches or exceeds full attention on OVO-Bench real-time, is comparable on StreamingBench real-time, and keeps peak VRAM flat up to 1,024 frames.

Critical strengths
  1. A practical, training-free systems design for a real bottleneck.
    • Per-frame descriptors on the GPU, full-fidelity KV in pinned RAM, group-wise selection and layer-ahead prefetching are a sensible combination that is easy to adopt.
    • Fig. 3 shows peak VRAM staying roughly flat up to 1,024 frames, at ≈4.7 GiB for 0.5B and ≈18.5 GiB for 7B. Full attention grows to ≈10.4 and ≈52 GiB.
    • At 1,024 frames, E2E latency is roughly a quarter of full attention's at 7B (≈40 vs. ≈150 s) and about half at 0.5B. It stays close to the GPU-resident variant.
  2. A broad, mostly controlled experimental setup with competitive accuracy.
    • It covers four backbones from two families, with the same sampling and prompts across methods.
    • It includes two controlled variants (LifeKV-GPU, LifeKV-Sync), ablations of G, δ, c and k, and a measurement of top-k overlap between prefetched and synchronous selections.
    • On OVO-Bench real-time, LifeKV is above full attention on all four backbones.
    • On full 12-task OVO-Bench (Table 4), its overall score beats full attention on three of the four backbones (+1.12, +1.08 and +1.60). The appendix also reports the tasks where it loses.
Critical weaknesses
  1. The "full-lifecycle" claim is unsupported and largely not new.
    • Table 2 ablates the three retrieval stages for latency and VRAM only, so it is never shown whether visual-prefill (VP) or decoding (DEC) retrieval improves accuracy. The full three-stage setting is also the slowest (31.53 vs. 26.68 s for question-prefill retrieval alone). DEC retrieval is effectively untested, because both benchmarks are short-answer multiple choice.
    • The VRAM savings come from offloading, not VP retrieval. ReKV is equally flat in Fig. 3 without any VP retrieval.
    • The building blocks already exist, and only InfiniGen is cited. InfLLM (NeurIPS 2024) retrieves CPU-offloaded blocks during chunked prefilling and decoding. Quest (ICML 2024) does query-aware selection during decoding. OmniKV (ICLR 2025) reuses one selection across layers with CPU offload. InfiniGen already prefetches the next layer's KV.
    • ReKV is misdescribed. Sec. 2 says ReKV's sliding window "may discard early evidence", but App. A.1 correctly states that ReKV offloads all KV and retrieves it at question time.
  2. The controlled comparisons are internally inconsistent.
    • LifeKV-Sync matches neither reading of "synchronous". It scores 42.90 on 0.5B OVO-Bench. Selecting at the target layer (δ=0, Table 3) gives 45.20, and blocking on the default δ=2 selection (LifeKV itself) gives 43.58. Fig. 4 at G=4 (≈45.2) agrees with δ=0, so LifeKV-Sync is the outlier.
    • LifeKV-GPU is inconsistent across settings. It differs from LifeKV only in where the KV lives, and it matches LifeKV exactly on 0.5B OVO-Bench. Elsewhere, it differs from both variants by 2–5 points, e.g., 73.45 vs. 77.37 (LifeKV-Sync) and 77.78 (LifeKV) on Qwen3-VL-8B OVO real-time.
    • LifeKV vs. LifeKV-Sync is not a pure scheduling comparison. Prefetching also changes which frames are retrieved (top-k overlap 0.63–0.73), and synchronous δ=0 is 1.6 points more accurate than the default.
  3. The baselines and evaluation scope do not support the accuracy claims.
    • Re-implemented baselines collapse in suspicious patterns. MuKV is near chance (28.8–30.4) on OVO real-time for three of four backbones, including Qwen3-VL-8B. ReKV falls 18–24 points below full attention only on Qwen3-VL OVO-Bench, while matching it on StreamingBench. This suggests porting or answer-parsing errors, and baseline configurations and budgets are not reported.
    • The headline results use only real-time subsets, which exercise long-range memory least. On full OVO-Bench, LifeKV is below full attention on Forward for all four backbones (−1.6 to −5.1), and no long-horizon benchmark is used.
    • The full-attention reference is unclear for long videos. At 196 tokens per frame, LLaVA-OV's context holds only about 160 frames. If full attention runs past that limit, "exceeds full attention" means beating a degraded reference.
  4. The efficiency evidence is thin.
    • Scaling study (Fig. 3): It averages only 5 videos with no error bars. It also uses a different frame rate from the accuracy runs (1.0 vs. 0.5 fps) and omits LifeKV-Sync, so the benefit of asynchrony at scale is never shown.
    • Unexplained latencies (Table 3): δ=1 is slower than synchronous recall (8.63 vs. 7.96 s), so a breakdown into scoring, transfer and stall time is needed.
    • Host RAM is not reported, although memory is moved rather than removed and still grows linearly with stream length.
    • Hyperparameters are chosen on test data: G and δ are selected on OVO-Bench, despite App. A.3's claim that no hyperparameters are tuned on the evaluation sets.
Questions for authors
  1. Reproducibility.
    • Why does LifeKV-Sync (42.90) match neither δ=0 (45.20) nor LifeKV (43.58)?
    • Which δ do LifeKV-GPU and Fig. 4 use?
    • Please report mean ± std over repeated runs for the main comparisons.
  2. Stage contributions.
    • Please ablate VP, QP and DEC retrieval for accuracy, and define the "VP off" setting in Table 2, including video lengths.
    • Please compare against ReKV-style offloading with the same active budget and local window.
    • How does Full Attention handle videos longer than the backbone's context?
5 comments

We thank the reviewer for the detailed feedback. We have revisited the experimental configurations, corrected reporting inconsistencies, and conducted additional evaluations. Our responses are organized below.

W1. Full Lifecycle Retrieval and Novelty

W1-1. Retrieval Stage Contributions and Decoding Limitations

We additionally evaluate the accuracy contribution of visual prefill (VP), question prefill (QP), and decoding (DEC) retrieval on LLaVA-OV-0.5B using OVO-Bench.

VPQPDECBackwardRealtimeForwardOverall
OffOnOff37.9953.9040.9244.27
OnOnOff37.6353.4141.0044.01
OffOnOn37.9953.9040.9244.27
OnOnOn37.6353.4141.0044.01

The results show that enabling VP and DEC retrieval has little effect on accuracy in the evaluated setting. VP retrieval primarily controls GPU memory growth while enabling incoming visual chunks to access relevant historical evidence. DEC retrieval allows the model to dynamically refresh historical evidence as autoregressive generation progresses. Without DEC retrieval, two alternatives exist. First, retaining only the KV segments selected during question prefilling reduces GPU memory consumption but restricts decoding to a fixed, limited historical context. Second, retaining all historical KV states allows decoding to access the complete history but sacrifices the GPU memory savings. DEC retrieval addresses this tradeoff by keeping the full historical KV memory in RAM while selectively recalling relevant segments as decoding proceeds.

We acknowledge that the short multiple-choice answers in our benchmarks do not sufficiently demonstrate this benefit. Moreover, repeated retrieval during token generation introduces additional latency, explaining the higher E2E latency of the full three-stage configuration. Nevertheless, DEC retrieval extends LifeKV beyond fixed question-prefill retrieval, enabling adaptive access to historical evidence during longer autoregressive generation. We will clarify this motivation and the current evaluation limitation in the revision.

W1-2. GPU Memory Savings and Visual Prefill Retrieval

We agree that GPU memory savings primarily arise from offloading historical KV states rather than VP retrieval alone. As shown in Figure 3, ReKV also maintains relatively stable VRAM usage through offloading and a fixed local attention window. The distinction lies in how historical KV is accessed during visual prefilling. ReKV uses a local sliding window, whereas LifeKV selectively retrieves historical segments relevant to incoming visual chunks. This enables nonlocal historical interaction while maintaining a compact active context. Figure 3 further demonstrates a favorable latency and memory tradeoff for LifeKV, while the accuracy experiments evaluate its effectiveness across different backbones.

W1-3. Novelty Relative to Existing KV Retrieval Methods

We appreciate the references to InfLLM, Quest, OmniKV, and InfiniGen, and will include them in the revised related work. These studies establish important foundations for memory retrieval, query-aware selection, cross-layer reuse, and KV prefetching in LLM inference.

Our contribution is not the invention of these individual operations, but their integration into a unified retrieval framework tailored to streaming VideoQA. Unlike conventional long-text inference, streaming VideoQA requires continuous visual prefilling, preservation of extensive historical visual evidence, and low-latency responses to queries arriving at arbitrary timestamps. LifeKV addresses these requirements through retrieval across VP, QP, and DEC, together with group-wise selection and layer-ahead asynchronous prefetching. This design jointly targets historical evidence accessibility, GPU memory usage, and retrieval latency in streaming video inference.

W1-4. Clarification of ReKV's Sliding Window

We agree that our original description of ReKV was imprecise. ReKV preserves historical KV states through offloading rather than permanently discarding them. Our intended distinction concerns historical information accessible during visual prefilling. ReKV processes incoming frames using a fixed local sliding window, so earlier evidence outside that window is not directly available for attention at this stage, although it remains retrievable during question answering. In contrast, LifeKV retrieves relevant historical KV segments during visual prefilling, enabling interactions with nonlocal visual evidence. We will revise Section 2 to clarify this distinction.

W2. Consistency of Controlled Comparisons

W2-1. Correction of LifeKV-Sync Results

We acknowledge an incorrect result entry for LifeKV-Sync in the original manuscript. LifeKV-Sync corresponds to synchronous retrieval at the target layer group, namely δ = 0, rather than blocking on the default δ = 2 selection. Accordingly, the correct LLaVA-OV-0.5B OVO-Bench overall accuracy is 45.20, as reported for δ = 0 in Table 3, rather than 42.90%. The corresponding task-group results will also be aligned across the main table and appendix.

W2-2. LifeKV-GPU Configuration and Accuracy Differences

LifeKV-GPU keeps historical KV states on the GPU and does not employ asynchronous prefetching. Therefore, it differs from LifeKV in both memory placement and retrieval timing, rather than storage location alone. A repeated evaluation on LLaVA-OV-0.5B OVO-Bench obtains 45.11%, correcting the duplicated result in the original table.

More generally, LifeKV performs Top-K retrieval over a small number of historical segments. Small changes in retrieval scores can alter which segments are selected, and these differences may propagate through subsequent retrieval stages. Therefore, different retrieval schedules and execution paths can produce different accuracy results. We will clarify the configuration differences .

W2-3. Retrieval Differences Under Asynchronous Prefetching

We agree that LifeKV and LifeKV-Sync do not constitute a pure scheduling comparison with identical retrieved segments. Asynchronous prefetching selects historical segments using query representations from earlier layers, whereas synchronous retrieval selects them at the target layer group. Consequently, the selected segments may differ, as reflected by the Top-K overlap of 0.634 to 0.732 in Table 3. The primary purpose of asynchronous prefetching is to reduce exposed retrieval latency rather than improve accuracy. We will explicitly describe the associated tradeoff between retrieval freshness and latency reduction.

W3. Baseline Reliability and Evaluation Scope

W3-1. Baseline Reproduction and Configuration

We reexamined the ReKV and MuKV evaluations. The low OVO-Bench accuracy of MuKV was reproduced in additional runs, while we identified an error in the original ReKV evaluation and corrected the corresponding results.

The corrected results on Qwen3-VL are summarized below, alongside LifeKV for reference.

BenchmarkModelReKVMuKVLifeKV
OVO-BenchQwen3-VL-2B53.9633.4454.63
OVO-BenchQwen3-VL-8B56.6837.0157.87
StreamingBenchQwen3-VL-2B70.9871.7870.58
StreamingBenchQwen3-VL-8B74.6776.1176.79

The revised results show that LifeKV remains competitive, although the relative performance varies across backbones and benchmarks.

For reproducibility, our baseline reproductions follow the original methods' default configurations, without intentionally tuning their retrieval budgets for LifeKV. ReKV uses a local window of 15,000 tokens and a retrieval size of 64 in its official configuration. MuKV uses a retrieval size of 64 together with its default multigranularity compression and reranking settings. In comparison, LifeKV retrieves only eight historical single-frame segments by default. We will report the configurations more explicitly and moderate the corresponding accuracy claims.

W3-2. Full OVO-Bench Results and Long Video Evaluation

Table 1 focuses on real-time tasks because of space limitations, while Appendix Table 4 provides results across all three OVO-Bench task groups.

We acknowledge that LifeKV underperforms Full Attention on Forward tasks. These tasks require integrating evidence across successive future observations, for which a fixed Top-K historical retrieval budget can be restrictive. This limitation motivates adaptive retrieval budgets based on temporal scope, while the current results demonstrate stronger accuracy retention on real-time tasks.

To further assess long-video understanding, we evaluate LifeKV, Full Attention, ReKV, and MuKV on RVS-Movie, containing 22 movie videos with an average duration of approximately one hour. All methods are evaluated using the same GLM-5.3-Flash judge.

MethodRVS-Movie Accuracy (%)
Full Attention26.51
LifeKV25.40
ReKV25.11
MuKV22.53

LifeKV approaches Full Attention and remains competitive with the retrieval baselines. These results extend our evaluation to longer videos while also indicating that substantial room remains for improving long-horizon reasoning.

W3-3. Full Attention Beyond the Native Context Window

We agree that Full Attention should not be interpreted as an unconstrained reference when video inputs exceed the backbone's supported context length. In practice, Full Attention must operate within the available context budget, often requiring frame subsampling or truncation for longer videos. We therefore use very long inputs, such as 512 and 1,024 frames, primarily for system efficiency comparisons rather than claims of superior answer accuracy.

Reviewer_xWCROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the growth of GPU memory and inference latency in streaming VideoQA, where frames arrive continuously and each new frame contributes hundreds of visual tokens to an ever-expanding KV cache. The authors' framing is that existing KV-retrieval methods activate retrieval only at selected points in the inference pipeline — typically once a question arrives — so historical KV states still accumulate on the GPU during visual prefilling, and the retrieved context stays frozen while decoding proceeds. LifeKV is a training-free, model-agnostic framework that extends retrieval across all three stages of the inference lifecycle (visual prefilling, question prefilling, and decoding) against a single evolving historical visual KV memory, with the complete full-fidelity memory held in pinned RAM and only lightweight per-segment descriptors plus the currently active recalled blocks resident on the GPU. Retrieval works at segment granularity (one frame by default): each segment is summarized by a mean-pooled key descriptor, the current stage's query states are mean-pooled into a single vector, and the top-k segments by scaled inner product are gathered from RAM and transferred back. Two system-level optimizations reduce the resulting overhead — retrieval is performed once per group of G consecutive Transformer layers and the selected indices are reused within the group, and layer-ahead asynchronous prefetching launches retrieval for an upcoming group from a source layer δ positions earlier so that selection and RAM-to-GPU transfer overlap with ongoing Transformer computation rather than sitting on the critical path. Across four backbones from two model families (LLaVA-OV 0.5B/7B, Qwen3-VL 2B/8B) on OVO-Bench and StreamingBench, LifeKV matches or slightly exceeds full-attention accuracy while flattening peak VRAM growth and slowing latency growth markedly out to 1,024-frame contexts, and two controlled variants (LifeKV-GPU, LifeKV-Sync) cleanly separate the effect of RAM offloading from that of asynchronous scheduling. The engineering is careful and the systems evidence is convincing. My reservations concern the retrieval side rather than the systems side: nothing in the paper establishes that the top-k selected segments actually contain the evidence a question requires, there is no error analysis attributing failures to retrieval misses rather than model limitations, and the sensitivity to the prefetch distance δ is both under-explored and, on the paper's own numbers, less benign than the text suggests.

Critical strengths

Strengths

  1. The paper address a timely issue of KV-Retrieval while maintaining memory and efficiency.
  2. The results are strong and since nothing is compressed, it makes the full system pretty practical.
Critical weaknesses

Weaknes

  1. How do you ensure that retrieved topk historical segments in current retrieval stage have all the accurate information that is needed to answer the question? Analytically can we see in the errors which all places the model missed because of not having enough context.
  2. How finicky is the performance of the model to delta? The performance as delta varys is unexplored and delta has only been ablated on 4 values and one backbone. Need more results.
  3. Can we measure the divergence between LifeKV-prefilled and full-attention KV states? and whether this compunds as the length of video increases?
Questions for authors
  1. Also what is the RAM used , the paper only reports peak VRAM.
  2. Why does the performance of Qwen3-VL collapse by 24 points? It's too big of a drop. Overall paper as nice engineering results but the paper still lacks ablations and analysis.
1 comment

We thank the reviewer for the constructive comments. We have conducted additional experiments and analyses to address the concerns regarding retrieval coverage, prefetch distance, KV divergence, memory usage, and baseline accuracy.

W1. Retrieval Coverage and Fixed Budget Tradeoffs

We acknowledge that fixed Top-K retrieval cannot guarantee complete coverage of the evidence required to answer a question. However, similarity threshold based retrieval also lacks such guarantees and introduces model dependent threshold selection and unpredictable memory consumption. LifeKV adopts fixed Top-K retrieval for its simplicity and predictable GPU memory budget, making it suitable for memory constrained deployment. Figure 5 further shows that accuracy generally improves with larger retrieval budgets and gradually saturates. We will clarify the limitation for questions requiring evidence distributed across many frames.

W2. Sensitivity to Prefetch Distance

We extend the prefetch distance δ beyond the original range of 0 to 4 and evaluate two backbones on a single NVIDIA A800 GPU. Larger distances allow earlier retrieval but may introduce less recent retrieval decisions.

δLLaVA-OV-0.5B Accuracy (%)Top-k OverlapQwen3-VL-2B Accuracy (%)Top-k Overlap
045.101.00052.781.000
244.010.73353.410.852
444.560.63453.910.821
644.790.74953.700.842
845.180.69354.430.807

Accuracy remains relatively stable across different prefetch distances, despite variations in retrieval overlap. This suggests that cross-layer retrieval redundancy allows larger lookahead windows without substantial accuracy degradation. Exploring larger retrieval groups may further reduce retrieval overhead.

W3. KV State Divergence Across Video Lengths

We additionally compare LifeKV and Full Attention KV states on LLaVA-OV-0.5B using OVO-Bench. Keys and Values are compared at corresponding frame, token, and layer positions.

Metric128 Frames512 Frames1024 Frames
Key Cosine Similarity ↑0.99530.99260.9856
Value Cosine Similarity ↑0.98450.97560.9592
Key Normalized L2 ↓0.07130.09510.1258
Value Normalized L2 ↓0.12780.17350.2246

LifeKV maintains high KV similarity to Full Attention, although divergence gradually increases with video length, particularly for Value states. This indicates that sparse retrieval introduces some accumulated representation differences, while preserving substantial similarity to the full-context states.

Q1. Host RAM Consumption

We additionally report LifeKV's host RAM usage on Qwen3-VL-2B using a single NVIDIA A800 GPU.

FramesPeak VRAM (GiB)Host RAM (GiB)
1284.985.73
5125.0717.53
10245.1432.74

Host RAM usage increases as historical KV states accumulate, while peak VRAM remains nearly constant. This reflects LifeKV's intended tradeoff of using host memory to achieve scalable GPU memory consumption.

Q2. Correction of ReKV Accuracy on Qwen3-VL

Upon reexamining our ReKV reproduction, we identified and corrected an implementation issue responsible for the unexpectedly large accuracy drop. The corrected ReKV achieves 53.96 overall accuracy on Qwen3-VL-2B and 56.68 on Qwen3-VL-8B, matching Full Attention and our LifeKV. We will update the corresponding results and discussion. While the corrected ReKV narrows the accuracy difference, LifeKV's primary advantage remains its full-lifecycle retrieval design and improved long-context inference efficiency, particularly in latency and GPU memory usage.

Comments on the paper

1 comment

Dear Reviewers,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.