acceptodds
Under review as a conference paper at ICLR 2027

Redundancy-Aware Visual KV Memory for Streaming Video Understanding

Abstract

Streaming video understanding requires MLLMs to maintain informative visual context under a bounded memory budget as frames arrive continuously. Existing visual KV compression methods largely evaluate KVs as individual units, although feature-similar KVs can encode highly redundant visual content. Such redundancy affects both cross-layer capacity allocation and within-layer KV retention: repeated representations can distort token-level statistics used to estimate layer-wise memory demand and consume multiple retention slots while contributing limited additional information. This work introduces RaKV, a training-free framework for redundancy-aware visual KV memory management. RaKV defines a unified KV score that combines query-agnostic importance with feature representativeness and uses it for both memory decisions. For cross-layer allocation, we aggregate feature-similar KVs into visual groups and measure group-level entropy, yielding a layer-wise demand estimate that is less sensitive to repeated representations. For within-layer retention, the same redundancy-aware score favors KVs that are important and representative of distinct visual patterns. Experiments across streaming video benchmarks and MLLM backbones show that RaKV consistently improves streaming video understanding under a bounded visual KV budget, with gains of up to 9.50% over the original backbone. Our code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Multimodal Large Language Models, KV Cache Compression, Visual Memory, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BhrxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RAKV, a training-free, query-agnostic framework for maintaining a bounded visual KV memory during streaming video understanding. It combines a base importance score with a density-peak representativeness score through a residual multiplicative rule. During one-time calibration on unlabeled videos, it aggregates these scores over spatial visual groups and uses group-level entropy to allocate a fixed total KV budget across decoder layers. During streaming inference, it applies depth-specific retention rules to select visual KVs within the calibrated layer capacities. The paper evaluates LLaVA-OneVision-7B and Qwen2.5-VL-7B on StreamingBench and the real-time perception/backward-tracing subsets of OVO-Bench, with additional offline results on VideoMME and EgoSchema. The results show improvements over the evaluated training-free baselines, with the larger gains on OVO-Bench. The ablations examine retention, budget allocation, score combinations, calibration granularity, calibration data, and selected hyperparameters.

Critical strengths
  1. A useful grouping-level intervention with a reasonably controlled calibration protocol. The method directly addresses the possibility that token-level entropy includes dispersion among redundant representations. Its main configuration calibrates on 64 unlabeled FineVideo training videos and then fixes the layer budgets for downstream evaluation, rather than calibrating on the evaluation benchmarks. Table 2c reports higher OVO-Bench accuracy for group-level than token-level calibration. This makes the proposed calibration unit an interesting, testable contribution even though the entropy identity itself is standard.

  2. Consistent empirical gains on two streaming backbones, particularly on OVO-Bench. Table 1 reports gains over HERMES of 2.02 and 3.52 percentage points in OVO-Bench average accuracy for LLaVA-OneVision-7B and Qwen2.5-VL-7B, respectively. Improvements occur in both current perception and backward tracing. The retention-only and retention-plus-budget results in Table 2a provide some evidence that both components are useful. This is more consequential than the smaller StreamingBench gains, and supports the practical relevance of the proposed combination.

Critical weaknesses
  1. The incremental novelty is insufficiently established. RAKV combines established components: pseudo-query retention from StreamMem/HERMES, depth-dependent recency and attention scoring in HERMES, entropy-based layer-wise allocation in MEDA, and density-peak representativeness. Its potential novelty lies in combining density-modulated retention with offline group-entropy calibration, but the experiments do not cleanly isolate these contributions. EAKV is neither cited nor evaluated despite relevant overlap in its publicly described entropy-driven allocation and redundancy handling; this warrants discussion. The authors should clarify what is inherited versus new, add the missing EAKV citation and a matched comparison where applicable, and isolate the proposed components through controlled retention-by-allocation ablations.

  2. Redundancy reduction and streaming overhead require stronger validation. Proposition 1 is an entropy decomposition for fixed groups and group masses, not a redundancy-invariance result: duplication can change the group masses in Eq. 6, while the spatial grouping in Eq. 5 does not explicitly consolidate repeated content across frames. Likewise, density-modulated scoring followed by independent Top-K does not guarantee coverage of distinct modes, so reduced redundancy requires empirical validation rather than following from the formulation alone. Controlled repetition experiments and matched component ablations would help establish whether the proposed statistics and retention rule deliver the intended benefits. Table 2f also leaves unclear whether TTFT and peak memory include online scoring and compression, making ingestion throughput, compression time, and temporary-memory measurements necessary to assess the claimed streaming efficiency.

Questions for authors

Please supplement the related-work and clarify the delta over EAKV, StreamMem, HERMES, MEDA. EAKV is not currently cited, yet its public description (entropy-driven adaptive KV compression, important-token preservation, and distillation of redundant context into compact anchors) appears closely related to RAKV.

Under the same backbone, frame sampling, candidate pool, total KV budget, layer partition, and scoring implementation, how do importance-only versus density-modulated retention compare under uniform, token-entropy, and group-entropy allocation? In Table 2c, is the token comparator an otherwise identical implementation with only the aggregation unit changed, or an implementation of MEDA? How were the allocation temperature and other hyperparameters selected without using the downstream evaluation labels?

Were the main HERMES comparisons rerun under an identical protocol, and can you report paired uncertainty over videos for the small StreamingBench gains? Please define the TTFT/peak-memory measurement boundaries and report stream-ingestion throughput, compression time, and temporary-memory costs for representative stream lengths and KV budgets.

If the issues mentioned above are resolved, I will increase the score.

Reviewer_APw5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses visual KV cache management for streaming video understanding under a fixed memory budget. Its central observation is that existing methods score KVs as independent units, so feature-similar KVs encoding the same visual content distort two query-agnostic decisions: cross-layer budget allocation (token-level entropy treats repeated representations as separate statistical units, inflating estimated demand) and within-layer retention (near-duplicates each consume a retention slot). The proposed framework, RAKV, is training-free and introduces a single redundancy-aware score S = I(1 + λR) that multiplicatively modulates a depth-specific query-agnostic importance I by a density-peak representativeness term R built from feature-space local density and higher-density separation. The same score drives both decisions: visual tokens are grouped via Otsu-thresholded cosine distance over 8-neighborhood grid pairs with union-find, group-level entropy is computed from the aggregated score mass, and a temperature softmax converts per-layer entropies into per-layer capacities in a one-time offline calibration on 64 unlabeled FineVideo videos; online retention then keeps the top-B candidates per layer. Proposition 1 decomposes token-level entropy into group-level and within-group terms, and is used to argue the group-level statistic is invariant to pure within-group redistribution. Experiments on StreamingBench and OVO-Bench with LLaVA-OneVision-7B and Qwen2.5-VL-7B show consistent gains over training-free baselines at a 6K-per-layer budget with near-zero latency and memory overhead relative to HERMES, supported by ablations on score form, calibration granularity/source/size, hyperparameters, and offline transfer to VideoMME/EgoSchema. Overall, the motivation is clear, the method is cheap and clean, and the empirical study is reasonably thorough — but the margins over the strongest baseline are narrow on StreamingBench, and the three choices most central to the claimed mechanism (how calibration videos are selected, why grouping is restricted to spatial 8-neighborhoods, and how score mass actually splits between versus within groups) are asserted rather than substantiated.

Critical strengths

Strengths Clear problem framing with a precise formal hook. Identifying representation redundancy as a shared factor behind both cross-layer allocation and within-layer retention is a useful perspective, and the entropy decomposition in Proposition 1 (H_tok = H_grp + H_within, with H_within ≤ Σ P(G)log|G|) makes precise exactly why token-level entropy can be inflated by repeated representations without any new visual information.

Practical, training-free, and essentially free at inference. The method operates on frozen backbones with a one-time offline calibration, and Table 2f shows only +0.01s TTFT and +0.03 GB peak memory over HERMES at the same 6K budget — so the reported gains are not bought with extra compute or memory.

Reasonably thorough empirical validation. Gains hold across two streaming benchmarks and two backbones, are reported as mean ± std over three calibration seeds, and are supported by incremental component ablations, a direct token- vs group-level calibration comparison, four score-combination variants, calibration source/size robustness, four hyperparameter sweeps, and generalization to offline VideoMME/EgoSchema (notably +4.04% on the Long subset).

Critical weaknesses
  1. The offline calibration set is underspecified, and the one-shot calibration design is not compared against an adaptive alternative. The layer-wise capacity is calibrated on a "length-stratified" 64-video subset of FineVideo (§4.1), but the selection criteria beyond duration are not described. Since the whole point of group-level entropy is to measure redundancy, the redundancy characteristics of the calibration videos (camera motion, scene-change rate, static vs dynamic content, typical group-size distribution) seem directly relevant to the resulting allocation — a corpus skewed toward static-camera, high-redundancy footage could plausibly bias the profile in Fig. S2. Tables 2d and 2e show robustness across three sources and four sizes, but all three sources are standard curated video benchmarks and the seeds in Table 1 are drawn from the same source, so the experiments establish insensitivity within a fairly narrow regime rather than genuine worst-case robustness. Can the authors report a deliberately skewed or adversarial calibration set (e.g., exclusively static-scene vs exclusively high-motion videos) and show the allocation and accuracy that result? Relatedly: the calibrated capacity is fixed once and held constant for the entire stream, yet the redundancy structure of a stream clearly varies over time. Why not accumulate the statistic online — an exponential moving average of group-level entropy, or another slow-update scheme that lets per-layer capacity drift as the stream evolves? A comparison against such a lightweight online/EMA allocation would clarify whether the offline calibration is a principled design choice or merely a simplification.
  2. Visual group construction is restricted to 8-neighborhood adjacent token pairs, which captures only spatially local redundancy. Eq. 5 restricts the union-find edges to 8-neighborhood pairs on the 2D token grid, so two tokens can only be merged if they are spatially adjacent. This excludes exactly the redundancy that seems most consequential in streaming video: feature-similar tokens that are spatially distant within a frame (repeated texture, background, sky/wall regions) and, more importantly, tokens that recur across frames within the same 16-frame chunk, where consecutive frames are often near-identical. If a large share of real redundancy is temporal, the group-level statistic would still be contaminated by it, which weakens the mechanistic story behind Table 2c. Could the authors ablate the connectivity choice — 4- vs 8-neighborhood, feature-space k-NN edges, and cross-frame/temporal edges — and report both accuracy and the resulting group statistics (number of groups, group-size distribution) for each?
    1. There is no empirical analysis of the entropy decomposition that the central claim rests on. Proposition 1 is used to argue that H_grp excludes the redundancy-driven dispersion present in H_tok, but the paper never measures how score mass is actually distributed between groups versus within them. Concretely, I would like to see, per decoder layer on the calibration set: the measured H_grp and H_within, the ratio H_within/H_tok, and how close H_within sits to its upper bound Σ P(G) log|G|. Without this, the key quantity is unverified — if H_within turns out to be small or roughly constant across layers, then H_grp and H_tok induce nearly the same allocation and the +2.12% in Table 2c would need a different explanation; if it is large and layer-dependent, the paper's argument is strongly supported, and it would be the single most convincing piece of evidence in the paper. A token-level counterpart to Fig. S2 (the two allocation profiles overlaid) plus a breakdown of H_grp/H_within by depth regime (perception / transition / semantic) would directly substantiate the claimed mechanism rather than leaving it inferred from a downstream accuracy delta.
  3. The set of query-agnostic streaming KV baselines seems incomplete, and coverage is uneven across backbones. Table 1 lists five training-free comparisons for LLaVA-OneVision-7B but only two (InfiniPot-V, HERMES) for Qwen2.5-VL-7B, and two of the LLaVA-OneVision entries report no OVO-Bench numbers at all. The strongest claims in the paper — the 9.50% average gain and the Backward Tracing improvements — therefore rest on a comparison set that is both narrower and differently composed than the one used for StreamingBench, which makes it hard to tell whether RAKV leads the retrieval-free field or leads the two methods that happen to be reported on both backbones. Since §2 positions the contribution specifically against retrieval-free, query-agnostic bounded-memory methods, I would expect that category to be covered consistently: could the authors complete the grid, running every training-free baseline on both backbones and both benchmarks, or state explicitly which baselines are infeasible to run and why? Two related points. First, §A.1 adopts its attention-scoring guidance prompt from prior query-agnostic streaming KV memory work that is discussed in §2 but never appears as a baseline; a method whose scoring signal is inherited seems like a natural and necessary comparison, and its absence is conspicuous. Second, the offline results in Table S1 are compared only against self-reproduced HERMES numbers that fall below the published ones, with the gap attributed to a repository issue. Given that the reported margins are 1.20% and 0.27%, the offline comparison is difficult to interpret; reporting both the published and reproduced HERMES numbers, and describing what differs in the setup, would make §C.1 considerably more convincing.
Questions for authors

Overall more analysis is needed in the paper. The results are somewhat incremental but acceptable, but analysis is weak.

Reviewer_tGrYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RAKV, a training-free framework for visual KV cache management in streaming video understanding. It combines query-agnostic importance with density-peak representativeness to guide within-layer KV retention and uses group-level entropy to calibrate cache capacity across decoder layers. Experiments on two streaming video benchmarks and two backbones present improvements of RAKV in comparison to existing methods.

Critical strengths
  • The paper addresses a practically relevant problem about KV cache in streaming video understanding.
  • The framework connects cross-layer capacity allocation and within-layer retention through a shared redundancy-aware perspective.
  • RAKV improves performance across both evaluated backbones, with particularly encouraging gains on OVO-Bench.
  • The calibration experiments are nice, which present sensitivity to the source and size of the calibration data.
Critical weaknesses
  1. Missing ablation isolating the contribution of representativeness. The paper presents representativeness scoring as a core component, but all scoring formulations compared in Table 2b include a representativeness term. A quantitative comparison against importance-only scoring, Siℓ=IiℓS_i^\ell=I_i^\ell, is missing. I suggest fixing the per-layer budgets, candidate filtering procedure, and other settings, and comparing importance-only scoring against Siℓ=Iiℓ(1+λℓRiℓ)S_i^\ell=I_i^\ell(1+\lambda^\ell R_i^\ell) specifically for within-layer retention.

  2. Limited evaluation across cache budgets. The main experiments focus on an average budget of 6K visual KVs per layer. Figure S2 shows allocation across layers under a fixed total budget, rather than performance across different budgets. The results therefore demonstrate improved accuracy at this operating point, but do not establish whether the advantage persists under tighter budgets or how much cache can be saved at comparable accuracy. Evaluating a few low, medium, and high budgets on one backbone and one main benchmark, with accuracy and cache-update time reported against the main baselines, would clarify the performance–resource trade-off.

  3. Missing comparison with or adequate discussion of DSCache. DSCache also targets streaming video understanding under bounded cache capacity and is a relevant system-level baseline. The distinction between cache construction and cache retention does not, by itself, preclude a system-level comparison. I suggest evaluating the methods with matched backbones, video input settings, and actual memory resources, accounting for feature-cache storage and additional query-time computation. If a fair comparison is infeasible, the reasons should be explicitly discussed. Without such a comparison or discussion, RAKV’s accuracy–efficiency competitiveness relative to this class of methods remains unclear. There is also a lof of relevant work in LLM and VLMs that using redundancy-aware KV cache, such as R-KV, etc. A comparison showing the specific challenge in adopting such method for streaming video understanding is important.

  4. Incomplete evaluation of online cache-maintenance overhead. Table 2f reports TTFT and peak GPU memory, but the timing boundaries are unclear, particularly whether cache-update operations are included. Since RAKV introduces feature-distance, density, and separation computations during online retention, query-time latency alone may not capture its additional streaming processing cost. This does not necessarily imply that the overhead is substantial, but the reported measurements are insufficient to establish that it is marginal. Reporting per-chunk cache-update time and end-to-end video processing throughput under matched settings would provide a more complete efficiency assessment.

Questions for authors
  1. How does RAKV compare with importance-only retention under fixed budgets and candidate filtering?
  2. Does RAKV retain its advantage across lower and higher cache budgets?
  3. Could you compare with DSCache under matched resources and clarify the benefit over R-KV-style redundancy scoring?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.