- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RAKV, a training-free, query-agnostic framework for maintaining a bounded visual KV memory during streaming video understanding. It combines a base importance score with a density-peak representativeness score through a residual multiplicative rule. During one-time calibration on unlabeled videos, it aggregates these scores over spatial visual groups and uses group-level entropy to allocate a fixed total KV budget across decoder layers. During streaming inference, it applies depth-specific retention rules to select visual KVs within the calibrated layer capacities. The paper evaluates LLaVA-OneVision-7B and Qwen2.5-VL-7B on StreamingBench and the real-time perception/backward-tracing subsets of OVO-Bench, with additional offline results on VideoMME and EgoSchema. The results show improvements over the evaluated training-free baselines, with the larger gains on OVO-Bench. The ablations examine retention, budget allocation, score combinations, calibration granularity, calibration data, and selected hyperparameters.
Critical strengths
-
A useful grouping-level intervention with a reasonably controlled calibration protocol. The method directly addresses the possibility that token-level entropy includes dispersion among redundant representations. Its main configuration calibrates on 64 unlabeled FineVideo training videos and then fixes the layer budgets for downstream evaluation, rather than calibrating on the evaluation benchmarks. Table 2c reports higher OVO-Bench accuracy for group-level than token-level calibration. This makes the proposed calibration unit an interesting, testable contribution even though the entropy identity itself is standard.
-
Consistent empirical gains on two streaming backbones, particularly on OVO-Bench. Table 1 reports gains over HERMES of 2.02 and 3.52 percentage points in OVO-Bench average accuracy for LLaVA-OneVision-7B and Qwen2.5-VL-7B, respectively. Improvements occur in both current perception and backward tracing. The retention-only and retention-plus-budget results in Table 2a provide some evidence that both components are useful. This is more consequential than the smaller StreamingBench gains, and supports the practical relevance of the proposed combination.
Critical weaknesses
-
The incremental novelty is insufficiently established. RAKV combines established components: pseudo-query retention from StreamMem/HERMES, depth-dependent recency and attention scoring in HERMES, entropy-based layer-wise allocation in MEDA, and density-peak representativeness. Its potential novelty lies in combining density-modulated retention with offline group-entropy calibration, but the experiments do not cleanly isolate these contributions. EAKV is neither cited nor evaluated despite relevant overlap in its publicly described entropy-driven allocation and redundancy handling; this warrants discussion. The authors should clarify what is inherited versus new, add the missing EAKV citation and a matched comparison where applicable, and isolate the proposed components through controlled retention-by-allocation ablations.
-
Redundancy reduction and streaming overhead require stronger validation. Proposition 1 is an entropy decomposition for fixed groups and group masses, not a redundancy-invariance result: duplication can change the group masses in Eq. 6, while the spatial grouping in Eq. 5 does not explicitly consolidate repeated content across frames. Likewise, density-modulated scoring followed by independent Top-K does not guarantee coverage of distinct modes, so reduced redundancy requires empirical validation rather than following from the formulation alone. Controlled repetition experiments and matched component ablations would help establish whether the proposed statistics and retention rule deliver the intended benefits. Table 2f also leaves unclear whether TTFT and peak memory include online scoring and compression, making ingestion throughput, compression time, and temporary-memory measurements necessary to assess the claimed streaming efficiency.
Questions for authors
Please supplement the related-work and clarify the delta over EAKV, StreamMem, HERMES, MEDA. EAKV is not currently cited, yet its public description (entropy-driven adaptive KV compression, important-token preservation, and distillation of redundant context into compact anchors) appears closely related to RAKV.
Under the same backbone, frame sampling, candidate pool, total KV budget, layer partition, and scoring implementation, how do importance-only versus density-modulated retention compare under uniform, token-entropy, and group-entropy allocation? In Table 2c, is the token comparator an otherwise identical implementation with only the aggregation unit changed, or an implementation of MEDA? How were the allocation temperature and other hyperparameters selected without using the downstream evaluation labels?
Were the main HERMES comparisons rerun under an identical protocol, and can you report paired uncertainty over videos for the small StreamingBench gains? Please define the TTFT/peak-memory measurement boundaries and report stream-ingestion throughput, compression time, and temporary-memory costs for representative stream lengths and KV budgets.
If the issues mentioned above are resolved, I will increase the score.