- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper reframes KV-cache low-rank compression for Video LLMs from tuning aggregate task accuracy to analyzing the behavioral intervention set —the exact inputs whose answers change under rank reduction. The authors show has three core properties: it is sparse (5–14% of inputs drive task accuracy knees), unreadable via static or spectral signals (predicting whether more rank fixes an input is at chance, AUC 0.47), and repeatable across nuisance variations, compressors, and architectures (, 55.9% hit rate vs 11.7% base rate). Crucially, is query-specific and fails to transfer across different questions on the same video. These insights enable a "measure-then-amortize" paradigm that labels only the disagreement union (reducing labeling needs to 25–49% of the pool) and reuses profiles to gain +2.4pp at matched budgets. The framework is rigorously validated on MVBench, STAR, and CLEVRER using pre-registered experiments across Qwen2-VL, Qwen2.5-VL, and LLaVA-OneVision.
Critical strengths
The paper’s standout strength is its paradigm-shifting reframe of KV-cache compression from aggregate accuracy-tuning to a controlled behavioral intervention on the input set . By proving that static spectral signals predict failure but are literally at chance when predicting whether more rank fixes an input (AUC 0.47), the authors pull the rug out from under existing spectral allocators and expose what rank calibration actually measures. Equally consequential is the paper's gold-standard empirical hygiene: pre-registering decision rules and run files before scoring, using exact McNemar bounds, and proving that is sparse (5–14%) yet remarkably repeatable () across nuisance axes, compressors, and architectures. This surgical rigor ensures that the proposed measure-then-amortize workflow is not an empirical fluke, but a bulletproof, deployment-ready insight.
Critical weaknesses
While the paper's empirical execution is practically flawless, its primary weakness lies in the restrictive real-world scope required for its "measure-then-amortize" paradigm to be practical. Because the behavioral response set is strictly query-specific and fails to transfer across different questions on the same video (+0.19pp), profiling requires the exact same video-question pair to recur multiple times ( to repetitions) just to break even on compute overhead. In realistic Video LLM deployments where user queries are ad-hoc and rarely repeat verbatim, this profiling pass cannot be amortized, limiting the practical utility of the proposed deployment pattern to niche, highly repetitive workloads.
Questions for authors
(1) What specific production workloads feature exact video-question pair repetitions exceeding the compute break-even threshold ( to passes), given that profiling fails to transfer across different questions on the same video?
(2) For unprofiled, ad-hoc queries (), how much of the +2.4pp gain is retained under the static task fallback, and does cold-start latency degrade overall serving efficiency?
We thank the reviewer.
Q1 — production exact-pair repetitions and compute break-even. Measured compute: profiling costs two one-time forwards per input, i.e. 40% / 8% / 2% added compute over 5 / 25 / 100 repetitions (12.47 s per forward at either rank; the paper's parameter proxy gave 47% / 9% / 2%). It is not repaid by per-forward savings, since the two ranks take the same time in our implementation; the paper's M* ≈ 6 counted visual-key parameters, not compute, and §8 is corrected accordingly (measurement table in our reply to KMnt, Q3). Concrete workloads re-run the same (video, question) under nuisance variations and aggregate all outputs, so whole-answer caching cannot replace the repeated forwards: (i) test-time scaling over frame subsets, e.g. Video Parallel Scaling [Chung et al., CVPR Findings 2026] runs J = 2–16 streams of the same video-question, each on a different frame subset, related to our random frame-resampling axis; (ii) option-permutation evaluation and debiasing, e.g. MMBench's CircularEval [Liu et al., ECCV 2024] (one pass per option) and permutation majority voting over up to 24 orders [Guda et al., IJCNLP-AACL 2026], matching our option-order axis; reusing a cached answer would defeat their purpose. In both, the repetition is generated by the inference procedure itself, so it arises even for a single ad-hoc query: the profile is taken on one realization and reused by the other streams or permutations (VPS uses disjoint frame offsets; we measured random re-sampling). H_xview gains +2.4 pp [1.25, 3.72] when profiling one realization and evaluating another. We have now run CircularEval itself (pre-registered): profiling the original order and evaluating the held-out cyclic rotations under CircularEval-strict (correct only if correct on every rotation) gives +2.43 pp [+1.30, +3.63] (n = 593, three tasks; strict accuracy 66.8% for the policy vs 64.4% for random allocation of the same budget). Profiling on the original frames and evaluating on re-sampled frames gives +4.73 / +1.95 / +2.36 pp (lower bounds +2.09 / +0.47 / +0.49) on moving_attribute / object_existence / STAR; for VPS this remains a proxy. With the M aggregation forwards unchanged and no profiling output reused, profiling makes it M + 2 forwards, an overhead of 2/M: 50–12.5% for M = 4–16 streams, 50–40% for 4–5 circular shifts, 8.3% for 24 permutations.
Q2 — unprofiled ad-hoc queries. Under the static task fallback, ≈0.2 pp of the +2.4 pp (≈8%) is retained (§4: the task axis adds ≈0.2 pp over a static rank at the operating budget). Cold-start latency: an unprofiled input is served directly at its task's rank, with no profiling forward, so it has the same latency as a static-rank deployment (12.47 s per forward in our measurement, identical at r_d = 15 and 30).