acceptodds
Under review as a conference paper at ICLR 2027

What Rank Calibration Actually Measures: The Sparse, Repeatable, Query-Specific Response of Video LLMs to KV-Cache Compression

Abstract

Low-rank compression of a video language model’s key cache is usually tuned by reading a rank budget off the cache’s spectrum and reporting the change in average accuracy. We treat compression as a controlled intervention instead and ask where it acts. Holding the input and the query fixed and moving only the rank, the set D of inputs whose answer changes has three properties, measured on MVBench with Qwen2-VL-7B under rules frozen before the data. It is sparse: a task’s accuracy-versus-rank knee is carried by 5–14% of its inputs, which is why a per-task allocator gains little over the best uniform rank (+1.3pp, CI [−0.3, +3.0]). Its allocation-relevant part is unreadable from the signals we test: whether a cheap rank breaks an input is predictable (AUC 0.755), but whether more rank fixes it is at chance (AUC 0.47, [0.38, 0.57]). And it repeats: across option order, frame sampling, a second rank pair, real video, a second compressor and a second architecture, an input that changed its answer once changes it again with probability 56% against a base rate of 12% (n=2,392, κ=0.50). That repeatability belongs to the video–question pair: profiling a video on some questions does not inform its others (+0.19pp against a video-level oracle of +8.3). Together these identify what a rank calibration has to measure—the pair, not the task, the spectrum or the video—and imply a measure-then-reuse pattern that labels only the inputs where two ranks disagree (25–49% of the pool, an exact bound) and returns +2.4pp (CI [+1.25, +3.72]) on held-out realizations at a matched key-cache budget, in workloads whose outputs must be recomputed rather than looked up.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, video language models, tensor decomposition, high-resolution video, memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pYrgOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper reframes KV-cache low-rank compression for Video LLMs from tuning aggregate task accuracy to analyzing the behavioral intervention set D\mathcal{D}—the exact inputs whose answers change under rank reduction. The authors show D\mathcal{D} has three core properties: it is sparse (5–14% of inputs drive task accuracy knees), unreadable via static or spectral signals (predicting whether more rank fixes an input is at chance, AUC 0.47), and repeatable across nuisance variations, compressors, and architectures (κ=0.50\kappa=0.50, 55.9% hit rate vs 11.7% base rate). Crucially, D\mathcal{D} is query-specific and fails to transfer across different questions on the same video. These insights enable a "measure-then-amortize" paradigm that labels only the disagreement union (reducing labeling needs to 25–49% of the pool) and reuses profiles to gain +2.4pp at matched budgets. The framework is rigorously validated on MVBench, STAR, and CLEVRER using pre-registered experiments across Qwen2-VL, Qwen2.5-VL, and LLaVA-OneVision.

Critical strengths

The paper’s standout strength is its paradigm-shifting reframe of KV-cache compression from aggregate accuracy-tuning to a controlled behavioral intervention on the input set D\mathcal{D}. By proving that static spectral signals predict failure but are literally at chance when predicting whether more rank fixes an input (AUC 0.47), the authors pull the rug out from under existing spectral allocators and expose what rank calibration actually measures. Equally consequential is the paper's gold-standard empirical hygiene: pre-registering decision rules and run files before scoring, using exact McNemar bounds, and proving that D\mathcal{D} is sparse (5–14%) yet remarkably repeatable (κ=0.50\kappa=0.50) across nuisance axes, compressors, and architectures. This surgical rigor ensures that the proposed measure-then-amortize workflow is not an empirical fluke, but a bulletproof, deployment-ready insight.

Critical weaknesses

While the paper's empirical execution is practically flawless, its primary weakness lies in the restrictive real-world scope required for its "measure-then-amortize" paradigm to be practical. Because the behavioral response set D\mathcal{D} is strictly query-specific and fails to transfer across different questions on the same video (+0.19pp), profiling requires the exact same video-question pair to recur multiple times (M∗≈6M^* \approx 6 to 130130 repetitions) just to break even on compute overhead. In realistic Video LLM deployments where user queries are ad-hoc and rarely repeat verbatim, this profiling pass cannot be amortized, limiting the practical utility of the proposed deployment pattern to niche, highly repetitive workloads.

Questions for authors

(1) What specific production workloads feature exact video-question pair repetitions exceeding the compute break-even threshold (M∗≈6M^* \approx 6 to 130130 passes), given that profiling fails to transfer across different questions on the same video?

(2) For unprofiled, ad-hoc queries (M<M∗M < M^*), how much of the +2.4pp gain is retained under the static task fallback, and does cold-start latency degrade overall serving efficiency?

1 comment

We thank the reviewer.

Q1 — production exact-pair repetitions and compute break-even. Measured compute: profiling costs two one-time forwards per input, i.e. 40% / 8% / 2% added compute over 5 / 25 / 100 repetitions (12.47 s per forward at either rank; the paper's parameter proxy gave 47% / 9% / 2%). It is not repaid by per-forward savings, since the two ranks take the same time in our implementation; the paper's M* ≈ 6 counted visual-key parameters, not compute, and §8 is corrected accordingly (measurement table in our reply to KMnt, Q3). Concrete workloads re-run the same (video, question) under nuisance variations and aggregate all outputs, so whole-answer caching cannot replace the repeated forwards: (i) test-time scaling over frame subsets, e.g. Video Parallel Scaling [Chung et al., CVPR Findings 2026] runs J = 2–16 streams of the same video-question, each on a different frame subset, related to our random frame-resampling axis; (ii) option-permutation evaluation and debiasing, e.g. MMBench's CircularEval [Liu et al., ECCV 2024] (one pass per option) and permutation majority voting over up to 24 orders [Guda et al., IJCNLP-AACL 2026], matching our option-order axis; reusing a cached answer would defeat their purpose. In both, the repetition is generated by the inference procedure itself, so it arises even for a single ad-hoc query: the profile is taken on one realization and reused by the other streams or permutations (VPS uses disjoint frame offsets; we measured random re-sampling). H_xview gains +2.4 pp [1.25, 3.72] when profiling one realization and evaluating another. We have now run CircularEval itself (pre-registered): profiling the original order and evaluating the held-out cyclic rotations under CircularEval-strict (correct only if correct on every rotation) gives +2.43 pp [+1.30, +3.63] (n = 593, three tasks; strict accuracy 66.8% for the policy vs 64.4% for random allocation of the same budget). Profiling on the original frames and evaluating on re-sampled frames gives +4.73 / +1.95 / +2.36 pp (lower bounds +2.09 / +0.47 / +0.49) on moving_attribute / object_existence / STAR; for VPS this remains a proxy. With the M aggregation forwards unchanged and no profiling output reused, profiling makes it M + 2 forwards, an overhead of 2/M: 50–12.5% for M = 4–16 streams, 50–40% for 4–5 circular shifts, 8.3% for 24 permutations.

Q2 — unprofiled ad-hoc queries. Under the static task fallback, ≈0.2 pp of the +2.4 pp (≈8%) is retained (§4: the task axis adds ≈0.2 pp over a static rank at the operating budget). Cold-start latency: an unprofiled input is served directly at its task's rank, with no profiling forward, so it has the same latency as a static-rank deployment (12.47 s per forward in our measurement, identical at r_d = 15 and 30).

Reviewer_VQxhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a systematic empirical investigation into what rank calibration actually measures in low-rank KV-cache compression for video language models. The authors study compression as a controlled intervention and characterize the input-level behavioral response to changes in rank.

Critical strengths

This paper presents an insightful problem formulation and a meaningful conceptual contribution, supported by a rigorous and comprehensive empirical evaluation.

Critical weaknesses
  1. Although the paper evaluates multiple model architectures and compression settings, the main held-out policy gains are evaluated on a relatively small set of tasks. Could the method generalize to a broader range of real-world video understanding tasks and query distributions.

  2. While the experimental evaluation is comprehensive, some results could be presented more clearly. For example, Table 4 reports held-out policy improvements and repeatability statistics under different rank-pair configurations, which may require readers to carefully consult the accompanying explanations.

Questions for authors

The evaluation considers several rank pairs, but the main policy results are concentrated on a limited number of operating points. How sensitive are the observed profiling benefits to the choice of rank pair and compression level?

1 comment

We thank the reviewer.

Q — sensitivity to rank pair and compression level. Held-out policy value H (profile one realization, evaluate another), Table 4:

  • {15, 30}: pooled +2.4 pp [+1.25, +3.72]; moving_attribute +4.3 (LB +1.9).
  • {11, 30}: object_existence +5.6 (LB +2.9), counterfactual_inference +6.1 (LB +3.2), STAR +5.0 (LB +2.3).
  • Repeatability of the rescued set at {11, 30} (option shuffle): κ = 0.70 (object_existence), 0.36 (counterfactual).
  • Disagreement-set repeatability at the second pair {11, 30} (Table 3, n = 398): an input in D on one realization is in D on the other 53.3% of the time vs 8.4% for inputs not in D; κ = 0.48 (κ = 0.64 at {15, 30}, frames axis).

W1 — broader tasks. We added a 200-question LongVideoBench evaluation over four duration groups (8 s–1 h) with Tucker, flat-SVD and Palu; the complete results, including dense accuracy and all baseline comparisons, are in our reply to KMnt (part 3/3). The paper also includes the Qwen2.5-VL replication (knee-carrying steps change 6.5–10.7% of inputs).

W2 — Table 4. Revised: the rank pair is a separate column for every row, and the H columns and the two κ columns are separated, with their definitions in the caption.

Reviewer_KMntOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which inputs respond to low-rank compression of video-language-model key caches. It defines a disagreement set D between two ranks and distinguishes membership in D from the signed accuracy benefit of using the higher rank. Experiments investigate sparse responses, prediction from static signals, and repeatability across nuisance realizations. The central result is within-configuration cross-view repeatability: Table 3 reports a 55.9% disagreement hit rate against an 11.7% base rate, with kappa=0.50. A policy profiled on one realization gives +2.43 percentage points on another at a matched rank budget (Section 7). The paper also establishes the exact label-saving identity that examples with identical predictions contribute zero to a paired accuracy difference. It presents these findings as an empirical measurement framework, with a narrowly scoped profile-and-reuse deployment pattern.

Critical strengths
  1. Rather than relying on aggregate accuracy curves, the paper separates prediction disagreement D from signed accuracy benefit and tests whether both repeat across held-out realizations of the same question (Sections 6–7). Table 3 (p. 7) reports a 55.9% cross-view disagreement hit rate against an 11.7% base rate (kappa = 0.50) with a question-clustered bootstrap over 798 distinct questions, and Section 7 (Table 4) shows that a sign-sensitive rank-allocation policy profiled on one view earns +2.43 percentage points (95% CI [+1.25, +3.72]) on another view at a matched rank budget.

  2. The paper is unusually thorough in reporting negative results and scope boundaries: Table 1 and Appendix A flag the non-budget-matched action-antonym task, Section 3 separates the 5-frame flat-SVD substrate from the 32-position experiments, Table 5 shows near-zero cross-question transfer (+0.19pp), Section 7 reports failed cross-compressor transfer of beneficial inputs (H = +0.01), and Section 8 directly compares against answer caching.

Critical weaknesses
  1. Several central claims and deployment assumptions in the paper are overstated relative to the actual results. First, Section 4 claims that each rank step changes predictions on only 5–14% of inputs, which holds only for the three knee-setting steps in Figure 2 while other steps reach 19.5%–36.7% disagreement (Appendix F), and Section 2 claims that the same sensitive inputs recur across compressors and architectures even though Table 3 only tests within-model repeatability and cross-compressor transfer is near zero (H = +0.01 in Section 7). Second, the profile-and-reuse deployment analysis in Section 8 uses key-parameter counts (c15 = 14,992, c30 = 21,532) as a direct proxy for forward-pass compute, and in all tested views the ground-truth answer never changes across repetitions—meaning simple answer caching already achieves 73.4% accuracy at zero forward cost versus 73.3% for the profiling policy. Please reconcile the per-step disagreement statistics and cross-compressor claims, and demonstrate a realistic recurring workload where answer caching is inapplicable under actual forward-compute costs to better justify the soundness and practical value of your proposed method.

  2. In Section 3, the flat-SVD baseline is only evaluated on the 5-frame setup and omitted from the 32-position experiments in Sections 4–7, and all tested benchmarks (MVBench, STAR, CLEVRER) consist of short clips. Moreover, while the paper cites Palu [R2] and VL-Cache [R3] and includes a simplified "Palu-style" spatial-SVD variant in Table 3, it does not clarify how this variant differs from full Palu [R2]. Please include the flat-SVD baseline in the 32-position setting, compare against full Palu [R2] and adaptive VLM KV-cache compression methods such as VL-Cache [R3] under matched budgets, and evaluate on longer video benchmarks like LongVideoBench [R1] to better validate the generality and effectiveness of your proposed method.

References:

[R1] Wu et al. (2024). LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. https://arxiv.org/abs/2407.15754

[R2] Chang et al. (2024). Palu: Compressing KV-Cache with Low-Rank Projection. https://arxiv.org/abs/2407.21118

[R3] Tu et al. (2024). VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration. https://arxiv.org/abs/2410.23317

Questions for authors
  1. Can you provide a complete per-step table of total prediction changes, wrong-to-correct, correct-to-wrong, and wrong-to-wrong flips across all ladder steps, and revise the 5–14% statement in Section 4 so it clearly refers only to the three knee-setting steps?

  2. Did any experiment directly test whether the same input IDs fall into D across different architectures or compressors (beyond the H = +0.01 cross-basis signed-benefit test in Section 7), and if not, can you rephrase the claim in Section 2 to distinguish replication of the phenomenon from cross-configuration identity overlap?

  3. Under actual wall-clock latency and total GPU memory measurements (including Tucker factorization/reconstruction and untouched value caches), how do the break-even repetition counts M* in Section 8 change, and what concrete video-LLM workload requires recomputing the forward pass on nuisance variations of the same clip and query instead of caching the answer?

5 comments

(part 1/3)

We thank the reviewer for the detailed reading. Answers in order.

Q1 — per-step flip table. Full table (new appendix), Qwen2-VL-7B, committed ladder records, no new inference. "Changes" = predicted letter differs between the two ranks.

taskstepNchanges%W→CC→WW→W
object_existence8→111502114.01470
11→13150117.3740
13→1515064.0510
15→3015085.3440
moving_attribute8→112004120.528112
11→132002010.08111
13→15200115.5740
15→30200199.51720
counterfactual_inference8→111504328.7191014
11→131502516.71474
13→15150117.3731
15→301502114.01164

§4 now reads: a step changes the prediction on 4–29% of a task's inputs (largest at 8→11, up to 28.7%), while the three steps that set the tasks' depths (bold) move 5–14%. The reviewer's two figures are correctness flips (W→C + C→W) in the cached ladder records: 19.5% is moving_attribute 8→11 (39/200, the row above), and 36.7% is counterfactual_inference taken end to end over the whole ladder, 8→30 (55/150; prediction changes 71/150 = 47.3%), i.e. the four steps compounded, not one step; its adjacent steps flip 19.3 / 14.0 / 6.7 / 11.3%. The end-to-end 8→30 rows are added to the appendix table (prediction changes 22.7 / 26.5 / 47.3%, correctness flips 22.7 / 26.0 / 36.7% for object_existence / moving_attribute / counterfactual_inference), so both single-step and whole-ladder magnitudes are stated.

Q2 — direct test of identities across architectures/compressors. Across model variants, yes (new): on 500 inputs scored by both Qwen2-VL-7B and Qwen2.5-VL-7B (records aligned by position and checked: over all rank arms, both-correct answer letters always agree and one-correct letters always differ), D shows weak above-chance overlap: 20.8% of Qwen2-VL's D inputs are also in Qwen2.5-VL's D vs 8.2% otherwise (κ = 0.13; task- and difficulty-stratified permutation p = 0.014; lift +0.126, 95% CI [0.014, 0.256]), well below within-model frame-resampling repeatability (κ = 0.64, Table 3). Rescued-set overlap is inconclusive (CI crosses 0). Across compressors there is no direct identity test; profile transfer gives H = +0.01 pp [−0.32, +0.42] (§7). §1 and §2 are qualified accordingly: Table 3's second-compressor (κ = 0.33) and second-architecture (κ = 0.59) arms measure repeatability within each configuration.

(part 2/3)

Q3 — M* under wall-clock and total GPU memory. Measured on Qwen2-VL-7B, 30 inputs (10 per primary task), arm order rotated per input, synchronized timing, same decode-split protocol as the paper.

Measured:

denser_d = 15r_d = 30
wall-clock per forward (prefill + answer token)11.12 s12.47 s12.47 s
peak GPU memory (current implementation)25.85 GiB25.85 GiB25.85 GiB

Tucker factorization and reconstruction add 1.35 s (+12%). The r_d = 30 − r_d = 15 difference is 0.003 s, 95% CI [−0.010, +0.016] s. Our implementation reconstructs the key and stores it at full size (it measures the accuracy effect of the compressed key, as in fake quantization), so peak memory is identical across ranks. Hence there is no measurable wall-clock or current-memory break-even.

Analytic, hypothetical factor storage: stored total KV (compressed visual K + uncompressed text K + all V) is 0.516 (r_d = 15) and 0.520 (r_d = 30) of dense.

The paper's M* = 5.97 is a crossover in visual-key-parameter accounting (c15 = 14,992, c30 = 21,532) against always using r_d = 30, not a forward-compute break-even. Under hypothetical factor storage with uncompressed text K and V counted, the byte-accounting crossover is M* = 263. The two profiling forwards add wall-clock overhead of 40% / 8% / 2% over 5 / 25 / 100 evaluations.

Does the amortization signal survive more aggressive total-KV compression? We additionally 2-bit quantize all V (KIVI-style, groups of 32, fp16 scale/min) on top of the compressed visual K, n = 598 over the three tasks, views A/B; a --no_vq bypass control reproduces the archived runs bit-identically on two machines. The pre-registered cross-view amortization signal survives: H_xview = +2.34 pp, 95% bootstrap CI [+1.11, +3.64], versus +2.43 pp with bf16 V (survival, not equivalence — per-task point estimates drop on attribute 4.34→3.56 and counterfactual 1.26→0.97). The query-specific disagreement structure is preserved on object/attribute (Cohen κ = 0.54–0.62) but weak on counterfactual (κ = 0.14–0.23). Added dense-accuracy cost is modest except on one counterfactual view (−6 pp, McNemar p = .012, unadjusted; the other view −2 pp, p = .45); compressed-arm deltas lie in −7..+1 pp. Under the byte model this lowers the total-KV crossover from M* = 263 to 57 (2-bit V with fp16 group metadata), substantially but not fully closing it.

Concrete workloads re-run the same (video, question) under nuisance variations and aggregate all outputs, so whole-answer caching cannot replace the repeated forwards: (i) test-time scaling over frame subsets, e.g. Video Parallel Scaling [Chung et al., CVPR Findings 2026] runs J = 2–16 streams of the same video-question, each on a different frame subset, related to our random frame-resampling axis; (ii) option-permutation evaluation and debiasing, e.g. MMBench's CircularEval [Liu et al., ECCV 2024] (one pass per option) and permutation majority voting over up to 24 orders [Guda et al., IJCNLP-AACL 2026], matching our option-order axis; reusing a cached answer would defeat their purpose. In both, the repetition is generated by the inference procedure itself, so it arises even for a single ad-hoc query: the profile is taken on one realization and reused by the other streams or permutations (VPS uses disjoint frame offsets; we measured random re-sampling). H_xview gains +2.4 pp [1.25, 3.72] when profiling one realization and evaluating another. We have now run CircularEval itself (pre-registered): profiling the original order and evaluating the held-out cyclic rotations under CircularEval-strict (correct only if correct on every rotation) gives +2.43 pp [+1.30, +3.63] (n = 593, three tasks; strict accuracy 66.8% for the policy vs 64.4% for random allocation of the same budget). Profiling on the original frames and evaluating on re-sampled frames gives +4.73 / +1.95 / +2.36 pp (lower bounds +2.09 / +0.47 / +0.49) on moving_attribute / object_existence / STAR; for VPS this remains a proxy. With the M aggregation forwards unchanged and no profiling output reused, profiling makes it M + 2 forwards, an overhead of 2/M: 50–12.5% for M = 4–16 streams, 50–40% for 4–5 circular shifts, 8.3% for 24 permutations. For ordinary repeated answering, caching is applicable and scores 73.4%, versus 73.3% for recomputation with the policy (§8).

§8 is revised accordingly: the phrase "breaks even after M* ≈ 6" is replaced by the matched-budget statement (+2.4 pp, two one-time forwards, 40/8/2% overhead) with each accounting crossover named by its unit.

(part 3/3)

All runs: Qwen2-VL-7B, the paper's decode-split protocol, paired per input against Tucker in the same process, exact McNemar, 95% paired-bootstrap CI. Δ = Tucker − baseline accuracy (pp).

Flat-SVD at 32 positions (per-head SVD of the 32-frame visual key cache), MVBench, n = 198–200 per task. Integer rank cannot match Tucker's bytes within 2%, so flat-SVD receives more bytes than Tucker (1.22× at r_d = 15, 1.14× at r_d = 30).

taskΔ r_d = 15Δ r_d = 30
object_existence+19.2 [11.1, 27.3], p < 10⁻³+15.2 [7.1, 23.2], p = 4×10⁻⁴
moving_attribute+22.0 [14.0, 30.0], p < 10⁻³+29.5 [22.0, 37.0], p < 10⁻³
counterfactual_inference+18.0 [10.5, 25.5], p < 10⁻³+16.0 [7.5, 24.5], p = 3×10⁻⁴

Palu. The official Palu code pins transformers 4.37.2, which predates Qwen2-VL (added in 4.45), so we re-implemented Palu's low-rank decomposition of k_proj on Qwen2-VL-7B: the rank-r subspace minimizing output reconstruction error on calibration data (equivalent to Palu's whitened SVD of the weight; bias kept exact), calibrated once on 64 held-out videos disjoint from all evaluation tasks, applied before RoPE as in Palu, visual K only (as for Tucker). Two variants: per-head (M-LRD) and group-head (G-LRD) with group size 4 (the official example's setting; here all 4 KV heads). Ranks give each variant at least Tucker's per-input bytes (M-LRD 1.20× / 1.11×, G-LRD 1.10× / 1.04× at r_d = 15 / 30); the shared basis is not charged. Rank search, quantization and kernels were not evaluated.

Per-head (M-LRD):

taskΔ r_d = 15Δ r_d = 30
object_existence+19.2 [9.1, 29.3], p = 3×10⁻⁴+15.7 [5.6, 25.8], p = 0.004
moving_attribute+40.0 [31.5, 48.5], p < 10⁻³+39.5 [31.0, 48.0], p < 10⁻³
counterfactual_inference+16.0 [8.0, 24.0], p = 3×10⁻⁴+16.0 [8.0, 24.5], p = 3×10⁻⁴

Group-head (G-LRD, group size 4; latent rank 11 / 15 of 512):

taskΔ r_d = 15Δ r_d = 30
object_existence+13.6 [4.6, 23.2], p = 0.006+13.1 [4.6, 21.7], p = 0.004
moving_attribute+29.5 [21.5, 37.5], p < 10⁻³+24.0 [17.0, 31.5], p < 10⁻³
counterfactual_inference+17.5 [9.5, 25.5], p < 10⁻³+20.5 [12.5, 29.0], p < 10⁻³

VL-Cache (re-implemented: layer-wise sparsity-adaptive budget + modality-aware token scoring, K and V evicted), at four budgets keeping 22–48% of total KV, n = 198–200 per task:

taskdenseVL-Cache, four budgets (keeps 22–48% of total KV)Tucker r15 / r30 (in prefill)Tucker r15 / r30 (after prefill)
object_existence80.380.366.2 / 65.280.3 / 80.3
moving_attribute90.090.0–90.580.0 / 84.090.5 / 90.5
counterfactual_inference57.058.0–58.552.0 / 55.558.5 / 58.0

VL-Cache compresses after the full prompt (video and question) is prefilled; the paper's Tucker compresses during prefill. The last column applies the same Tucker reconstruction at VL-Cache's point (after prefill): against the best VL-Cache budget per task, McNemar p = 1.0 in all six cells. For reference, Tucker's stored total KV under hypothetical factor storage is 52% of dense, i.e. more than every VL-Cache budget. Per-budget table in the new appendix.

LongVideoBench (validation, 200 questions, 50 per official duration group, 64 frames → 5,760 visual tokens). Baselines receive 1.09–1.20× Tucker's bytes.

duration groupndenseTucker r15 / r30flat-SVD r15 / r30Palu (per-head) r15 / r30
8–15 s5064.050.0 / 50.040.0 / 48.036.0 / 38.0
15–60 s5066.048.0 / 54.044.0 / 42.042.0 / 46.0
3–10 min5048.036.0 / 38.038.0 / 40.034.0 / 32.0
15–60 min5042.028.0 / 28.030.0 / 26.026.0 / 26.0
all20055.040.5 / 42.538.0 / 39.034.5 / 35.5

Overall Δ: vs Palu +6.0 [1.5, 11.0] p = 0.023 (r15), +7.0 [2.0, 12.0] p = 0.013 (r30); vs flat-SVD +2.5 [−2.5, 7.5] p = 0.42 (r15), +3.5 [−1.5, 8.5] p = 0.25 (r30).

All tables are added to the revision (new appendix).

Comments on the paper

1 comment

General response to all reviewers

We thank the three reviewers for careful readings and for the shared recognition that the paper's reframe — treating rank calibration as a controlled intervention and separating the disagreement set D from signed benefit — is well-posed and that the empirical protocol (pre-registration, exact McNemar bounds, nuisance survival tests) is sound. We have uploaded a revision; what changed, and where each point is answered:

New measurements (added as appendices; no claim in the main text relies on them without pointer).

  • Measured cost, and a 2-bit value-cache stress test. We replace the parameter-count proxy for cost with wall-clock and memory: the two ranks take the same time per forward (12.47 s), so §8 now states the break-even by explicit unit and reports no measurable wall-clock/memory crossover. Composing the method on top of 2-bit value quantization (the full KV compressed, not only the key cache), the cross-view signal survives (H_xview +2.34pp, CI [+1.11, +3.64] vs +2.43 at bf16). Answers KMnt Q3.
  • Baselines at a matched per-input byte budget. flat-SVD at 32 positions, full Palu (M-LRD and G-LRD), VL-Cache per-budget, and LongVideoBench are now reported under matched bytes. Answers KMnt (weaknesses) and VQxh.
  • Cross-model identity of D. A direct per-input overlap test across Qwen2-VL and Qwen2.5-VL is now reported, and §2 states the within-configuration repeatability result explicitly, distinct from cross-configuration identity. Answers KMnt Q2.

Clarifications the reviewers requested. The 5–14% figure in §4 is stated with its per-step scope, with the full per-step flip table in the appendix (KMnt Q1); the cost discussion names each crossover by its unit (KMnt Q3); Table 4 is restructured with the rank pair and the two repeatability statistics defined in the caption (VQxh).

On the practical-value question raised by KMnt and pYrg. The gain is for workloads whose repetition is internal to a single ad-hoc query and whose outputs must each be recomputed, so whole-answer caching does not apply: test-time scaling over frame subsets (e.g. Video Self-Play) and option-rotation voting (e.g. CircularEval) re-run the same (video, question) 2–24 times. The profile is measured on the first pass and reused on the rest; unprofiled queries fall back to the task's fixed rank at the same latency. We have now run CircularEval itself (pre-registered): a policy profiled on the original option order and evaluated on the held-out cyclic rotations under CircularEval-strict gains +2.43 pp [+1.30, +3.63] (n = 593, three tasks), and profiling on re-sampled frames gains +4.73 / +1.95 / +2.36 pp (all lower bounds > 0). Details are under KMnt Q3 and pYrg Q1–Q2.

Point-by-point replies follow under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.