acceptodds
Under review as a conference paper at ICLR 2027

DOVE: D-Optimal Visual Evidence for Extreme Video Token Compression

12 views

Abstract

Video large language models process long videos as dense sequences of visual tokens, making multimodal prefilling increasingly expensive. At extreme compression ratios, preserving task-relevant information becomes particularly challenging because decisive visual evidence can be sparse, transient, and distributed across different spatial and temporal contexts. We introduce DOVE (D-Optimal Visual Evidence), a training-free framework for extreme video-token compression. DOVE constructs a quality-weighted evidence geometry that combines visual semantics with spatial, temporal, dynamic, and optional query-dependent cues, and retains a fixed-size subset by maximizing the log-determinant of a ridge-regularized information matrix. Under this criterion, candidates aligned with already well-covered evidence directions provide limited additional volume, whereas reliable candidates spanning weakly covered directions yield larger marginal gains. We optimize the objective efficiently over a budget-proportional candidate pool using exact leverage-based greedy updates and positive-gain local refinement. A fixed-length residual readout recovers compatible discarded information without changing the selected set or output length. Across VideoMME, EgoSchema, LongVideoBench, and MVBench with three Video LLM backbones, DOVE outperforms the strongest baseline's average score by 3.5 points at 1% layer-averaged retention and by 2.9 points at 0.5% percentage. These results demonstrate the effectiveness of quality-weighted D-optimal evidence compression under extremely constrained visual-token budgets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video-Language Models, Visual Token Compression, D-Optimal Design

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_TW46OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DOVE, a training-free video token compression framework targeting sub-1% layer-averaged retention. The method constructs a quality-weighted evidence geometry from multiple visual and contextual cues, applies regularized D-optimal subset selection, and aggregates discarded information through a fixed-length residual readout. Evaluation on four benchmarks and three Video LLM backbones reports consistent gains over four selected training-free baselines in the targeted extreme-compression regime.

Critical strengths

Strong empirical performance: DOVE consistently outperforms the evaluated training-free baselines across three Video LLM backbones and four benchmarks under 0.5–1% token retention.

Critical weaknesses
  1. Narrow and insufficiently motivated scope: The evaluation is restricted to 0.5–1% layer-averaged token retention, but the paper does not establish a realistic use case where such extreme compression, despite substantial information loss, would be preferable to a moderate operating point.

  2. Incomplete validation and insufficient robustness analysis: The comparison omits relevant compression baselines such as DivPrune ( CVPR 2025), DyCoke (CVPR 2025), FLoC (ICLR 2026), STTM (ICCV 2025), LLaVA-PruMerge (ICCV 2025), LLaVA-Scissor (Arxiv 2025), and PruneVID (ACL 2025). It also excludes benchmarks specifically designed for extreme long-video understanding, such as MLVU(CVPR 2025), LVBench (https://lvbench.github.io/) and InfiniBench (EMNLP 2025). Moreover, DOVE introduces numerous handcrafted cues and hyperparameters without sufficient component-wise ablations or sensitivity analyses, leaving its robustness and generality unclear.

Questions for authors

Clarification requests

  1. What practical deployment scenario motivates the 0.5–1% layer-averaged retention regime, and how was this range selected over more moderate compression levels?

  2. How were the numerous cue weights and hyperparameters chosen, and were any evaluated benchmarks used during their development or tuning?

  3. What advantages does DOVE provide over existing diversity-based token selection methods such as DivPrune and FLoC? Direct comparisons would be important to determine whether the proposed D-optimal objective offers meaningful benefits beyond prior diversity- and coverage-aware selection strategies.

  4. The experiments use only 32 or 64 sampled frames. How does DOVE perform when the number of input frames is increased to 128, 256, or denser sampling such as 1 FPS before applying extreme token compression? This evaluation would help determine whether DOVE can effectively preserve sparse evidence from genuinely long videos, rather than only compressing a small set of uniformly sampled frames.

Suggestions for future work: None. These questions concern evidence needed to assess the paper’s current claims.

12 comments
AuthorsOpenReview ↗

Response to Reviewer TW46 [Part 1/5]

We sincerely thank your valuable comment.

Q1, Q4 & W1: Practical Motivation for the 0.5%0.5\%--1%1\% Regime & Scaling to 128128 Sampled Frames

What practical deployment scenario motivates the 0.50.5--1%1\% layer-averaged retention regime, and how was this range selected over more moderate compression levels? The experiments use only 3232 or 6464 sampled frames. How does DOVE perform when the number of input frames is increased to 128128, 256256, or denser sampling such as 11 FPS before applying extreme token compression?

Response:

  1. Practical Deployment Scenario Motivating Extreme (0.5%0.5\%--1%1\%) Retention: In pre-indexed video repositories, cloud video QA, and high-concurrency multi-tenant LLM serving (unlike live raw-pixel edge capture where online ViT encoding dominates), frame features are extracted offline or asynchronously once and cached, making O(N2)\mathcal{O}(N^2) LLM prefill attention and GPU KV-cache memory the primary serving bottleneck under a strict per-request LLM token cap (e.g., ∼60\sim 60--250250 visual tokens). Sparse 3232-frame sampling permanently misses brief inter-frame events before token compression even begins—indeed, even when 3232-frame baselines are relaxed to a 2×2\times gentler budget (R=2%R=2\% in original Table 5), FastV (48.348.3) and FlashVID (49.949.9) still trail 1%1\% DOVE (50.750.7). Extreme 1%1\% compression overcomes this temporal-sampling bottleneck by permitting 4×4\times denser frame input (128128 frames, N=25,088N = 25{,}088 raw visual tokens) while compressing the layer-averaged LLM sequence down to ∼251\sim 251 tokens (∼4%\sim 4\% of a standard 3232-frame raw sequence's 6,2726{,}272 tokens).
  2. New Experiment: Scaling from 3232 to 128128 Sampled Frames at R=1%R = 1\% (Table 1): Following Q4, we scaled LLaVA-OneVision's input from its official 3232-frame setting to 128128 frames (N=25,088N = 25{,}088 raw visual tokens, where uncompressed quadratic LLM prefilling exceeds single-32GB-GPU serving envelopes) compressed by DOVE at R=1%R = 1\% with zero hyperparameter changes:
    • Substantial Gains Across Benchmarks: Scaling from 3232f (1%1\%) to 128128f (1%1\%) boosts VideoMME Overall from 50.750.7 to 54.4\mathbf{54.4} (+3.7+3.7), EgoSchema from 56.8/54.156.8 / 54.1 to 60.0/58.6\mathbf{60.0 / 58.6} (+4.5+4.5 Total), LongVideoBench from 48.448.4 to 50.0\mathbf{50.0} (+1.6+1.6), and MVBench from 49.749.7 to 54.4\mathbf{54.4} (+4.7+4.7). On VMME-Long (46.7→46.246.7 \to 46.2, already retaining 94.5%94.5\%--95.5%95.5\% of 3232f Vanilla's 48.948.9), LLaVA-OneVision's 3232-frame instruction-tuning context prior and DOVE's untuned temporal-bin cap Kt=min⁡(12,T)K_t = \min(12, T) (Table 7, mapping both 3232 and 128128 frames into 1212 coarse temporal intervals) keep long-split scores stable within single-split variance, whereas multi-event and motion-intensive benchmarks (EgoSchema, LongVideoBench, MVBench, and VMME-Short/Medium 55.1→62.955.1 \to 62.9 / 50.3→54.250.3 \to 54.2) improve substantially.
    • Recovering 93.2%93.2\% of Standard 3232-Frame Vanilla Accuracy at 100×100\times Compression: Overall, 128128-frame DOVE (1%1\%) reaches 54.4\mathbf{54.4} Avg.—recovering 93.2%\mathbf{93.2\%} of uncompressed 3232-frame Vanilla (58.458.4) under 100×100\times LLM sequence compression.

Table 1: Frame-Density Scaling from 3232 to 128128 Sampled Frames Under Extreme Compression on LLaVA-OneVision

ConfigurationFramesRRVMME-SVMME-MVMME-LVMME-AllEgo-SubEgo-TotLongVBMVBenchAvg.Rel. vs. 3232f Vanilla (%\%)
Vanilla (Official 3232f)3232100%100\%69.969.956.756.748.948.958.558.562.262.260.360.356.656.658.358.358.458.4100.0100.0
DOVE (200×200\times comp.)32320.5%0.5\%53.453.447.047.043.743.748.048.051.251.249.949.945.745.746.046.047.447.481.281.2
DOVE (100×100\times comp.)32321%1\%55.155.150.350.346.7\mathbf{46.7}50.750.756.856.854.154.148.448.449.749.750.750.786.886.8
DOVE (4×4\times Denser Input)128\mathbf{128}1%\mathbf{1\%}62.9\mathbf{62.9}54.2\mathbf{54.2}46.246.254.4\mathbf{54.4}60.0\mathbf{60.0}58.6\mathbf{58.6}50.0\mathbf{50.0}54.4\mathbf{54.4}54.4\mathbf{54.4}93.2\mathbf{93.2}
AuthorsOpenReview ↗

Response to Reviewer TW46 [Part 2/5]

Q3 & W2: Comparisons with DivPrune, FLOC, PruneVID, Merging Baselines, and Long-Video LVBench

What advantages does DOVE provide over existing diversity-based token selection methods such as DivPrune and FLOC? Direct comparisons would be important to determine whether the proposed D-optimal objective offers meaningful benefits beyond prior diversity- and coverage-aware selection strategies. It also excludes benchmarks specifically designed for extreme long-video understanding, such as MLVU, LVBench, and InfiniBench.

Response:

  1. Direct Evaluations of DivPrune (CVPR 2025) & PruneVID (ACL 2025) (Table 2): Under strict rebuttal compute constraints—having prioritized full five-benchmark runs of DivPrune (CVPR 2025) and PruneVID (ACL 2025), 128128-frame scaling (25,08825{,}088 tokens), hour-long LVBench, noise stress-testing, and 1616-setting sensitivity sweeps—we could not additionally run empirical evaluations for FLOC (ICLR 2026), DyCoke, STTM, LLaVA-PruMerge, LLaVA-Scissor, or MLVU/InfiniBench beyond LVBench, and will thoroughly discuss all of these in revised Section 2 (Related Work, noting that spatiotemporal/connected-component merging and dynamic KV pruning are represented empirically by FlashVID, VisionZip, PruneVID, and FastV). Across all four primary benchmarks (Table 2), DOVE beats DivPrune on every dataset at R=1%R=1\% (50.750.7 vs. 48.748.7 Avg., +2.0+2.0 points) and leads at R=0.5%R=0.5\% (47.447.4 vs. 46.546.5), while outperforming PruneVID by +1.5+1.5 points at 1%1\% (50.750.7 vs. 49.249.2) and +2.6+2.6 points at 0.5%0.5\% (47.447.4 vs. 44.844.8).
  2. Why Regularized D-Optimality Outperforms Pairwise Dispersion (DivPrune) & Exemplar Coverage (FLOC) (Table 3): Pairwise max-min dispersion (DivPrune) and submodular facility-location / exemplar-coverage objectives (FLOC) rely on pairwise similarities: facility-location saturates once nearby tokens share a representative, while unweighted max-min dispersion treats noisy outliers as isolated clusters. Crucially, pairwise metrics cannot detect higher-order linear subspace collinearity—multiple tokens lying in an already-spanned 2D/3D subspace remain pairwise separated yet contribute zero new rank for linear VLLM value projections. DOVE's ridge-regularized log⁡det⁡(λI+∑i∈Swieiei⊤)\log \det(\lambda I + \sum_{i \in \mathcal{S}} w_i e_i e_i^\top) explicitly maximizes quality-weighted joint subspace volume. Even under the exact same DOVE pool P\mathcal{P}, geometry, and readout (Table 3), pairwise/metric-coverage selectors (Farthest-first 49.749.7, FPS 49.949.9) and unregularized k-DPP MAP (49.349.3) consistently trail regularized D-optimality (50.7\mathbf{50.7}).
  3. Evaluation on Hour-Long LVBench (Table 4): On LVBench (hour-long videos demanding multi-event retrieval), DOVE ranks first across all 77 compressed methods at both R=1%R = 1\% (33.25\mathbf{33.25}, +1.68+1.68 over PruneVID and +3.04+3.04 over DivPrune) and R=0.5%R = 0.5\% (30.41\mathbf{30.41}).
AuthorsOpenReview ↗

Response to Reviewer TW46 [Part 3/5]

Table 2: Direct Comparison with DivPrune, PruneVID, and Prior Baselines on LLaVA-OneVision (3232 Frames)

MethodRRVMME-SVMME-MVMME-LVMME-AllEgo-SubEgo-TotLongVBMVBenchAvg.Rel. (%\%)
Vanilla100%100\%69.969.956.756.748.948.958.558.562.262.260.360.356.656.658.358.358.458.4100.0100.0
FastV$^\dagger$1%1\%49.649.647.147.141.941.946.246.246.446.447.747.745.545.548.548.547.047.080.580.5
VisionZip1%1\%45.145.144.044.043.043.044.044.045.245.244.144.143.743.740.440.443.143.173.873.8
FastVID1%1\%46.346.345.445.443.243.245.045.044.244.244.444.445.245.239.939.943.643.674.774.7
FlashVID1%1\%49.149.147.147.144.344.346.846.852.852.849.649.646.146.143.743.746.646.679.879.8
PruneVID (ACL'25)1%1\%57.0\mathbf{57.0}49.949.943.743.750.250.252.652.650.250.247.447.448.948.949.249.284.284.2
DivPrune (CVPR'25)1%1\%54.854.847.347.344.844.849.049.052.652.650.250.247.247.248.348.348.748.783.483.4
DOVE (Ours)1%1\%55.155.150.3\mathbf{50.3}46.7\mathbf{46.7}50.7\mathbf{50.7}56.8\mathbf{56.8}54.1\mathbf{54.1}48.4\mathbf{48.4}49.7\mathbf{49.7}50.7\mathbf{50.7}86.8\mathbf{86.8}
FastV$^\dagger$0.5%0.5\%45.845.846.246.241.941.944.644.645.645.645.345.345.145.145.945.945.245.277.477.4
VisionZip0.5%0.5\%43.143.143.943.940.940.942.642.641.041.041.441.442.742.739.739.741.641.671.271.2
FastVID0.5%0.5\%45.445.443.943.941.941.943.743.740.840.842.542.543.743.739.539.542.442.472.672.6
FlashVID0.5%0.5\%43.243.242.642.641.641.642.542.540.040.041.441.443.243.239.339.341.641.671.271.2
PruneVID (ACL'25)0.5%0.5\%48.848.845.845.841.041.045.245.246.046.045.045.045.045.044.244.244.844.876.776.7
DivPrune (CVPR'25)0.5%0.5\%51.851.844.944.944.1\mathbf{44.1}46.946.948.448.447.347.345.645.646.4\mathbf{46.4}46.546.579.679.6
DOVE (Ours)0.5%0.5\%53.4\mathbf{53.4}47.0\mathbf{47.0}43.743.748.0\mathbf{48.0}51.2\mathbf{51.2}49.9\mathbf{49.9}45.7\mathbf{45.7}46.046.047.4\mathbf{47.4}81.2\mathbf{81.2}

†^\daggerFastV retains 100%100\% visual tokens in Layers 11--22 (∼8.1%\sim 8.1\% / 7.6%7.6\% layer-averaged visual tokens).

Table 3: Controlled Comparison Against Diversity & Coverage Objectives Under Identical Pool P\mathcal{P} & Geometry (R=1%R=1\%)

Selection Objective (Shared P\mathcal{P}, Geometry & Readout)VideoMMEEgoSchemaLongVideoBenchMVBenchAvg. ScoreRel. Acc. (%\%)
kk-DPP MAP (Unregularized determinant)49.349.351.451.447.147.149.449.449.349.384.484.4
Farthest-first (Minimax metric coverage)49.849.853.053.046.446.449.649.649.749.785.185.1
FPS (Iterative point coverage)49.149.153.953.947.247.249.549.549.949.985.485.4
Regularized D-optimal (DOVE, Ours)50.7\mathbf{50.7}54.1\mathbf{54.1}48.4\mathbf{48.4}49.7\mathbf{49.7}50.7\mathbf{50.7}86.8\mathbf{86.8}

Table 4: Evaluation on Hour-Long Video Benchmark LVBench (LLaVA-OneVision, 3232 Frames)

Retention RRVanilla (100%100\%)FastV$^\dagger$VisionZipFastVIDFlashVIDPruneVIDDivPruneDOVE (Ours)
R=1%R = 1\%39.2539.2530.5430.5428.0228.0230.0230.0229.5029.5031.5731.5730.2130.2133.25\mathbf{33.25}
R=0.5%R = 0.5\%39.2539.2528.9228.9227.5727.5729.0529.0527.3727.3728.9928.9929.5729.5730.41\mathbf{30.41}
Reviewer_cPA9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the extreme compression of video tokens in Video LLMs. The authors propose DOVE, a training-free framework that combines visual and contextual cues to select a small set of complementary tokens using a D-optimal objective. A residual readout merges information from discarded tokens into the selected ones. Across three Video LLMs and four benchmarks, DOVE improves accuracy over existing compression methods at 1% and 0.5% token retention.

Critical strengths
  1. The paper addresses a meaningful practical problem. At extreme compression, selecting individually tokens can overlook complementary evidence for answering a given question.

  2. The reported gains are consistent across different models and budgets. Tables 1–2 show the best average score in all models and budgets settings.

Critical weaknesses
  1. Though the overall performance of DOVE is better than previous methods, the whole frameworks introduce too many hyperparameters (as shown in Table 7). The ablation studies of most of those hyperparameters are not provided (e.g., Candidate multiplier, Whitening strength, etc). It is hard to generalize DOVE to other Video LLM and other tasks with so many hyperparameters to tune.

  2. PruneVid (ACL 2025), SparseVLM (ICML 2025) and MixKV (ICLR2026) are important baselines, however, they are not compared in experiments.

  3. Table 5 compares DOVE at 1% retention against FastV and FlashVID at 2% retention, which is confusing. Latency and memory should be compared at an identical 1% budget.

  4. From Table 3 we can see that, Mass Top-K already outperforms FastV, suggesting that components beyond the D-optimal objective contribute meaningfully to DOVE’s performance gain. Although D-optimal adds another 2.3 points, the paper should more clearly isolate the contributions of handcrafted scoring, candidate filtering, and residual readout.

Questions for authors
  1. DOVE use prompt to guide compression, how to adapt this method to multi-turn conversation without prefill twice?
  2. How are query embeddings mapped into the same 96-dimensional space as visual features?
  3. How compression overhead scales with input length?
7 comments
AuthorsOpenReview ↗

Response to Reviewer cPA9 [Part 1/5]

We sincerely thank your insightful comment.

W1: Hyperparameter Count, Generalization Without Tuning, and Sensitivity Ablations

Though the overall performance of DOVE is better than previous methods, the whole framework introduces too many hyperparameters (as shown in Table 7). The ablation studies of most of those hyperparameters are not provided (e.g., Candidate multiplier, Whitening strength, etc). It is hard to generalize DOVE to other Video LLM and other tasks with so many hyperparameters to tune.

Response: We address this concern from three angles:

  1. Reproducibility Documentation vs. Task-Specific Knobs: Multi-stage training-free video compressors (such as density-peak or tree-merging pipelines like FastVID and FlashVID) naturally contain internal stage thresholds and mixing weights; we explicitly listed every internal constant in Table 7 strictly for complete reproducibility, not as user-tuned knobs.
  2. Zero Per-Model or Per-Task Tuning: Applying DOVE to a new Video LLM or benchmark requires zero tuning: the exact same frozen configuration in Table 7 is shared unchanged across all three heterogeneous VLLMs (LLaVA-OneVision 3232f, LLaVA-Video 6464f, and Qwen3-VL 3232f with dynamic resolution and 3D M-RoPE), 128128-frame scaling (54.454.4 Avg.), and all five benchmarks (including newly evaluated hour-long LVBench, where DOVE ranks first at 33.2533.25 vs. 31.5731.57 PruneVID / 30.5430.54 FastV at 1%1\%, and beats PruneVID by +3.9+3.9 on EgoSchema Total).
  3. 1616-Setting Sensitivity Analysis (Table 1): Perturbing 88 core constants across wide (0.5×0.5\times to 2×2\times) shifts—including Candidate multiplier μ\mu, Whitening strength η\eta, Ridge λ\lambda, Base readout strength κ0\kappa_0, Metric dimension dud_u, Mutual-match threshold, Structural/query coordinate mass, and Max query atoms KqK_q—keeps 1414 of 1616 runs within [49.56,50.37][49.56, 50.37], well within the ∼1σ\sim 1\sigma statistical fluctuation band (≈±0.9\approx \pm 0.9 points on 2,7002{,}700-question VideoMME) of default 50.7050.70. Only two extreme boundary shifts drop noticeably for clear mechanistic reasons: μ=1.5\mu = 1.5 (48.5248.52, where restricting P\mathcal{P} to 1.5B1.5 B under-covers candidate directions before log⁡det⁡\log \det selection, whereas μ=4.0\mu = 4.0 remains stable at 50.2250.22) and Kq=16K_q = 16 (49.0849.08, where doubling max query atoms on short QA prompts dilutes visual subspace coordinates under fixed λ=0.50\lambda = 0.50, whereas Kq=4K_q = 4 remains at 50.0750.07).

Table 1: Hyperparameter Sensitivity Analysis on LLaVA-OneVision (VideoMME, R=1%R=1\%, Default =50.70= 50.70)

HyperparameterDefaultPerturbed ValueShortMediumLongOverall
Candidate multiplier μ\mu2.52.51.51.554.5654.5647.2247.2243.7843.7848.5248.52
Candidate multiplier μ\mu2.52.54.04.054.3354.3350.1150.1146.2246.2250.2250.22
Whitening strength η\eta0.500.500.250.2554.3354.3349.4449.4445.7845.7849.8549.85
Whitening strength η\eta0.500.500.750.7555.3355.3349.4449.4446.1146.1150.2950.29
Ridge parameter λ\lambda0.500.500.100.1054.6754.6749.3349.3346.3346.3350.1150.11
Ridge parameter λ\lambda0.500.501.001.0054.8954.8949.5649.5646.4446.4450.3050.30
Base readout strength κ0\kappa_0 (orig. αj\alpha_j, Table 7)0.120.120.060.0654.1154.1149.7849.7846.1146.1150.0050.00
Base readout strength κ0\kappa_0 (orig. αj\alpha_j, Table 7)0.120.120.240.2454.5654.5649.5649.5645.8945.8950.0050.00
Metric dimension dud_u9696646454.4454.4450.0050.0046.6746.6750.3750.37
Metric dimension dud_u969612812855.6755.6749.8949.8944.8944.8950.1550.15
Mutual-match threshold0.820.820.750.7554.2254.2248.7848.7846.2246.2249.7449.74
Mutual-match threshold0.820.820.900.9054.8954.8949.4449.4445.8945.8950.0750.07
Structural/query coord. mass0.320.320.160.1654.0054.0048.7848.7845.8945.8949.5649.56
Structural/query coord. mass0.320.320.480.4854.8954.8948.5648.5646.4446.4449.9649.96
Maximum query atoms KqK_q884455.0055.0049.7849.7845.4445.4450.0750.07
Maximum query atoms KqK_q88161654.5654.5649.0049.0043.6743.6749.0849.08
AuthorsOpenReview ↗

Response to Reviewer cPA9 [Part 2/5]

W2: Comparison with Recent Baselines (PruneVID, DivPrune, SparseVLM, MixKV)

PruneVid (ACL 2025), SparseVLM (ICML 2025) and MixKV (ICLR 2026) are important baselines, however, they are not compared in experiments.

Response: Within rebuttal compute constraints, we conducted full benchmark evaluations of two recent video/diversity compression methods—PruneVID (ACL 2025) and DivPrune (CVPR 2025)—across all four primary benchmarks (Table 2) and on hour-long LVBench, while expanding our architectural comparison with SparseVLM and MixKV in Related Work:

  • Consistent Gains Over PruneVID & DivPrune: Across the four benchmarks (Table 2), DOVE outperforms PruneVID by +1.5+1.5 Avg. points (50.750.7 vs. 49.249.2, +2.6%+2.6\% Rel. Acc.) at R=1%R=1\% and +2.6+2.6 points (47.447.4 vs. 44.844.8, +4.5%+4.5\% Rel. Acc.) at R=0.5%R=0.5\%, while also surpassing DivPrune at both budgets (50.750.7 vs. 48.748.7 at 1%1\%; 47.447.4 vs. 46.546.5 at 0.5%0.5\%). On hour-long LVBench, DOVE likewise leads all methods at 1%1\% (33.2533.25 vs. 31.5731.57 PruneVID / 30.2130.21 DivPrune) and 0.5%0.5\% (30.4130.41 vs. 28.9928.99 / 29.5729.57).
  • Architectural Relationship to SparseVLM & MixKV: SparseVLM (ICML 2025) is an image-VLM progressive inner-LLM text-visual pruner sharing the unpruned-early-layer paradigm represented in our tables by FastV (which incurs a 3.93.9-TFLOPs / 8.1%8.1\% KV floor from processing 100%100\% visual tokens in early LLM layers); for Video LLMs, PruneVID directly extends text-guided pruning/merging with temporal video clustering at the 0.40.4-TFLOPs pre-LLM budget, serving as the stronger video-native counterpart. Meanwhile, MixKV (ICLR 2026) evicts head-wise KV-cache entries after full-sequence prefill projections to save decoding KV memory rather than prefill FLOPs/TTFT, making it orthogonal and composable with DOVE.

Table 2: Main Benchmark Comparison with PruneVID and DivPrune on LLaVA-OneVision (3232 Frames)

MethodRRVMME-SVMME-MVMME-LVMME-AllEgo-SubEgo-TotLongVBMVBenchAvg.Rel. (%\%)
Vanilla100%100\%69.969.956.756.748.948.958.558.562.262.260.360.356.656.658.358.358.458.4100.0100.0
FastV$^\dagger$1%1\%49.649.647.147.141.941.946.246.246.446.447.747.745.545.548.548.547.047.080.580.5
VisionZip1%1\%45.145.144.044.043.043.044.044.045.245.244.144.143.743.740.440.443.143.173.873.8
FastVID1%1\%46.346.345.445.443.243.245.045.044.244.244.444.445.245.239.939.943.643.674.774.7
FlashVID1%1\%49.149.147.147.144.344.346.846.852.852.849.649.646.146.143.743.746.646.679.879.8
PruneVID (ACL'25)1%1\%57.0\mathbf{57.0}49.949.943.743.750.250.252.652.650.250.247.447.448.948.949.249.284.284.2
DivPrune (CVPR'25)1%1\%54.854.847.347.344.844.849.049.052.652.650.250.247.247.248.348.348.748.783.483.4
DOVE (Ours)1%1\%55.155.150.3\mathbf{50.3}46.7\mathbf{46.7}50.7\mathbf{50.7}56.8\mathbf{56.8}54.1\mathbf{54.1}48.4\mathbf{48.4}49.7\mathbf{49.7}50.7\mathbf{50.7}86.8\mathbf{86.8}
FastV$^\dagger$0.5%0.5\%45.845.846.246.241.941.944.644.645.645.645.345.345.145.145.945.945.245.277.477.4
VisionZip0.5%0.5\%43.143.143.943.940.940.942.642.641.041.041.441.442.742.739.739.741.641.671.271.2
FastVID0.5%0.5\%45.445.443.943.941.941.943.743.740.840.842.542.543.743.739.539.542.442.472.672.6
FlashVID0.5%0.5\%43.243.242.642.641.641.642.542.540.040.041.441.443.243.239.339.341.641.671.271.2
PruneVID (ACL'25)0.5%0.5\%48.848.845.845.841.041.045.245.246.046.045.045.045.045.044.244.244.844.876.776.7
DivPrune (CVPR'25)0.5%0.5\%51.851.844.944.944.1\mathbf{44.1}46.946.948.448.447.347.345.645.646.4\mathbf{46.4}46.546.579.679.6
DOVE (Ours)0.5%0.5\%53.4\mathbf{53.4}47.0\mathbf{47.0}43.743.748.0\mathbf{48.0}51.2\mathbf{51.2}49.9\mathbf{49.9}45.7\mathbf{45.7}46.046.047.4\mathbf{47.4}81.2\mathbf{81.2}

†^\daggerFastV retains 100%100\% visual tokens in Layers 11--22 (∼8.1%\sim 8.1\% / 7.6%7.6\% layer-averaged visual tokens).

AuthorsOpenReview ↗

Response to Reviewer cPA9 [Part 3/5]

W3: Latency and Memory Comparison at Identical 1%1\% Budget in Table 5

Table 5 compares DOVE at 1%1\% retention against FastV and FlashVID at 2%2\% retention, which is confusing. Latency and memory should be compared at an identical 1%1\% budget.

Response: Original Table 5 included 2%2\% baselines solely to show that 1%1\% DOVE surpasses baselines given 2×2\times the token budget. Following your suggestion, we re-profiled latency and dynamic visual KV-cache memory across all methods at the nominal 1%1\% setting in a single unified session on 6060 LVBench questions (Table 3, also fixing Table 5's header alignment):

  • Memory Footprint: Because all training-free methods share the same frozen model parameters and Vision Encoder activation peak, dynamic visual KV-cache memory scales strictly linearly with the layer-averaged visual token retention (Vis. KV (%)). DOVE reduces dynamic visual KV storage by 100×100\times relative to Vanilla (1.0%1.0\% vs. 100.0%100.0\%)—matching pre-LLM methods (VisionZip, FastVID, FlashVID, PruneVID, DivPrune) and requiring >8.1×>8.1\times less visual KV memory than FastV (8.1%8.1\%), whose nominal 1%1\% retains 100%100\% of visual tokens in Layers 11--22.
  • Latency vs. Accuracy Trade-Off at 1%1\%: DOVE is faster and more accurate than 1%1\% FastV (55.255.2 vs. 68.168.1 ms prefill; 316.0316.0 vs. 328.5328.5 ms TTFT; +3.7+3.7 Avg.). Relative to the profiled pre-LLM heuristics (VisionZip, FastVID, FlashVID) at an identical 1.0%1.0\% visual KV footprint and 0.40.4 LLM TFLOPs, DOVE adds only 14.014.0--21.321.3 ms in TTFT (<7%<7\% of TTFT; 2.4×2.4\times vs. 2.5×2.5\times--2.6×2.6\times speedup over Vanilla's 754.3754.3 ms) while gaining +4.1+4.1 over FlashVID, +7.1+7.1 over FastVID, and +7.6+7.6 over VisionZip (and outperforming PruneVID / DivPrune by +1.5+1.5 / +2.0+2.0 Avg. points in the same 0.40.4-TFLOPs / 1.0%1.0\%-KV regime).

Table 3: Equal-Budget (1%1\% Retention) Latency, Dynamic Visual KV Memory & Accuracy on LLaVA-OneVision

MethodRRTFLOPsVis. KV (%\%)Vision Enc. (ms)Comp. (ms)LLM Fwd (ms)Prefill Total (ms)TTFT (ms)Avg.Rel. (%\%)
Vanilla100%100\%48.848.8100.0%100.0\%260.3260.3--494.1494.1494.1 (1.0×)494.1\ (1.0\times)754.3 (1.0×)754.3\ (1.0\times)58.458.4100.0100.0
FastV1%†1\%^\dagger3.93.98.1%8.1\%260.5260.50.30.367.867.868.1 (7.3×)68.1\ (7.3\times)328.5 (2.3×)328.5\ (2.3\times)47.047.080.580.5
VisionZip1%1\%0.40.41.0%1.0\%260.5260.54.84.829.429.434.2 (14.4×)34.2\ (14.4\times)294.7 (2.6×)294.7\ (2.6\times)43.143.173.873.8
FastVID1%1\%0.40.41.0%1.0\%263.8263.81.71.729.229.231.0 (16.0×)31.0\ (16.0\times)294.7 (2.6×)294.7\ (2.6\times)43.643.674.774.7
FlashVID1%1\%0.40.41.0%1.0\%260.5260.59.49.432.232.241.5 (11.9×)41.5\ (11.9\times)302.0 (2.5×)302.0\ (2.5\times)46.646.679.879.8
DOVE (Ours)1%\mathbf{1\%}0.4\mathbf{0.4}1.0%‡\mathbf{1.0\%}^\ddagger260.7\mathbf{260.7}23.4\mathbf{23.4}31.9\mathbf{31.9}55.2 (8.9×)\mathbf{55.2\ (8.9\times)}316.0 (2.4×)\mathbf{316.0\ (2.4\times)}50.7\mathbf{50.7}86.8\mathbf{86.8}

†^\daggerFastV retains 100%100\% tokens in Layers 11--22 (3.93.9 TFLOPs). ‡^\ddaggerLayer-averaged across 20+820+8 layers: (20×1.30%+8×0.25%)/28=1.0%(20 \times 1.30\% + 8 \times 0.25\%)/28 = 1.0\%. Latencies were re-profiled uniformly on 6060 LVBench questions (Avg. cites four-benchmark score; minor ∼1\sim 1--22 ms differences vs. VideoMME Table 5 reflect LVBench's longer text prompts).

Reviewer_dvghOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DOVE, a training-free video token compressor which embeds tokens into a evidence geometry. It selects a subset by log determinant maximization and adds a residual readout, reporting gains over prior methods on multiple benchmarks and backbones.

Critical strengths

Proposed method is training-free, the authors claim consistent gains across backbones, benchmarks, and budgets. The authors also scope their claims honestly where the guarantee is stated only relative to the fixed candidate pool.

Critical weaknesses
  1. Atribution of gains is unclear. Mass top-k already reaches 82.9%, above every baseline. Therefore it is natual to infer that a substantial share of the gain comes from the hand-crafted score, the candidate pool, or hybrid schedule, not from D-optimality. Ablations cover only position coordinates and the readout. Query cues, dynamic cues, whitening, exchange refinement and the hybrid schedule are not ablated.

  2. The comparison is not fair or complete enough. Baselines are pushed outside their design regim and no results are given at conventional budgets. Layer-averaged retention is not compute-equivalent, at the same 2% in Table 5, FastV costs 3.9 TFLOPs and FlashVID 0.8. The efficiency table also compares DOVE at 1% against others at 2%. Roughly 30 hand-set constants are claimed to involve no tuning, but their origin is not explained and no sensitivity analysis is given, I recommend such an addition.

  3. Statistical and reporting rigor is insufficient. No variance or confidence intervals, I suggest to check the conclusions on coordinates and readout against noise. Also the numbers of FastVID and VisionZip under EcoSchema are identical, which is suspicious. Math notations are overloaded in this paper.

Please address above problems for a reconsideration of a score raising.

Questions for authors

Questions are addressed in weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer dvgh [Part 1/5]

We sincerely thank your constructive comment.

W1: Attribution of Gains, Component Ablations, and the 20+820+8 Hybrid Schedule

Attribution of gains is unclear. Mass top-k already reaches 82.9%82.9\%, above every baseline. Therefore it is natural to infer that a substantial share of the gain comes from the hand-crafted score, the candidate pool, or hybrid schedule, not from D-optimality. Ablations cover only position coordinates and the readout. Query cues, dynamic cues, whitening, exchange refinement and the hybrid schedule are not ablated.

Response: We isolate every factor through stepwise and spectral-interaction diagnostics (Table 1), same-schedule (20+820+8) four-benchmark ablations (Table 2), comparison with recent flat pre-LLM baselines (Table 3), equal-compute/KV profiling (Table 4), and calibrated Full-DOVE perturbations (Table 5):

  1. Compute/KV Equivalence & Controlled Attribution of the 20+820+8 Hybrid Schedule and Mass Score:
    • Exact Single-Stage DOVE Execution: DOVE executes evidence geometry construction, log⁡det⁡\log \det optimization, and residual readout only once, prior to LLM Layer 1 (1.30%1.30\% initial anchors on 2828-layer LLaVA; 1.25%1.25\% on 3636-layer Qwen3-VL). At Layer 20 (Layer 28 on Qwen3-VL), a lightweight parameter-free filter retains the top 0.19230.1923 (0.100.10) fraction of surviving visual anchors ranked by cross-head-averaged text-to-visual self-attention from that same forward pass—without re-running DOVE geometry, determinant updates, or readout (we will state this explicitly in Sec. 3.1 and clarify App. 9 line 805).
    • Strict Compute & KV-Cache Parity: In uniform decoder-only transformer blocks, per-layer visual prefill FLOPs and visual KV-cache storage sum linearly across depth. Allocating (20×1.30%+8×0.25%)/28=1.00%(20 \times 1.30\% + 8 \times 0.25\%)/28 = 1.00\% simply shifts a fixed layer-summed budget from deep text-refinement layers to early/mid multimodal fusion layers, yielding the exact same 0.40.4 LLM TFLOPs, virtually identical LLM Forward latency (31.931.9 ms vs. 29.229.2--32.232.2 ms for flat 1%1\% pre-LLM methods), and identical 1.0%1.0\% layer-averaged visual KV footprint (Table 4).
    • Why Neither 20+820+8 Nor Handcrafted Mass Explains DOVE's Lead: Crucially, Mass Top-K (48.448.4 Avg., 82.9%82.9\%) already incorporates both our handcrafted mass score wiw_i and the exact same 20+820+8 schedule, yet in Table 3 it underperforms recent flat (1.0×1.0\times across all 2828 layers) pre-LLM baselines PruneVID (49.249.2 Avg., 84.2%84.2\%) and DivPrune (48.748.7 Avg., 83.4%83.4\%). Likewise, alternative subset selectors sharing DOVE's exact geometry, readout, and 20+820+8 schedule (k-DPP MAP 49.349.3, Farthest-first 49.749.7, FPS 49.949.9, Table 2) all fall clearly short of DOVE (50.7\mathbf{50.7} Avg., +2.3+2.3 over Mass Top-K).
  2. Candidate Pool Filtering Gives Zero Standalone Gain: Restricting scalar ranking to P\mathcal{P} (Mass + candidate pool, Table 1 Row 2: 48.748.7) yields zero improvement over global Mass Top-K (Row 1: 48.848.8); P\mathcal{P} strictly bounds search complexity.
  3. Isolating Visual D-Optimality, Whitening, Dynamic Cues, Query, Exchanges, Position & Readout:
    • Core Stepwise Gain (Table 1 Part A): Holding scalar mass wiw_i, pool P\mathcal{P}, and the 20+820+8 schedule fixed to Row 2 (48.748.7), switching to D-optimality over raw pooled visual directions ei=uie_i = u_i alone (Row 3, η=0\eta=0—removing all position, dynamic, and query directional coordinates, whitening, exchanges, and readout) immediately jumps VideoMME to 50.0\mathbf{50.0} (+1.3+1.3), and adding shrinkage whitening (η=0.50\eta=0.50, Row 4) lifts it to 50.3\mathbf{50.3} (+1.6+1.6; weakening η\eta from 0.50→0.250.50 \to 0.25 on Full DOVE drops 50.70→49.8550.70 \to 49.85 in Table 5).
    • Why Whitening (η=0.50\eta=0.50) Is a Prerequisite for Auxiliary Coordinates & Fedorov Exchanges (Table 1 Part B vs. Table 2 & Table 5): On unwhitened Row 3 (η=0\eta=0, Rows 6--9), a few high-variance background channels dominate MSM_{\mathcal{S}}: dynamic cues cic_i (Row 6: 50.350.3) lift Long-video accuracy (44.8→45.344.8 \to 45.3), query atoms rir_i (Row 7: 49.949.9) lift Short/Medium (55.9/50.455.9 / 50.4), and spatiotemporal bins (Row 9: 49.149.1) lift Short (56.856.8), whereas local Fedorov exchanges on an uncalibrated spectrum (Row 8: 49.749.7 at 22 sweeps / 49.549.5 at 66 sweeps) greedily trade semantic coverage for high-variance background axes. Once eigenvalues are balanced via shrinkage whitening (η=0.50\eta=0.50) in DOVE (Full) (Row 5: 50.7\mathbf{50.7} Overall, 46.7\mathbf{46.7} Long), this behavior reverses: spatiotemporal coordinates improve VideoMME from 50.0→50.750.0 \to 50.7 (+0.7+0.7, Table 2), residual readout adds +0.5+0.5 four-benchmark Avg. (50.2→50.750.2 \to 50.7, +1.0+1.0 LongVideoBench), and halving calibrated structural/query coordinate mass (0.32→0.160.32 \to 0.16) or reducing query atoms (Kq:8→4K_q: 8 \to 4) lowers Full DOVE to 49.5649.56 and 50.0750.07 (Table 5).
AuthorsOpenReview ↗

Response to Reviewer dvgh [Part 2/5]

Table 1: Stepwise Attribution & Spectral-Calibration Diagnostics on LLaVA-OneVision (VideoMME, R=1%R=1\%)

Variant / ConfigurationShortMediumLongOverallMechanistic Takeaway
Part A: Calibrated Stepwise Progression
1.1. Mass Top-K (Scalar mass wiw_i + 20+820+8 schedule)52.652.648.348.345.645.648.848.8Scalar ranking without volume maximization
2.2. Mass + candidate pool P\mathcal{P}51.951.948.248.246.046.048.748.7Pool filtering alone gives zero gain (−0.1-0.1)
3.3. Visual-direction D-optimal (on P\mathcal{P}, ei=uie_i = u_i, η=0\eta=0)55.455.449.949.944.844.850.0\mathbf{50.0}Raw visual D-optimality adds +1.3+1.3 over Row 2
4.4. Row 3 ++ shrinkage whitening (η=0.50\eta=0.50)56.456.449.449.445.145.150.3\mathbf{50.3}Spectrum calibration adds +1.6+1.6 over Row 2
5.5. DOVE (Full: Calibrated Geometry + Readout)55.155.150.350.346.7\mathbf{46.7}50.7\mathbf{50.7}Best Overall (50.750.7) & Long (46.746.7, +1.6+1.6 over Row 4)
Part B: Sub-Split Diagnostics on Unwhitened Row 3 (η=0\eta=0)(Shows why whitening η=0.50\eta=0.50 is required first)
6.6. Row 3 ++ dynamic context cues cic_i (η=0\eta=0)55.755.749.949.945.345.350.350.3Lifts Long-video event coverage (44.8→45.344.8 \to 45.3)
7.7. Row 3 ++ query atoms rir_i (η=0\eta=0)55.955.950.4\mathbf{50.4}43.443.449.949.9Lifts Short/Medium (+0.5+0.5); needs whitened Long axes
8.8. Row 3 ++ Fedorov exchange, 22 / 66 sweeps (η=0\eta=0)55.9/55.755.9 / 55.748.2/48.648.2 / 48.645.1/44.345.1 / 44.349.7/49.549.7 / 49.5Swaps chase high-variance background axes if η=0\eta=0
9.9. Row 3 ++ temporal/spatial bins (η=0\eta=0)56.8\mathbf{56.8}47.647.642.942.949.149.1Lifts Short (+1.4+1.4); adds +0.7+0.7 VMME once whitened

Table 2: Same-Schedule (20+820+8) Leave-One-Out & Objective Attribution Across Four Benchmarks (R=1%R=1\%)

Variant (All Under Identical 20+820+8 Schedule)VideoMMEEgoSchemaLongVideoBenchMVBenchAvg. ScoreRel. Acc. (%\%)
Mass Top-K (Scalar mass only, w/o D-optimal)48.848.851.951.946.046.046.946.948.448.482.982.9
kk-DPP MAP (Shared P\mathcal{P}, geometry & readout)49.349.351.451.447.147.149.449.449.349.384.484.4
Farthest-first (Shared P\mathcal{P}, geometry & readout)49.849.853.053.046.446.449.649.649.749.785.185.1
FPS (Shared P\mathcal{P}, geometry & readout)49.149.153.953.947.247.249.549.549.949.985.485.4
DOVE w/o residual readout50.150.154.054.047.447.449.449.450.250.286.086.0
DOVE w/o position coordinates50.050.054.2\mathbf{54.2}48.148.149.7\mathbf{49.7}50.550.586.586.5
DOVE (Full, Regularized D-optimal)50.7\mathbf{50.7}54.154.148.4\mathbf{48.4}49.7\mathbf{49.7}50.7\mathbf{50.7}86.8\mathbf{86.8}
AuthorsOpenReview ↗

Response to Reviewer dvgh [Part 3/5]

W2: Fair Baselines, Operating Budgets, Equal-Budget (1%1\%) Efficiency, and Sensitivity Study

The comparison is not fair or complete enough. Baselines are pushed outside their design regime and no results are given at conventional budgets. Layer-averaged retention is not compute-equivalent, at the same 2%2\% in Table 5, FastV costs 3.93.9 TFLOPs and FlashVID 0.80.8. The efficiency table also compares DOVE at 1%1\% against others at 2%2\%. Roughly 3030 hand-set constants are claimed to involve no tuning, but their origin is not explained and no sensitivity analysis is given, I recommend such an addition.

Response:

  1. Extreme-Pruning Baselines, Scope vs. Conventional Budgets & Larger Token Caps (Table 3): At gentler budgets (10%10\%--25%25\%), existing methods cluster near Vanilla accuracy with limited differentiation, whereas long-video serving under strict KV/latency caps requires aggressive (≤2%\le 2\%) compression where univariate rankers break down. For fair comparison in this regime, we added two strong recent baselines explicitly designed for aggressive reduction—PruneVID (ACL 2025) and DivPrune (CVPR 2025)—across all four benchmarks (Table 3, plus hour-long LVBench where DOVE leads at 33.2533.25 vs. 31.5731.57 / 30.2130.21 at 1%1\%). Moreover, original Table 5 showed that at R=2%R=2\%, FastV (48.348.3) and FlashVID (49.949.9) still trail 1%1\% DOVE (50.750.7); scaling the retained LLM token cap 4×4\times (∼251\sim 251 tokens) via 128128 sampled frames at R=1%R=1\% further lifts DOVE to 54.4\mathbf{54.4} Avg., recovering 93.2%\mathbf{93.2\%} of 3232-frame Vanilla (58.458.4).
  2. Equal-Budget (1%1\%) Latency, Visual KV Storage & TFLOPs (Table 4): FastV's nominal 1%1\% denotes post-pruning retention—its official design processes 100%100\% of visual tokens in LLM Layers 11--22 before pruning Layers 33--2828 to 1%1\%, incurring 3.93.9 TFLOPs and (2×100%+26×1%)/28≈8.1%(2\times 100\% + 26\times 1\%)/28 \approx 8.1\% layer-averaged visual KV storage (7.6%7.6\% at nominal 0.5%0.5\%). By contrast, DOVE and pre-LLM methods (VisionZip, FastVID, FlashVID, PruneVID, DivPrune) all maintain 1.0%1.0\% layer-averaged visual KV storage and 0.40.4 rounded LLM TFLOPs. Re-profiled uniformly in a single session on 6060 LVBench questions (Table 4), 1%1\% DOVE achieves lower TTFT than 1%1\% FastV (316.0316.0 vs. 328.5328.5 ms, +3.7+3.7 Avg.) and adds only 14.014.0--21.321.3 ms in TTFT relative to the profiled pre-LLM heuristics (VisionZip, FastVID, FlashVID) while gaining +4.1+4.1 to +7.6+7.6 Avg. points (and outperforming PruneVID / DivPrune by +1.5+1.5 / +2.0+2.0 Avg. points in the same 0.40.4-TFLOPs / 1.0%1.0\%-KV class).
  3. Origin of Constants & 1616-Setting Sensitivity Analysis (Table 5): Multi-stage training-free pipelines (including FastVID and FlashVID) naturally contain internal stage constants, which we tabulated exhaustively in Table 7 for reproducibility. They combine standard mathematical midpoints/defaults (η=0.50\eta = 0.50 whitening, λ=0.50\lambda = 0.50 ridge, 3×33\times 3 grid, ϵ=10−6\epsilon = 10^{-6}) with convex weights set once during initial formulation on [0,1][0,1]-normalized cue scales—with zero hyperparameter search on any evaluated benchmark split and zero retuning across all 33 VLLM architectures, 55 benchmarks, and 128128-frame scaling. Perturbing 88 core constants across 1616 wide (0.5×0.5\times--2×2\times) shifts (Table 5) keeps 14/1614/16 runs within [49.56,50.37][49.56, 50.37]—lying within the ∼1σ\sim 1\sigma statistical fluctuation band (≈±0.9\approx \pm 0.9 points on 2,7002{,}700-item VideoMME) of default 50.7050.70. Only two extreme boundary shifts degrade noticeably: μ=1.5\mu = 1.5 (48.5248.52, where a 1.5B1.5 B pool is too narrow before log⁡det⁡\log \det selection, whereas μ=4.0\mu = 4.0 stays at 50.2250.22) and Kq=16K_q = 16 (49.0849.08, where excessive query atoms dilute visual subspace axes on short single-sentence questions, whereas Kq=4K_q = 4 stays at 50.0750.07). Crucially, the exact same frozen configuration transfers zero-shot to beat PruneVID across all five benchmarks (including +3.9+3.9 on EgoSchema Total and +1.68+1.68 on LVBench).
Reviewer_EiQ4OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DOVE, a training-free framework for extreme visual token compression in video LLMs. It constructs quality-weighted evidence representations, employs a D-optimal objective to select complementary visual tokens, and introduces a residual readout to recover discarded information. Experiments across four benchmarks and three video LLMs demonstrate consistent improvements over existing compression methods under extremely limited token budgets.

Critical strengths

The paper addresses the practical challenge of reducing inference costs in video LLMs while preserving essential visual information. Its D-optimal formulation provides an intuitive way to balance token importance and information diversity. The extensive experiments across multiple benchmarks and model architectures demonstrate strong and consistent performance improvements, supported by efficiency analysis and ablation studies.

Critical weaknesses
  1. Diverse visual tokens may contain noise or low-information patterns. Although DOVE introduces quality-weighted evidence representations, these heuristic measures may not reliably distinguish meaningful information from irrelevant variations.

  2. DOVE relies on manually designed features and heuristic candidate pool construction. However, sufficient ablation studies are lacking to clarify how much the performance depends on these handcrafted components rather than the D-optimal objective itself.

  3. The theoretical analysis only guarantees approximate D-optimality within a fixed candidate pool, without addressing candidate pool quality or downstream task performance. Therefore, the theoretical results provide limited justification for DOVE's effectiveness in practical video understanding.

  4. The AI Use Statement and Related Work are placed outside the main text. In particular, moving the Related Work to the appendix weakens the discussion of the paper's novelty and its distinctions from existing methods. The AI Use Statement should also follow the placement recommended by the official ICLR template.

Questions for authors
  1. How does DOVE prevent noisy or low-information tokens from being selected merely because they provide diverse evidence directions? Can the authors provide robustness experiments?

  2. Can the authors conduct additional ablations on the handcrafted evidence features and candidate pool construction to clarify their individual contributions?

  3. Can the authors further clarify how the theoretical guarantees relate to downstream video understanding performance, given their restriction to a fixed candidate pool?

  4. Can the authors move the Related Work into the main text and place the AI Use Statement in accordance with the official ICLR template?

5 comments
AuthorsOpenReview ↗

Response to Reviewer EiQ4 [Part 1/3]

We sincerely thank your positive comment.

Q1 & W1: Preventing Noisy or Low-Information Tokens & Controlled Noise-Injection Robustness

How does DOVE prevent noisy or low-information tokens from being selected merely because they provide diverse evidence directions? Can the authors provide robustness experiments?

Response: DOVE prevents noisy or low-information tokens from hijacking the token budget through three complementary mechanisms:

  1. Multiplicative Reliability-Direction Coupling (zi=wieiz_i = \sqrt{w_i} e_i): Under Eq. (9), the marginal volume gain of candidate ii is Δ(i∣S)=log⁡(1+wiei⊤MS−1ei)\Delta(i \mid \mathcal{S}) = \log(1 + w_i e_i^\top M_{\mathcal{S}}^{-1} e_i) with ∥ei∥2=1\Vert e_i \Vert_2 = 1. Even if a low-salience patch points along an uncovered direction (ei⊤MS−1ei≤λ−1e_i^\top M_{\mathcal{S}}^{-1} e_i \le \lambda^{-1}), its marginal gain is scaled down by its scalar reliability mass wiw_i (analogous to precision weighting in heteroscedastic D-optimal experimental design).
  2. Damped Shrinkage Whitening (η=0.50\eta = 0.50), Ridge Floor (λ=0.50\lambda = 0.50), and Subspace Saturation: Full whitening (η=1,λ→0\eta = 1, \lambda \to 0) would amplify low-variance noise axes to unit variance. Setting η=0.50\eta = 0.50 with ridge λI\lambda I keeps minor noise eigenvalues small relative to λ\lambda, and log⁡det⁡(MS)\log \det(M_{\mathcal{S}}) rapidly saturates once shared noise directions in visual space are spanned.
  3. Temporal Mutual-Match Gating & Confidence-Protected Readout: Unlinked frame-level noise fails adjacent mutual-match verification (cosine threshold ≥0.82\ge 0.82), receiving zero same-component bonus (βc=0.08\beta_c = 0.08) in Eq. (13). Meanwhile, the top 15%15\% high-confidence anchors enforce an exact identity path (κj=0\kappa_j = 0, orig. αj=0\alpha_j = 0 in Eq. 15), preventing unselected noise tokens from corrupting clean anchors during residual readout.

Controlled Frame-Noise Injection Stress Test (Table 1): To directly stress-test W1 under a severe out-of-distribution perturbation, we evaluated LLaVA-OneVision on VideoMME across three conditions that preserve all 3232 clean frames in exact temporal order:

  • Clean (3232f): Original 3232-frame input (R=1%R = 1\% reference).
  • Duplicate Control (32+832+8f): Inserting 88 copied clean frames at random positions (4040 frames total) to control for sequence lengthening and temporal re-binning.
  • Gaussian Noise (32+832+8f): Inserting 88 i.i.d. full-frame RGB Gaussian noise images Inoise=clip⁡(0.5+0.2ϵ,0,1), ϵ∼N(0,1)I_{\text{noise}} = \operatorname{clip}(0.5 + 0.2\epsilon, 0, 1),\ \epsilon \sim \mathcal{N}(0, 1) (20.00%20.00\% of the 4040-frame timeline) at the exact same insertion positions.

All three conditions enforce the exact fixed absolute token budget of the original 3232-frame, R=1%R = 1\% setting (8282 outer anchors / 1616 inner tokens), granting zero extra token capacity to the 4040-frame inputs:

  • Statistically Indistinguishable Accuracy Shift: Comparing position- and budget-matched Gaussian Noise against Duplicate Control (Table 1), DOVE changes by only −0.59\mathbf{-0.59} percentage points (49.89→49.3049.89 \to 49.30), with its 95%95\% video-paired bootstrap confidence interval [−1.93, +0.74]\mathbf{[-1.93,\ +0.74]} spanning zero (and Short-video accuracy slightly rising by +0.33+0.33).
  • Subspace Suppression Below Timeline Share & Readout Quarantine: Because 88 entire frames (20.00%20.00\% of the video timeline) contain 100%100\% noise patches with zero clean alternatives—and candidate pool P\mathcal{P} intentionally performs stratified temporal-spatial sampling (with floor qmin⁡=0.15q_{\min}=0.15 and temporal-bin coordinates tit_i) so brief localized real-world events are never blindly skipped—those 88 timestamps naturally contribute candidates in proportion to their timeline share (21.38%21.38\% in P\mathcal{P}). Crucially, once inside P\mathcal{P}, DOVE's regularized log⁡det⁡\log \det objective actively suppresses noise selection below the 20.00%20.00\% timeline rate down to 17.05%\mathbf{17.05\%} (outer) and 16.91%\mathbf{16.91\%} (inner) as static directions saturate, while mutual-match gating (≥0.82\ge 0.82) and protected identity readout physically quarantine static tokens from contaminating clean anchors—preserving downstream QA accuracy.

Table 1: Controlled Frame-Noise Injection Robustness on LLaVA-OneVision (VideoMME, Fixed 3232f R=1%R=1\% Budget)

Input Condition (Full DOVE)ShortMediumLongOverallPool P\mathcal{P} NoiseSelected Noise (Outer / Inner)
Clean (3232f reference)55.1155.1150.3350.3346.6746.6750.70\mathbf{50.70}0.00%0.00\%0.00%0.00\% / 0.00%0.00\%
Duplicate Control (32+832+8f)54.8954.8948.3348.3346.4446.4449.8949.890.00%0.00\%0.00%0.00\% / 0.00%0.00\%
Gaussian Noise (32+832+8f, 20.00%20.00\% input noise)55.2255.2247.4447.4445.2245.2249.3049.3021.38%21.38\%17.05%\mathbf{17.05\%} / 16.91%\mathbf{16.91\%}
Paired Change (Noise −- Duplicate)+0.33+0.33−0.89-0.89−1.22-1.22−0.59\mathbf{-0.59}95%95\% Paired CI:[−1.93, +0.74]\mathbf{[-1.93,\ +0.74]} (spans 00)
AuthorsOpenReview ↗

Response to Reviewer EiQ4 [Part 2/3]

Q2 & W2: Ablations on Handcrafted Evidence Features, Candidate Pool, and D-Optimality

Can the authors conduct additional ablations on the handcrafted evidence features and candidate pool construction to clarify their individual contributions?

Response: DOVE decouples scalar reliability mass wiw_i (which weights observation quality so low-information background patches cannot dominate subspace volume) from evidence direction eie_i and candidate pool P\mathcal{P}. Table 2 (stepwise isolation on VideoMME) and Table 3 (four-benchmark leave-one-out ablations) quantify each factor at R=1%R=1\%:

  1. Candidate Pool Filtering Alone Gives Zero Accuracy Gain: Restricting scalar ranking to candidate pool P\mathcal{P} (Mass + candidate pool, Table 2 Row 2, 48.748.7) provides zero accuracy improvement over global Mass Top-K (Row 1, 48.848.8; note also that scalar Mass Top-K at 48.448.4 four-benchmark Avg. trails recent baselines PruneVID 49.249.2 and DivPrune 48.748.7). Thus, P\mathcal{P} serves purely to bound selection complexity from NN to ⌈μB⌉\lceil \mu B \rceil.
  2. Visual-Direction D-Optimality + Spectrum Whitening Form the Primary Engine: Holding the candidate pool P\mathcal{P} and scalar reliability mass wiw_i strictly identical to Row 2, replacing scalar Top-KK with regularized D-optimality over raw pooled visual directions ei=uie_i = u_i alone—stripping all handcrafted spatiotemporal, dynamic, and query directional coordinates (ti,si,ci,rit_i, s_i, c_i, r_i), whitening, exchanges, and readout (Visual-direction D-optimal, Row 3)—immediately jumps VideoMME from 48.748.7 to 50.0\mathbf{50.0} (+1.3+1.3), and adding shrinkage whitening (η=0.50\eta=0.50, Row 4) lifts it to 50.3\mathbf{50.3} (+1.6+1.6).
  3. Why Spectrum Calibration Unlocks Auxiliary Directional Cues & Readout: Adding dynamic trajectory/event coordinates cic_i onto Row 3 immediately aids Long-video coverage (44.8→45.344.8 \to 45.3, Overall 50.350.3, Row 5). Notably, shrinkage whitening (η=0.50\eta=0.50) is a critical prerequisite before appending discrete spatiotemporal or query coordinates: without whitening (η=0\eta=0), a few high-variance background channels dominate MSM_{\mathcal{S}}, so appending position bins onto unwhitened Row 3 pulls VideoMME to 49.149.1 (favoring Short 56.856.8 at the expense of Long 42.942.9). Once visual eigenvalues are calibrated in DOVE (Full) (Row 6: 50.7\mathbf{50.7} Overall, lifting Long to 46.7\mathbf{46.7}), the components work synergistically: in top-down leave-one-out ablations (Table 3), position coordinates improve VideoMME by +0.7+0.7 (50.0→50.750.0 \to 50.7, +0.2+0.2 Avg.), residual readout adds +0.5+0.5 Avg. (50.2→50.750.2 \to 50.7, +1.0+1.0 on LongVideoBench), and halving calibrated structural/query coordinate mass (0.32→0.160.32 \to 0.16) reduces VideoMME from 50.7050.70 to 49.5649.56 (−1.14-1.14).

Table 2: Stepwise Isolation of Candidate Pool, Visual D-Optimality & Calibrated Geometry (VideoMME, R=1%R=1\%)

Stepwise Variant (LLaVA-OneVision, R=1%R=1\%)ShortMediumLongOverallIsolated Role & Takeaway
1.1. Mass Top-K (Scalar mass wiw_i only)52.652.648.348.345.645.648.848.8Scalar ranking without volume maximization
2.2. Mass + candidate pool P\mathcal{P}51.951.948.248.246.046.048.748.7Pool filtering alone gives zero gain (−0.1-0.1)
3.3. Visual-direction D-optimal (on P\mathcal{P}, ei=uie_i = u_i, η=0\eta=0)55.455.449.949.944.844.850.0\mathbf{50.0}Raw visual D-optimality adds +1.3+1.3 over Row 2
4.4. Row 3 ++ shrinkage whitening (η=0.50\eta=0.50)56.456.449.449.445.145.150.3\mathbf{50.3}Spectrum calibration adds +1.6+1.6 over Row 2
5.5. Row 3 ++ dynamic context coordinates cic_i (η=0\eta=0)55.755.749.949.945.345.350.350.3Aids Long-video event coverage (+0.5+0.5) even unwhitened
6.6. DOVE (Full: Calibrated Geometry + Readout)55.155.150.3\mathbf{50.3}46.7\mathbf{46.7}50.7\mathbf{50.7}Best Overall (50.750.7) & Long (46.746.7, +1.6+1.6 over Row 4)

Table 3: Top-Down Leave-One-Out Ablation Across Four Benchmarks on LLaVA-OneVision (R=1%R=1\%)

VariantVideoMMEEgoSchemaLongVideoBenchMVBenchAvg. ScoreRel. Acc. (%\%)
Mass Top-K (w/o D-optimal volume)48.848.851.951.946.046.046.946.948.448.482.982.9
DOVE w/o position coordinates50.050.054.2\mathbf{54.2}48.148.149.7\mathbf{49.7}50.550.586.586.5
DOVE w/o residual readout50.150.154.054.047.447.449.449.450.250.286.086.0
DOVE (Full)50.7\mathbf{50.7}54.154.148.4\mathbf{48.4}49.7\mathbf{49.7}50.7\mathbf{50.7}86.8\mathbf{86.8}
AuthorsOpenReview ↗

Response to Reviewer EiQ4 [Part 3/3]

Q3 & W3: Connecting Fixed-Pool Theoretical Guarantees to Downstream Performance

Can the authors further clarify how the theoretical guarantees relate to downstream video understanding performance, given their restriction to a fixed candidate pool?

Response: This connection operates at two complementary levels:

  1. Empirical Saturation of Candidate Pool P\mathcal{P}: Theorem 2 guarantees a (1−1/e)(1 - 1/e) approximation (preserved monotonically by Corollary 1's positive-gain exchanges) relative to the optimal size-BB subset in P\mathcal{P} (∣P∣=⌈μB⌉\vert \mathcal{P} \vert = \lceil \mu B \rceil). Varying μ∈{1.5,2.5,4.0}\mu \in \{1.5, 2.5, 4.0\} on VideoMME (R=1%R=1\%) yields Overall scores of 48.5248.52, 50.7050.70 (default), and 50.2250.22: shrinking μ\mu to 1.51.5 under-covers the evidence space, whereas expanding μ\mu to 4.04.0 brings no further accuracy gain—confirming that μ=2.5\mu = 2.5 already captures the task-relevant subspace while filtering out >96%>96\% of redundant tokens.
  2. Why Ridge-Regularized Log-Determinant Volume Maps to VLLM Accuracy (Table 4): By the Kiefer-Wolfowitz equivalence theorem in optimal experimental design, maximizing log⁡det⁡(λI+ZS⊤ZS)\log \det(\lambda I + Z_{\mathcal{S}}^\top Z_{\mathcal{S}}) is dual to minimizing worst-case linear prediction variance across the evidence space. Because frozen Video LLM self-attention layers aggregate visual tokens through linear value projections, spanning semantic, spatiotemporal, and query axes minimizes prefill reconstruction error. Moreover, when B<dzB < d_z under extreme compression (e.g., B≤82B \le 82 while dz≈131d_z \approx 131), the ridge floor λI\lambda I prevents the rank-deficiency degeneracy that afflicts unregularized kk-DPP MAP determinants. Under the exact same pool P\mathcal{P}, geometry, and readout (Table 4), regularized D-optimality consistently outperforms unregularized kk-DPP MAP and metric coverage rules across all four benchmarks.

Table 4: Selection Objectives Under Identical Pool P\mathcal{P}, Geometry & Readout (LLaVA-OneVision, R=1%R=1\%)

Selection Objective (Shared P\mathcal{P}, Geometry & Readout)VideoMMEEgoSchemaLongVideoBenchMVBenchAvg. ScoreRel. Acc. (%\%)
kk-DPP MAP (Unregularized determinant)49.349.351.451.447.147.149.449.449.349.384.484.4
Farthest-first (Minimax metric coverage)49.849.853.053.046.446.449.649.649.749.785.185.1
FPS (Iterative point coverage)49.149.153.953.947.247.249.549.549.949.985.485.4
Regularized D-optimal (DOVE, Ours)50.7\mathbf{50.7}54.1\mathbf{54.1}48.4\mathbf{48.4}49.7\mathbf{49.7}50.7\mathbf{50.7}86.8\mathbf{86.8}

Q4 & W4: Placement of Related Work and AI Use Statement

Can the authors move the Related Work into the main text and place the AI Use Statement in accordance with the official ICLR template?

Response: Yes. In the revised manuscript, we will move Related Work into the main text as Section 2 (tightening overlapping introductory and implementation prose so the main text strictly obeys the ICLR 99-page limit) and place the Ethics, Reproducibility, and AI Use Statements in full compliance with the official ICLR 2027 template.

Reviewer_XmWcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper has good results. The performance is good even under very low compression rate. However, I think it lacks necessary comparion and discussion on its related methods. So it hard to differ whether their improvements have substianal contributions. I hope the author could provide more insights on the video compression problems in MLLM.

Critical strengths

Good performance. DOVE consistently outperforms existing baselines across multiple video benchmarks and VLLM backbones.

Training-free and efficient. The method requires no additional training and substantially reduces inference latency.

Critical weaknesses

Lack of solid demonstration of novelty. Query-aware token weighting has been explored in MustDrop[1], ReTaKe[2] and AdaReTaKe[3], while residual token merging resembles LOOK-M[4]. The paper lacks a clear discussion about these prior works. As a result, the paper does not provide sufficient comparisons or clearly distinguish its contributions from these methods. This is also reflected in the experimental results:

  1. Mass Top-K already outperforms all evaluated baselines

Table 1 and Table 3 show the following results on LLaVA-OneVision at 1% retention:

MethodAverage Accuracy (%)Gain over FastV (pp)
FastV (strongest external baseline)47.0—
Mass Top-K (without D-Optimal)48.4+1.4
DOVE (Full)50.7+3.7

We can see that, Mass Top-K, which uses the proposed evidence-mass scores without D-Optimal selection, already outperforms all evaluated external baselines.

In my opinion this suggests that a meaningful portion of DOVE's improvement may come from handcrafted importance scores, candidate construction, or other components rather than the D-Optimal objective itself. However, these components are not sufficiently isolated through ablation studies.

  1. D-Optimal provides relatively modest gains over alternative diversity-based selection methods

Table 4 compares different selection objectives under the same evidence geometry, candidate pool, token budget, and residual readout:

Selection ObjectiveAverage Accuracy (%)Relative Accuracy (%)
k-DPP MAP49.384.4
Farthest-first49.785.1
FPS49.985.4
D-Optimal (Ours)50.786.8

We can see that D-Optimal improves average accuracy by only 0.8 percentage points over FPS, the strongest alternative selection method. While this demonstrates a positive contribution, the improvement is relatively modest compared with the complexity introduced by the proposed optimization framework.

Overall, the current experiments establish the effectiveness of the complete DOVE framework but provide limited evidence that D-Optimal selection is the primary source of its improvements. I think more comprehensive component-wise ablations and comparisons with closely related compression methods are needed to clarify the paper's unique technical insights and advantages over prior work.

[1] Liu, T., Shi, L., Hong, R., Hu, Y., Yin, Q., & Zhang, L. (2024). Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model. arXiv preprint

[2 Wang, X., Si, Q., Wu, J., Zhu, S., Cao, L., & Nie, L. ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding. arXiv:2412.20504, 2024.

[3] Wang, X., Si, Q., Zhu, S., Wu, J., Cao, L., & Nie, L. AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding. Findings of ACL 2025, pp. 5417–5432.

[4] Wan, Z., Wu, Z., Liu, C., Huang, J., Zhu, Z., Jin, P., Wang, L., & Yuan, L. LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference. Findings of EMNLP 2024, pp. 4065–4078.

Questions for authors

See weakness. The authors should at least discuss the related methods, so we can gain more insights on the token compresson methods itself, rather than benchmaxxing some good results.

There are many questions in this field. For example, compression methods in vision encoder, prefilling, and decoding stage have very different characteristics. What is the best solution in certain scenrios?

3 comments
AuthorsOpenReview ↗

Response to Reviewer XmWc [Part 1/3]

We sincerely thank your constructive feedback. We especially appreciate your invitation to share broader insights into MLLM video compression across inference stages.

Q1: Stage-Wise Characteristics (Vision Encoder, Prefilling, Decoding) & Best Solutions by Scenario

Compression methods in vision encoder, prefilling, and decoding stage have very different characteristics. What is the best solution in certain scenarios?

Response: We fully agree that no single compression stage dominates every scenario. Following the lifecycle perspective in MustDrop [1], each stage targets a distinct bottleneck with different semantic visibility:

  • Vision Encoder Stage (e.g., ToMe, MustDrop [1] Stage 1): Compresses patch tokens inside ViT. It uniquely cuts ViT FLOPs alongside LLM cost, but operates query-agnostically and frame-locally before cross-modal alignment; aggressive early dropping risks irreversibly losing small or brief visual cues.
  • LLM Prefilling Stage: Targets O(LLLMN2D)\mathcal{O}(L_{\text{LLM}} N^2 D) prefill FLOPs and peak activation/KV memory (TTFT), splitting into two sub-regimes:
    • Pre-Layer-1 Compression (e.g., DOVE, PruneVID, DivPrune): Prunes/merges aligned tokens right before LLM Layer 1 (48.8→0.448.8 \to 0.4 TFLOPs and 1.0%1.0\% visual KV at R=1%R = 1\%), eliminating quadratic LLM cost across all layers, though query cues rely on lightweight input-space alignment rather than deep LLM attention.
    • Inner-LLM / Chunked Prefill (e.g., FastV, MustDrop Stage 2, ReTaKe [2], AdaReTaKe [3]): Leverages deep multi-head prompt-to-video attention inside the LLM, trading off either an early-layer compute/KV floor (FastV's unpruned Layers 1--2 cost 3.93.9 TFLOPs and 8.1%8.1\% average KV even at 1%1\% post-pruning retention) or sequential chunked forward passes (ReTaKe / AdaReTaKe).
  • Decoding / Post-Prefill KV Stage (e.g., LOOK-M [4], MustDrop Stage 3, MixKV): Merges or evicts K,VK, V states after prefilling or during generation. It cannot reduce initial prefill FLOPs or single-turn TTFT, but relieves decoding HBM bandwidth (TPOT) while preserving full multimodal prefill context.

Combined with our latency breakdown in Sec. 3.4 (where 3232-frame ViT takes ∼260 ms\sim 260\text{ ms} vs. 496 ms496\text{ ms} Vanilla LLM prefill and 31 ms31\text{ ms} compressed LLM forward) and 128128-frame scaling (50.7→54.450.7 \to \mathbf{54.4} Avg. at R=1%R = 1\%), different scenarios favor distinct stages (Table 1):

Table 1: Recommended Compression Stages Across Deployment Scenarios

ScenarioPrimary BottleneckBest-Fit Stage (Examples)Key Rationale
Live / On-Device VideoViT FLOPs (>80%>80\% of pruned TTFT)Vision Encoder + light prefill (ToMe, MustDrop [1])Only inside-ViT merging cuts the ∼260 ms\sim 260\text{ ms} ViT latency.
Pre-Indexed Cloud Video QAO(N2)\mathcal{O}(N^2) LLM prefill & KV quotaPre-Layer-1 Prefill (DOVE, PruneVID)Cached ViT; 1%1\% budget (0.40.4 TFLOPs) enables 4×4\times denser frames (128128f).
Ultra-Long Raw Video (103+10^3+ frames)Peak prefill HBM (OOM)Chunked Inner-LLM Prefill (ReTaKe [2], AdaReTaKe [3])Chunked prefill + adaptive KV bounds peak memory across hours.
Long CoT / Multi-Turn ChatDecoding bandwidth & topic shiftsDecoding / Post-Prefill KV (LOOK-M [4], MustDrop [1])Keeps broad prefill context for follow-up turns; merges KV pivots.

Crucially, these stages are composable: DOVE targets the extreme (≤1%\le 1\%) pre-LLM prefill regime and can pair orthogonally with upstream ViT merging, AdaReTaKe-style chunked KV allocation, or LOOK-M-style multi-turn KV merging. We will add this lifecycle discussion to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer XmWc [Part 2/3]

Q2 & W1 (Part 1): Discussion of MustDrop [1], ReTaKe [2], AdaReTaKe [3], and LOOK-M [4]

Query-aware token weighting has been explored in MustDrop[1], ReTaKe[2] and AdaReTaKe[3], while residual token merging resembles LOOK-M[4]. The paper lacks a clear discussion about these prior works.

Response: We agree that placing Related Work in the Appendix (Sec. 8, where MustDrop was only briefly cited and ReTaKe, AdaReTaKe, and LOOK-M were omitted) obscured DOVE's technical positioning. In the revised manuscript, we will move Related Work into Section 2 of the main text and add a dedicated comparison with [1--4]

  1. Query-Aware Weighting (MustDrop [1], ReTaKe [2], AdaReTaKe [3] vs. DOVE):

    • Stage: [1--3] compute prompt-to-vision cross-attention inside LLM layers (requiring unpruned early layers or chunked LLM prefills). DOVE operates before Layer 1 (0.40.4 TFLOPs) in the aligned embedding space, using peak-to-median confidence gating γ∈[0,1]\gamma \in [0, 1] (Eq. 16) so diffuse queries smoothly revert to visual geometry.
    • Subspace Volume vs. Scalar Ranking: While MustDrop [1] also avoids global prompt averaging by checking max token-level relevance, [1--3] apply query relevance as a univariate scalar ranker / budget allocator. At 0.5%0.5\%--1%1\% budgets (≤82\le 82 tokens), scalar query ranking concentrates tokens on whichever dominant frame matches the main noun, starving secondary events. DOVE summarizes the prompt into Kq≤8K_q \le 8 diverse query atoms and embeds them as directional coordinates rir_i in ziz_i (Eq. 5): once an atom's direction is covered in MSM_{\mathcal{S}}, log⁡det⁡(MS)\log \det(M_{\mathcal{S}}) diminishes gain along that axis and steers remaining budget toward uncovered query atoms and temporal states.
  2. Residual Readout (DOVE) vs. Pivotal KV Merging (LOOK-M [4]):

    • Both preserve anchor ("pivotal") states while absorbing discarded context via similarity-weighted fusion, but differ in three ways: (i) LOOK-M merges layer-wise K,VK, V caches across all layers after full prefill (saving decoding memory, not prefill FLOPs/TTFT), whereas DOVE's readout runs once in feature space RD\mathbb{R}^D before Layer 1 (8.9×8.9\times--9.3×9.3\times faster prefill); (ii) DOVE selects anchors via D-optimal subspace volume rather than scalar attention pivots; and (iii) to avoid cross-shot video blurring, DOVE restricts pooling to same/adjacent temporal bins, rewards mutually-matched trajectories (βc1[πi=πaj]\beta_c \mathbf{1}[\pi_i = \pi_{a_j}]), and enforces a hard identity path (αj=0\alpha_j = 0) on the top 15%15\% anchors.

W1 (Part 2): Isolating D-Optimality from Mass Top-K and Candidate Pool

Mass Top-K (48.4) already outperforms FastV (47.0), suggesting a meaningful portion of DOVE's improvement may come from handcrafted scores, candidate construction, or other components rather than D-Optimal itself.

Response: Evaluating recent 2025 baselines (PruneVID, ACL'25; DivPrune, CVPR'25) and bottom-up stepwise ablations (Table 2 & Table 3) directly resolves this concern:

  1. Mass Top-K Trails Recent 2025 Baselines (Table 2): Although Mass Top-K (48.448.4 Avg.) beat FastV (47.047.0), it underperforms both PruneVID (49.249.2) and DivPrune (48.748.7)—despite sharing our exact handcrafted mass wiw_i and compute-neutral 20+820+8 schedule (0.40.4 TFLOPs, 1.0%1.0\% KV). Regularized D-Optimality adds +2.3+2.3 Avg. points (48.4→50.748.4 \to \mathbf{50.7}), surpassing PruneVID and DivPrune at both R=1%R = 1\% (Table 2) and R=0.5%R = 0.5\% (47.447.4 vs. 44.844.8 / 46.546.5 Avg.; 30.4130.41 vs. 28.9928.99 / 29.5729.57 on hour-long LVBench).
  2. Candidate Pool Alone Gives Zero Gain (Table 3 Row 2): Adding pool P\mathcal{P} to scalar ranking (48.8→48.748.8 \to 48.7) purely bounds search size to ⌈2.5B⌉\lceil 2.5 B \rceil with zero accuracy boost.
  3. Raw Visual D-Optimality Drives the Primary Jump (Table 3 Rows 3--4): Holding P\mathcal{P}, scalar mass wiw_i, and the 20+820+8 schedule fixed to Row 2 (48.748.7), applying D-optimality to raw visual directions ei=uie_i = u_i alone (Row 3—stripping all spatiotemporal, dynamic, and query directional cues, whitening, exchanges, and readout) jumps VideoMME by +1.3+1.3 (50.0\mathbf{50.0}), and shrinkage whitening (η=0.50\eta = 0.50, Row 4) reaches 50.3\mathbf{50.3} (+1.6+1.6). Calibrated cues and readout (DOVE Full, 50.7\mathbf{50.7}) then lift Long-video accuracy (45.1→46.745.1 \to \mathbf{46.7}; leave-one-out Avg.: w/o readout 50.250.2, w/o pos. coords 50.550.5). Across 1616 wide (0.5×0.5\times--2×2\times) perturbations of 88 frozen constants, 14/1614/16 runs stay within [49.56,50.37][49.56, 50.37] on VideoMME without per-model/benchmark tuning.
AuthorsOpenReview ↗

Response to Reviewer XmWc [Part 3/3]

Table 2: Comparison Against Recent 2025 Baselines on LLaVA-OneVision (R=1%R = 1\%, 3232 Frames)

Method (R=1%R = 1\%)VideoMMEEgoSchemaLongVBMVBench4-Bench Avg.Rel. (%\%)LVBench (1h+)
Vanilla (100%100\%)58.558.560.360.356.656.658.358.358.458.4100.0100.039.2539.25
FastV$^\dagger$46.246.247.747.745.545.548.548.547.047.080.580.530.5430.54
FlashVID46.846.849.649.646.146.143.743.746.646.679.879.829.5029.50
DivPrune (CVPR'25)49.049.050.250.247.247.248.348.348.748.783.483.430.2130.21
PruneVID (ACL'25)50.250.250.250.247.447.448.948.949.249.284.284.231.5731.57
Mass Top-K (w/o D-Optimal)48.848.851.951.946.046.046.946.948.448.482.982.9--
DOVE (Full, Ours)50.7\mathbf{50.7}54.1\mathbf{54.1}48.4\mathbf{48.4}49.7\mathbf{49.7}50.7\mathbf{50.7}86.8\mathbf{86.8}33.25\mathbf{33.25}

†^\daggerFastV retains 100%100\% visual tokens in Layers 11--22 (3.93.9 TFLOPs, 8.1%8.1\% average KV vs. 0.40.4 TFLOPs, 1.0%1.0\% KV for pre-LLM methods and DOVE).

Table 3: Stepwise Isolation of Pool P\mathcal{P}, Raw Visual D-Optimality & Cues (VideoMME, R=1%R = 1\%)

Stepwise Variant (Shared 20+820+8 Schedule)ShortMediumLongOverallIsolated Contribution
1.1. Mass Top-K (Scalar mass wiw_i only)52.652.648.348.345.645.648.848.8Trails PruneVID (50.250.2) & DivPrune (49.049.0)
2.2. Mass + candidate pool P\mathcal{P}51.951.948.248.246.046.048.748.7Pool filtering alone gives zero gain (−0.1-0.1)
3.3. Visual-direction D-optimal (ei=uie_i = u_i, η=0\eta=0)55.455.449.949.944.844.850.0\mathbf{50.0}Raw visual D-optimality adds +1.3+1.3 (no extra cues)
4.4. Row 3 ++ shrinkage whitening (η=0.50\eta=0.50)56.456.449.449.445.145.150.3\mathbf{50.3}Spectrum calibration adds +1.6+1.6 over Row 2
5.5. DOVE (Full: Calibrated Geometry + Readout)55.155.150.3\mathbf{50.3}46.7\mathbf{46.7}50.7\mathbf{50.7}Lifts Long +1.6+1.6 over Row 4 (+2.0+2.0 over Row 2)

W2: Gain vs. Complexity of D-Optimal Selection Over FPS (+0.8 pp Avg. in Table 4)

D-Optimal improves average accuracy by only 0.8 percentage points over FPS (49.9 -> 50.7), which seems modest compared with the complexity of the optimization framework.

Response: Two observations clarify why D-optimality contributes meaningfully without heavy algorithmic overhead:

  1. FPS in Table 4 Already Inherits DOVE's Calibrated Geometry & Readout: Standalone diversity selection on raw visual features (DivPrune in Table 2) reaches only 48.748.7 Avg. (−2.0-2.0 points below DOVE). FPS reaches 49.949.9 in Table 4 only because it runs atop DOVE's whitened evidence geometry, stratified pool P\mathcal{P}, and protected readout—showing how the geometry and selector work as an integrated design.
  2. Clear Gains on Long, Multi-Event Videos (+1.6 VideoMME, +1.2 LongVideoBench) via Closed-Form Rank-One Updates: Pairwise point spacing (FPS) suffices on short single-scene clips (EgoSchema/MVBench), but cannot detect higher-order linear subspace collinearity (multiple tokens on an already-spanned plane remain pairwise distant yet add zero rank for linear VLLM value projections) and ignores token reliability wiw_i. Replacing FPS's pairwise distance rule with closed-form rank-one Sherman-Morrison leverage updates (Eq. 10, requiring zero matrix inversions) improves accuracy by +1.6+1.6 on VideoMME and +1.2+1.2 on LongVideoBench (Table 4), while also beating unregularized k-DPP MAP by +1.4+1.4 Avg. via ridge λI\lambda I when B<dzB < d_z.

Table 4: Selection Objectives Under Identical Pool P\mathcal{P}, Geometry & Readout (R=1%R = 1\%)

Selection ObjectiveVideoMMEEgoSchemaLongVBMVBench4-Bench Avg.Rel. (%\%)
kk-DPP MAP49.349.351.451.447.147.149.449.449.349.384.484.4
Farthest-first49.849.853.053.046.446.449.649.649.749.785.185.1
FPS49.149.153.953.947.247.249.549.549.949.985.485.4
Regularized D-Optimal (DOVE)50.7 (+1.6)\mathbf{50.7}\ (+1.6)54.1 (+0.2)\mathbf{54.1}\ (+0.2)48.4 (+1.2)\mathbf{48.4}\ (+1.2)49.7 (+0.2)\mathbf{49.7}\ (+0.2)50.7 (+0.8)\mathbf{50.7}\ (+0.8)86.8\mathbf{86.8}

We hope these stage-wise analyses, literature comparisons, and fine-grained ablations directly address your questions, and we warmly welcome further discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.