acceptodds
Desk rejected at ICLR 2027

VOLT: Joint Token Selection and Frame Budget Allocation via Subspace Volume Maximization

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Efficient video understanding with multimodal large language models requires compressing visual tokens without losing sparse events or local details. Existing training-free approaches commonly allocate equal token budgets to frames or separate frame-level budget prediction from token selection. These designs can retain redundant content and underrepresent frames containing informative events. We present VOLT, which unifies token selection and cross-frame budget allocation as constrained subspace volume maximization. A shared log-determinant objective measures the geometric diversity of retained token representations and jointly determines their identities and distribution across frames. A minimum retention constraint preserves coverage of each frame, while within-frame nearest-neighbor merging transfers information from discarded tokens to retained tokens without increasing sequence length. Experiments on five video understanding benchmarks with three video MLLM backbones show an improved accuracy-efficiency trade-off under aggressive compression. VOLT requires neither retraining nor modification of the backbone, providing a practical geometric approach to video token reduction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
video token compression, multimodal large language models, efficient inference, subspace volume maximization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_qtE3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes VOLT, a training-free method for video token compression. It reserves a minimum number of tokens per frame, then allocates the remaining budget across frames through greedy orthogonal-residual selection motivated by a log-determinant objective. Discarded tokens are assigned to their cosine-nearest retained tokens within the same frame and aggregated before the compressed sequence is passed to the frozen LLM. The paper evaluates the method on five video understanding benchmarks and three video MLLM backbones.

Critical strengths
  1. VOLT preserves strong accuracy under aggressive token compression. On LLaVA-OneVision at 10% retention (Table 1), it exceeds the uncompressed model on EgoSchema (61.2 vs. 60.4), LongVideoBench (57.1 vs. 56.4), and MLVU (64.4 vs. 64.0), while retaining 100.4% of the uncompressed five-benchmark average. Preserving this level of accuracy with approximately one tenth of the visual tokens is a useful result.

  2. The reported improvements extend across different backbones. At 10% retention on LLaVA-Video (Table 2), VOLT improves over the manuscript’s FlashVID baseline on LongVideoBench from 54.9 to 55.5 and on MLVU from 67.7 to 68.4, while retaining 96.0% of the uncompressed average accuracy. On Qwen2.5-VL at the same retention ratio (Table 3), it improves MVBench from 64.2 to 64.8 and LongVideoBench from 53.7 to 54.2 over the reported FlashVID baseline. These are incremental but consistent gains across the evaluated model families.

  3. I did not find major technical problems with the core method. The pipeline appears sound and operates without retraining the backbone. The evaluation covers five video understanding benchmarks, three video MLLM backbones, and several retention ratios.

Critical weaknesses
  1. Limited novelty and outdated baseline model makes the missing performance comparisons important. I view this primarily as a performance-centered paper. The comparison should therefore include established, closely related methods with competitive published results. In particular:

    • UniComp (CVPR 2026) reports a mean of 60.775 over LongVideoBench, EgoSchema, MLVU, and VideoMME on OneVision-7B with 32 frames at 25% retention, compared with 60.425 for VOLT on these four benchmarks. UniComp is higher on three of the four benchmarks. VOLT’s shared-four average is higher at 10% and 15%, so this is an omitted competitive comparison rather than evidence of general inferiority.
    • MMG-Vid (AAAI 2026) reports LongVideoBench scores of 58.8/58.6/57.7/56.3 at 25/20/15/10% retention on LLaVA-Video-7B with 64 frames. VOLT Table 2 reports 57.8/57.5/56.0/55.5. Both report an uncompressed LongVideoBench score of 58.9. VOLT is higher on the other three shared benchmarks and their mean, but MMG-Vid remains a relevant missing quantitative comparison, especially since it is already discussed in Section 2.

I understand that these paper might have their own reimplementation of the compared methods. But considering the proximity of the topic, I think they should also be included in the main comparison. Or explained why they are excluded in the comparison.

  1. I believe there is a factual error in Table 2 for the VisionZip entry. Its numbers are identical across the 25% and 15% retention settings. For such are large drop in the token count, it’s highly unlikely that the numbers would stay the same. Please check the presented numbers carefully.

  2. The writing and presentation could be significantly improved.

    • Citation formatting needs systematic correction. Narrative author–year citations are repeatedly appended where parenthetical citations are needed. For example, Section 3.2, p4, line 200, reads “…token features Taskar (2013)” and later “…feature directions Boyd & Vandenberghe (2004).” The same pattern appears in the opening sentence of Section 1: “Recent video MLLMs Li et al. (2024a); Zhang et al. (2024b); … encode …”. These supporting citations should be placed in parentheses or integrated grammatically into the sentence.

    • The DPP reference has incorrect author and publication metadata. The bibliography on p11 lists “Alex Kulesza Ben Taskar” as one name and gives “stat, 1050:10, 2013.” The canonical journal reference is Alex Kulesza and Ben Taskar, Determinantal Point Processes for Machine Learning, Foundations and Trends in Machine Learning 5(2–3):123–286, 2012. The corresponding citation should name both authors. If the authors intend the 2013 arXiv revision, they should identify that version consistently.

    • Figure 2 should include the reservation stage. Its global-selection loop starts with an empty set, while Section 3.2 first reserves mm tokens independently per frame and initializes the global basis from their union. Adding this stage would make the actual sequence of operations more clear.

    • Table 6 does not sufficiently define its fusion variants. The distinction between “Within-frame avg.” and “Within-frame nbhd.” is unclear because Section 3.3 already averages tokens assigned to cosine-nearest representatives. “Cross-frame top-k” also lacks a specified kk. Please define or explain what’s be compared here.

    • The diagnostic figures need clearer definitions. For example, figure 1(b,c) does not specify the feature projection, the conditioning and normalization of “log-det gain,” or the variables and statistic behind the displayed correlation.

Questions for authors
  1. Can the authors provide matched comparisons with UniComp and MMG-Vid, including the evaluation settings and actual retained-token counts? Or explain the exclusion.
  2. Can the authors check the number presented in Table 2 for VisionZip?
Reviewer_5W6AOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VOLT, a training-free video token compression method for efficient video MLLM inference. The main idea is to select tokens from a global cross-frame pool using a subspace volume maximization objective, while enforcing a minimum number of retained tokens per frame to avoid temporal starvation. The authors use a greedy orthogonal selection algorithm to approximate the objective and further merge discarded tokens into retained ones within each frame. Experiments on video understanding benchmarks and three different MLLM backbones show consistent improvements over existing methods, especially at low token retention ratios.

Critical strengths
  1. The paper is well-motivated. I agree that selecting tokens independently within each frame can lead to unnecessary redundancy across frames. The idea of considering global feature diversity while maintaining some temporal coverage is intuitive and makes sense for video compression.
  2. The proposed method is simple and practical. VOLT does not require additional training or modifications to the underlying MLLM. The subspace volume criterion provides an intuitive geometric motivation for token selection, and the overall framework can be integrated into existing models with relatively little modification.
  3. The experiments are fairly comprehensive, covering three backbones, five benchmarks, and several retention ratios. The consistent improvements over recent methods such as AOT and FlashVID are encouraging. I also appreciate the component ablations and the inclusion of actual inference latency measurements.
Critical weaknesses
  1. I have some concerns about the actual optimization procedure. The paper motivates the method using log-determinant maximization, but the implemented algorithm uses orthogonal residuals, which only approximate the exact marginal gains under certain conditions. A study of downstream accuracy and computational overhead of the two methods would be helpful. Moreover, per-frame reservations are performed independently before global selection, so it is unclear how well the resulting solution optimizes the constrained objective. One specific issue is that, since all tokens are normalized, their residual scores are identical at the first selection step. With the default (m=1), how is the reserved token in each frame chosen? I think this needs clarification, and a comparison with exact log-determinant greedy selection is needed.
  2. Temporal starvation is a central motivation, but I did not find a direct evaluation showing that VOLT better preserves sparse or short-lived events.
Questions for authors

Please see Critical Weaknesses, W1 as Q1, W2 as Q2

Reviewer_HBYCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes VOLT, a training-free, pre-LLM visual token compression framework for video MLLMs based on constrained subspace volume maximization. To address redundant cross-frame sampling and temporal starvation under aggressive compression, VOLT formulates token selection over the global cross-frame token pool X\mathcal{X} as maximizing a regularized log-determinant volume objective log⁡det⁡(Id+ϵ−1XS⊤XS)\log\det(I_d + \epsilon^{-1}X_S^\top X_S) subject to a total budget ∣S∣=B|S|=B and a per-frame retention floor ∣S∩Xf∣≥m|S \cap X_f| \ge m. Taking the ϵ→0\epsilon \to 0 limit, VOLT greedily selects ℓ2\ell_2-normalized tokens with the largest squared orthogonal residual energy ∥xi−Ut−1Ut−1⊤xi∥22\|x_i - U_{t-1}U_{t-1}^\top x_i\|_2^2 in two passes: first selecting mm base tokens independently within each frame (default m=1m=1), and then allocating the remaining B−FmB - Fm slots globally across frames. Finally, pruned tokens are merged into their cosine-nearest retained neighbors within the same frame. Experiments are reported on LLaVA-OneVision-7B, LLaVA-Video-7B, and Qwen2.5-VL-7B across five video understanding benchmarks.

Critical strengths
  1. Simple Pre-LLM Integration: Operating entirely before LLM ingestion without modifying attention layers or requiring auxiliary models, VOLT is architecture-agnostic and naturally compatible with FlashAttention.
  2. Unified Cross-Frame Formulation: Combining a per-frame retention floor with global subspace orthogonal selection provides an intuitive geometric mechanism to balance temporal support and cross-frame deduplication, supplemented by within-frame local token merging.
Critical weaknesses
  1. Highly Suspicious Arithmetic Patterns and Data Inconsistencies Across Experimental Tables:

    • Exact Arithmetic Progressions in Tables 4 and 7: In Appendix Table 7 (Qwen2.5-VL 3x frame scaling), every single one of the 45 reported benchmark scores ends in .0 or .5. Moreover, across five independent methods (VisionZip, FastVID, AOT, FlashVID, VOLT), the scores form exact arithmetic progressions with a constant step of +0.5: e.g., 61.0, 61.5, 62.0, 62.5, 63.0 on VideoMME Overall; 56.5, 57.0, 57.5, 58.0, 58.5 on LongVideoBench; and 60.5, 61.0, 61.5, 62.0, 63.0 on MLVU. Similar .0/.5 staircase patterns appear in Table 4 (5x and 10x scaling).
    • Constant Offsets Between Baselines in Tables 1–3: In Table 3 (Qwen2.5-VL-7B), FlashVID beats AOT by literally +0.1 on every single benchmark at 25% retention (66.5, 57.5, 55.9, 59.6, 59.2 vs. 66.4, 57.4, 55.8, 59.5, 59.1), by literally +0.1 on every single benchmark at 20% retention (66.1, 57.1, 55.4, 59.1, 58.6 vs. 66.0, 57.0, 55.3, 59.0, 58.5), and by literally +0.2 on every single benchmark at 10% retention (64.2, 55.7, 53.7, 57.6, 56.9 vs. 64.0, 55.5, 53.5, 57.4, 56.7). The exact same uniform +0.1 offset across all five datasets also occurs in Table 1 (LLaVA-OV at 20%). Such uniform constant offsets across five diverse benchmarks and multiple retention ratios are statistically implausible in real evaluations.
    • Swapped Columns Between Table 1 and Table 6: For VOLT on LLaVA-OV-7B at 10% retention, Table 1 reports 58.3 on VideoMME and 64.4 on MLVU, whereas Table 6 reports 64.40 on VideoMME and 58.33 on MLVU.
  2. Mathematical Degeneracy of Step-1 Selection Under ℓ2\ell_2-Normalization and m=1m=1:

    • Section 3.2 (Line 210) and Appendix C state that all token features are ℓ2\ell_2-normalized (∥xi∥2=1\|x_i\|_2 = 1) before selection, and the default per-frame floor is m=1m=1. However, during the first pass where m=1m=1 base token is selected independently within each frame ff, the initial orthonormal basis is empty (U0=∅U_0 = \emptyset). Consequently, every token xix_i in frame ff has the exact same residual energy gi(1)=∥xi−0∥22=∥xi∥22≡1g_i^{(1)} = \|x_i - 0\|_2^2 = \|x_i\|_2^2 \equiv 1, making the selection of the m=1m=1 base token in every single frame completely degenerate (an arbitrary tie-break)! Furthermore, ℓ2\ell_2-normalizing tokens discards feature norms (∥xi∥2\|x_i\|_2), which carry crucial visual saliency information, causing low-norm background noise tokens to be weighted equally with salient foreground tokens.
  3. Incremental Novelty Over Greedy DPP-MAP and Lack of Query Awareness / Key Baselines:

    • Maximizing log⁡det⁡(XSXS⊤+ϵI)\log\det(X_S X_S^\top + \epsilon I) (Eq. 4) is the standard MAP objective for Determinantal Point Processes (DPP-MAP), and the ϵ→0\epsilon \to 0 orthogonal residual rule is classical pivoted Gram-Schmidt / Cholesky greedy DPP inference. Yet Figure 5(b) plots DPP as a much weaker baseline without explaining how it differs from Eq. (4).
    • Because VOLT is completely query-agnostic, it allocates budget to visually diverse background variations rather than query-relevant evidence. Moreover, several highly relevant baselines cited in the paper (e.g., HoliTom, MMG-Vid, DivPrune, FrameFusion) as well as recent query-aware/metadata-guided video compressors (e.g., QuoTA, CDPruner, MeToM) are omitted from the main tables.
  4. Unreported Sequential Latency Overhead (O(MBd)\mathcal{O}(MBd)) at Larger Budgets and Frame Counts:

    • Greedy orthogonal selection requires BB strictly sequential steps (since step tt depends on orthogonalizing against ut−1u_{t-1}). On LLaVA-Video-7B (M=10,816M=10,816 tokens, d=3584d=3584) at 25% retention (B=2,688B=2,688), this requires 2,688 sequential GPU kernel steps. However, Figure 3(a) only reports TTFT on 32-frame LLaVA-OV at the smallest 10% budget (B=640B=640), omitting compression latency at 20%–25% budgets and on 64-frame LLaVA-Video or 160-frame Qwen2.5-VL.
Questions for authors
  1. Explanation of Regular Arithmetic Patterns in Tables 1–4 & 7: How do the authors explain that all 45 numbers in Table 7 end in .0 or .5 with exact +0.5 arithmetic progressions across five distinct methods, and that FlashVID exceeds AOT by a constant +0.1 (at 25% and 20%) and +0.2 (at 10%) across all five benchmarks in Table 3? Please provide raw evaluation logs for these runs.
  2. Degeneracy at U0=∅U_0 = \emptyset with ℓ2\ell_2-Normalized Tokens: Since all tokens are ℓ2\ell_2-normalized (∥xi∥2=1\|x_i\|_2 = 1), gi(1)=∥xi∥22=1g_i^{(1)} = \|x_i\|_2^2 = 1 for all tokens when U0=∅U_0 = \emptyset. With default m=1m=1, how does Stage 1 select the single base token per frame when all tokens tie at 1.0?
  3. Difference from DPP-MAP and Sequential Latency: How is the DPP baseline in Figure 5(b) implemented given that Eq. (4) is the standard DPP-MAP objective? What is the wall-clock compression overhead of VOLT on 64-frame LLaVA-Video-7B (M=10,816M=10,816) at 25% retention (B=2,688B=2,688 sequential steps)?
Reviewer_ZuZnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

VOLT is a training-free video token compressor for video MLLMs. It treats token selection and cross-frame budget allocation as one constrained log-determinant (subspace volume) maximization over a global token pool. It approximates the objective with greedy selection by orthogonal residual norm, enforces a per-frame floor of m tokens (default m=1), and merges pruned tokens into their nearest retained token within the same frame. The authors name two failure modes it targets: "redundant subspace sampling" and "temporal starvation." On five video benchmarks with LLaVA-OneVision-7B, LLaVA-Video-7B and Qwen2.5-VL-7B at 10–25% retention, VOLT has the best average score in every setting. It keeps 96.0% of LLaVA-Video accuracy at 10% of the tokens and improves further when the saved budget is spent on more frames.

Critical strengths
  • Clean, unified formulation: One volume criterion handles both which tokens to keep and how to distribute the budget across frames. This is conceptually tidier than two-stage frame scoring followed by token selection.

  • Broad, consistent evaluation: Three backbones, five benchmarks, four retention ratios and strong recent baselines (FlashVID, AOT, FastVID) are covered, and VOLT ranks first in every averaged configuration.

  • Practical and efficient: The method needs no query, no training and no backbone change. Selection costs O(MBd), and end-to-end TTFT (including compression) drops from 1684 ms to 541 ms on LLaVA-OV.

  • Useful frame-scaling result: Spending the compressed budget on 5–10× more frames improves Qwen2.5-VL beyond its uncompressed 16-frame accuracy (up to 106.3% relative). This is a compelling use case.

Critical weaknesses
  • Limited novelty relative to DPP-based selection: Greedy log-det maximization, and its ε→0 residual-norm limit, is standard greedy DPP-MAP / pivoted Gram–Schmidt. DPP-style token pruning for MLLMs already exists (e.g., CDPruner, "Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs", which is not cited). DivPrune and DPP-MAP appear only in a figure or appendix, not in the main tables.

  • Small margins without variance: Gains over FlashVID/AOT are typically 0.2–0.6 average points (e.g., 59.8 vs. 59.4 at 10% on LLaVA-OV). There are no seeds, CIs or significance tests, so the "best everywhere" claim may be within noise.

  • In Tables 3, 4 and 7 the baseline ranking (VisionZip < FastVID < AOT < FlashVID < VOLT) holds in almost every single column. Many frame-scaling entries are exact multiples of 0.5, even though benchmark granularity is finer. This calls for clarification and raw evaluation logs.

  • Insufficient support for the "temporal starvation" mechanism: The best floor is m=1, which mostly reduces to global selection. Adding the floor gives only +0.7 points (Table 5). No direct evidence (e.g., event-localized subsets) shows that event-bearing frames are rescued.

  • VOLT ignores the text query, yet query-aware baselines (SparseVLM, query-driven clustering) are mostly absent from the video tables. Its behavior on fine-grained, question-specific evidence is untested.

Questions for authors
  1. How does VOLT differ in practice from greedy DPP-MAP (Chen et al., 2018) on the same features, and why does DPP-MAP score lower in Figure 5(b) if VOLT is its ε→0 limit? Please add DivPrune, DPP-MAP and CDPruner to the main tables.
  2. Can you report mean ± std over multiple runs (or bootstrap CIs over questions) for VOLT and the strongest baselines? Were baseline numbers re-run under identical settings or copied from prior papers?
  3. Please explain the rounding in Tables 4 and 7 and the near-uniform ranking across columns, and release per-benchmark evaluation logs.
  4. What is the compression overhead (ms) separately for LLaVA-Video (M≈10.8K tokens), and how does it scale with the number of frames in the frame-scaling setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.