acceptodds
Under review as a conference paper at ICLR 2027

TeQCo: Temporal Quota Consistency for Training-Free Video Token Compression

Abstract

Video token compression must control both temporal coverage and task relevance, but sequential stages can conflict: a content-driven allocation stage may be overwritten by a global ranking that removes entire temporal positions. We identify this as an allocation conflict and propose TeQCo, a training-free framework that resolves it through temporal quota consistency. TeQCo allocates candidate capacity from first-order changes in normalized frame representations and aggregates all tokens within each frame into a compact candidate set. Candidates are then scored by prior-corrected task attention and selected within integer frame quotas inherited from the first-stage counts. Content variation determines temporal capacity; question relevance determines which candidates fill it. This design guarantees an exact token budget and a per-frame coverage floor whenever feasible. On the 2,700-question Video-MME benchmark with Qwen3-VL-8B, TeQCo achieves 68.52% accuracy at 510 ms TTFT, a 32.9% TTFT reduction with a 1.00-percentage-point accuracy decrease.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Language Model, Token Merge

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UVNdOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

To enable efficient video token compression in video large language models, this paper introduces TeQCo (Temporal Quota Consistency), a training-free framework with a two-stage compression pipeline. In the first stage, temporal feature changes guide token budget allocation, while spatial tokens are aggregated into representative candidates. The second stage selects task-relevant tokens using prior-corrected attention scores, subject to a temporal quota consistency mechanism that preserves frame-level allocation proportions. Evaluations across four video understanding benchmarks and four model backbones show that TeQCo can reduce time to first token (TTFT) while incurring relatively modest accuracy degradation in several settings.

Critical strengths
  1. Across four video understanding benchmarks and multiple model architectures, including Qwen3-VL, InternVL3.5, and LLaVA-OneVision, the proposed training-free framework achieves meaningful reductions in time to first token (TTFT) while maintaining competitive accuracy in several settings. These extensive evaluations highlight its practical value for efficient video understanding.
Critical weaknesses
  1. Limited Methodological Novelty. The extent to which TeQCo advances beyond existing techniques remains unclear. Its main components—content-adaptive token allocation, feature aggregation, and task-aware token selection—follow principles previously explored in V-CAST [1] , VidCom2 [2] , and PriorTR [3]. As currently formulated, the central contribution of temporal quota consistency appears to impose a relatively straightforward proportional constraint on frame-level token budgets during the second compression stage. A clearer explanation of what is methodologically new, beyond the integration of these components, would strengthen the paper.

  2. Insufficient Validation of the Core Contribution. Establishing the effectiveness of temporal quota consistency requires a controlled comparison between global Top-K selection and quota-constrained selection, with candidate representations, relevance scores, and final token budgets held fixed. Such a comparison is currently missing, despite the mechanism being positioned as the paper’s central contribution. Instead, the existing ablations vary multiple components or compression budgets simultaneously, leaving the source of the observed improvements unclear. Experiments that isolate the quota constraint are therefore needed to substantiate the proposed mechanism and its underlying motivation.

  3. Inconsistent Accuracy–Efficiency Trade-offs and Insufficient Budget-Matched Comparisons. The practical advantages of TeQCo remain difficult to establish because comparisons with existing methods use different operating points rather than matched token budgets. Although TTFT is consistently reduced, these gains are accompanied by substantial accuracy losses on some benchmarks, particularly LongVideoBench and MVBench. Faster TTFT also does not uniformly yield lower end-to-end inference latency; on Video-MME, total latency increases slightly. Evaluating accuracy–latency trade-offs at matched token budgets and equivalent compression ratios would provide a clearer basis for assessing the method’s practical benefits.

  4. Poor Figure Quality and Unprofessional Visual Presentation. The overall visual quality of the paper needs substantial improvement, particularly Figure 1, which appears rough and insufficiently polished. The excessive use of irregular, wavy lines and hand-drawn-style elements makes the figure visually cluttered and distracting. These wave-like patterns are visually jarring and can even make the figure uncomfortable to look at, rather than helping readers understand the proposed framework. As the main methodological illustration, Figure 1 should be redesigned with cleaner lines, consistent alignment, and a more professional visual style to improve readability and presentation quality.

[1] V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models. NeurIPS 2026

[2] Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models. EMNLP 2025

[3] Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction. ECCV 2026

Questions for authors

The paper is far below high level conference bar.

Reviewer_en8sOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes TeQCo, a training-free framework for video token compression. The first stage estimates temporal importance from changes in normalized frame representations, allocates per-frame token budgets, and aggregates tokens within each frame. The second stage uses prior-corrected task attention to select relevant candidates under the inherited frame quotas. This design ensures an exact final token budget while preserving frame coverage when feasible. Experiments across four backbones and four benchmarks evaluate accuracy and inference efficiency, with notable reductions in time to first token (TTFT) on Video-MME.

Critical strengths
  1. Clearly specified compression procedure. The paper presents a two-stage framework that separates temporal budget allocation from task-aware token selection. The integer allocation procedure is explicitly formulated, with guarantees on the final token count and feasible frame coverage. These properties make the proposed method relatively straightforward to understand and reproduce.

  2. Evaluation across multiple settings. The experiments include four backbones and four video understanding benchmarks. The reported TTFT accounts for candidate construction, attention scoring, token selection, and final prefill. The paper also reports total latency and peak memory, allowing a more complete examination of inference efficiency.

Critical weaknesses
  1. Insufficient validation of the central contribution. My primary concern is that the experiments do not directly establish the benefit of quota inheritance, which appears to be the main distinguishing contribution of TeQCo. In Table 4, the ablations do not isolate quota inheritance from other factors, as introducing task-aware selection also changes the token budget. More importantly, there is no controlled comparison against unrestricted global selection or global selection with the same minimum frame coverage, while keeping the candidates, relevance scores, and final token budget identical. Without these controls, it is difficult to determine whether the observed benefits arise from quota inheritance itself or simply from task-aware filtering and coverage preservation. Furthermore, the underlying assumption that question-independent temporal changes should determine task-specific token allocation remains insufficiently justified. Although Appendices A.4–A.6 establish formal properties under fixed quotas, these results do not demonstrate that the inherited allocation is empirically preferable. Given the central role of quota inheritance in the proposed framework, I consider this a significant limitation.

  2. Unclear advantage in the accuracy and efficiency tradeoff. While TeQCo achieves promising TTFT reductions in certain settings, the current comparisons do not convincingly establish its advantage over existing methods. Section 4.1 primarily compares methods at their reported operating points, without sufficiently detailed token retention settings or accuracy versus latency curves. Consequently, it remains unclear whether TeQCo offers a better tradeoff under comparable computational budgets. In fact, some results favor existing methods in both accuracy and TTFT. On Qwen3-VL-8B with LongVideoBench, TeQCo achieves 58.19% accuracy at 688 ms, compared with EarlyTom's 58.69% at 641 ms (Table 1). Similarly, on InternVL3.5-8B with MVBench, TeQCo achieves 66.13% at 186 ms, compared with VidCom2's 70.61% at 149 ms (Table 2). Table 3 further indicates that TTFT improvements do not consistently translate into reductions in total latency or peak memory. While I appreciate the authors' acknowledgment of these limitations, the current evidence is insufficient to establish a compelling practical advantage for the proposed framework.

Questions for authors

I am currently leaning toward Clear Rejection, primarily due to the insufficient validation of quota inheritance and the unclear comparative advantage. The following questions address these concerns directly. Convincing experimental evidence would help me reconsider my assessment.

  1. Isolating the benefit of quota inheritance. Could the authors compare inherited quota allocation with unrestricted global selection and global selection under the same minimum frame coverage, while keeping the candidate tokens, relevance scores, and final token budget fixed? Reporting both task accuracy and empty-frame rates would help clarify whether quota inheritance provides benefits beyond coverage preservation.

  2. Clarifying the competitive operating range. Could the authors report the actual retained token counts and compression settings of the baselines, along with accuracy versus TTFT curves for TeQCo and the strongest alternatives? I wonder whether TeQCo demonstrates a favorable tradeoff on LongVideoBench and MVBench, where the current comparisons are less convincing.

Reviewer_8LP3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an algorithm called TeQCo for video token compression. The proposed algorithm is a training-free framework that uses temporal quota consistency to alleviate the conflict between sequential content-driven allocation and task-driven token selection stages. TeQCo first allocates per-frame candidate capacity based on first-order changes and aggregates the tokens into a compact candidate set. It then scores the candidates using prior-corrected task attention and selects them within inherited frame quotas. The experimental results show that the proposed algorithm achieves lower TTFT with a 1-percentage-point decrease in accuracy.

Critical strengths
  • This paper is clearly written and easy to follow.
  • The proposed algorithm is technically sound and mathematically guarantees the maximization of the total relevance score under fixed quotas.
  • Extensive experiments across multiple models and benchmarks demonstrate a competitive accuracy–latency trade-off.
Critical weaknesses
  • More analysis directly supporting the paper’s core claim would strengthen the paper.
Questions for authors
  1. A qualitative visualization on real examples, showing the per-frame budget allocation and the tokens actually selected, would be helpful.
  2. Related quantitative analyses, such as empirical quota distributions, the fraction of frames preserved, and how the temporal allocation changes under global Top-K, would also be useful to understand the proposed algorithm more deeply.
  3. Table 4 does not appear to include an ablation that directly isolates temporal quota consistency. Since this is a central claim, a dedicated experiment would strengthen the paper. For example, comparing a) prior-corrected Quota-constrained Top-K and b) prior-corrected Global Top-K

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.