- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces OTT-Vid, a training-free framework that uses importance-aware optimal transport to guide temporal token compression and allocate compression budgets across adjacent frames. Evaluation on four Video-LLM backbones and six benchmarks shows improved performance retention over competing compression methods and inference speedups over uncompressed models.
Critical strengths
- Coherent compression framework. The method connects importance-aware temporal matching and frame-pair budget allocation through a shared transport formulation, addressing both which tokens to compress and where to allocate compression.
- Broad empirical validation. Evaluation across four backbones, six benchmarks, and multiple retention ratios demonstrates consistent performance retention and practical inference speedups. Including temporal grounding makes the evaluation particularly relevant to preserving temporal evidence.
Critical weaknesses
- Limited conceptual novelty. The method appears to be a well-engineered combination of existing heuristics, rather than a sufficiently distinct methodological advance over prior compression approaches.
- Insufficient justification of coupling-based ranking. Raw OT couplings depend on both marginal masses and matching costs, but their advantage over marginal-normalized scores is not established.
- Unvalidated compression-difficulty proxy. The paper does not directly establish that transport cost predicts compression-induced degradation or outperforms simpler similarity-based budget allocation.
- Indirect evidence for temporal preservation. Aggregate temporal-grounding results do not isolate the claimed preservation of persistence, duration, and event boundaries. Targeted evaluation is needed to substantiate this central motivation.
- Incomplete baseline coverage. Standalone comparisons with DivPrune (CVPR 2025), FLoC (ICLR 2026), STTM (ICCV 2025), PruMerge (ICCV 2025), and PruneVid (ACL 2025) are needed.
- Limited benchmark coverage. Additional evaluation on LVBench, EgoSchema, and NExT-QA would strengthen the evidence for generalization, particularly for long-video understanding and temporal reasoning.
Questions for authors
- Clarification: Coupling ranking. Why are raw OT couplings appropriate for ranking compression candidates, given their dependence on marginal masses? Have marginal-normalized alternatives been evaluated?
- Evidence request: Budget allocation. Does transport cost correlate with actual compression-induced performance degradation, and does it outperform simpler similarity-based allocation signals?
- Evidence request: Temporal preservation. Can the authors provide targeted results on persistence, duration, or event-boundary questions to substantiate the central motivation?
- Additional evaluation request. Can the authors provide matched-budget comparisons with standalone DivPrune, FLoC, PruMerge, and PruneVid, and results on LVBench, EgoSchema, and NExT-QA? These would help assess whether the reported advantages generalize beyond the current comparisons.
7 comments
[W3, Q2]. Transport cost as a budget-allocation signal
Why transport cost. Our choice follows directly from the OT objective. The transport plan is optimized to minimize token-level alignment cost under importance-aware marginal constraints, with entropy regularization. The resulting cost, , quantifies alignment difficulty using the same plan and cost matrix that guide compression. We use it as a proxy for compressibility, assigning more compression to lower-cost frame pairs.
Our manuscript does not claim an established correlation between transport cost and task degradation. Directly establishing that relationship is challenging because the effect of compression depends on the question, the existing allocation, and the amount of additional compression. We instead assess its practical utility through controlled comparisons with alternative allocation signals.
Why compare with pooled similarity. Frame-level similarity is used for temporal segmentation in FastVID and frame grouping and allocation in UniComp. Motivated by these approaches, we use pooled-frame dissimilarity as a simpler allocation signal. Pooling can capture broad changes in frame content, such as scene transitions, but local changes may be diluted or canceled in the frame average. Transport cost instead evaluates token-level correspondences under marginal constraints and incorporates spatial distance.
Controlled comparison. On Qwen3-VL-8B at 10% token retention, we keep the importance-aware masses, OT matching, compression procedure, and total token budget fixed, changing only the frame-pair allocation rule. We compare uniform allocation, pooled-frame dissimilarity, and pooled-frame dissimilarity computed after spatial pruning.Both similarity-based alternatives use cosine dissimilarity between mean-pooled frame features, with a budget temperature of 0.3.
| Allocation rule | MVBench | VideoMME | LongVideoBench | MLVU | VQA Avg. (%) | Charades-TimeLens | ActivityNet-TimeLens | VTG Avg. (%) |
|---|---|---|---|---|---|---|---|---|
| Uniform | 60.0 | 60.0 | 58.8 | 60.7 | 94.3 | 42.6 | 39.1 | 87.3 |
| Frame similarity | 60.2 | 60.0 | 58.8 | 60.8 | 94.4 | 42.9 | 38.9 | 87.4 |
| Frame similarity (after pruning) | 60.1 | 59.8 | 59.2 | 60.3 | 94.3 | 42.3 | 39.0 | 86.9 |
| Transport cost | 59.9 | 60.4 | 59.5 | 61.1 | 94.8 | 43.0 | 39.5 | 88.2 |
The gains are modest, but transport-derived allocation achieves the highest average retention on both tasks and outperforms both similarity-based alternatives on five of the six benchmarks. Computing pooled similarity after spatial pruning does not improve aggregate retention, suggesting that the advantage is not explained solely by using the retained token set.
Where allocation helps. Appendix C reports the per-video coefficient of variation of frame-pair budgets, . MVBench and Charades-TimeLens have the lowest median CVs (0.095 and 0.113), indicating nearly uniform allocations. Relative to pooled-frame dissimilarity, transport-derived allocation provides no gain on MVBench and only a small gain on Charades-TimeLens. LongVideoBench and ActivityNet-TimeLens have higher CVs (0.174 and 0.234), with gains of 0.6 accuracy points and 0.6 mIoU points, respectively. This pattern is consistent with adaptive allocation being more useful when alignment difficulty varies across frame pairs.
[W4, Q3]. Targeted evaluation of persistence, boundaries, and duration
We agree with the reviewer that temporal-grounding mIoU scores alone are insufficient to substantiate our claims about preserving persistence, duration, and event boundaries. We therefore examine persistence-related question subsets and separately measure boundary and duration errors at 10% token retention.
Persistence-related reasoning. On Qwen3-VL-8B, we evaluate two subsets defined by the benchmarks’ original task labels: object existence, object shuffle, and state change in MVBench. Object tracking and attribute change in LongVideoBench. These questions assess reasoning related to persistence, rather than directly measuring persistence preservation.
Each column reports accuracy retention (%) relative to Vanilla on the same subset. Vanilla correctly answers 370/600 MVBench questions and 152/270 LongVideoBench questions.
| Method | MVBench (600 questions) | LongVideoBench (270 questions) |
|---|---|---|
| FastVID | 91.9 | 95.4 |
| UniComp | 84.3 | 94.1 |
| ST-SimDiff | 86.2 | 95.4 |
| SemVID | 91.1 | 96.7 |
| OTT-Vid | 98.9 | 98.0 |
OTT-Vid achieves the highest subset retention on both benchmarks, correctly answering 366/600 and 149/270 questions, respectively. We therefore interpret these results as supportive evidence for persistence-related reasoning.
Boundary localization and duration estimation. We additionally decompose grounding errors on Charades-TimeLens and ActivityNet-TimeLens into two metrics in Qwen3-VL 10% retention setting.
- Boundary error: the mean absolute error of the predicted start and end times, in seconds.
- Duration error: the absolute log ratio of predicted to ground-truth interval length. Each cell reports Charades-TimeLens / ActivityNet-TimeLens. Lower is better; bold indicates the best compressed result.
| Method | Boundary error (s) | Duration error |
|---|---|---|
| Vanilla | 2.88 / 19.90 | 0.497 / 0.784 |
| FastVID | 4.44 / 26.70 | 0.608 / 0.919 |
| UniComp | 4.94 / 31.16 | 0.587 / 0.879 |
| ST-SimDiff | 4.44 / 26.53 | 0.609 / 0.881 |
| SemVID | 4.02 / 25.08 | 0.611 / 0.832 |
| OTT-Vid | 3.38 / 23.75 | 0.539 / 0.799 |
Together, these analyses provide targeted evidence supporting our motivation: persistence-related question performance complements direct measurements of boundary localization and duration estimation, going beyond aggregate temporal-grounding mIoU.
[W1]. Contribution beyond individual components
We appreciate the reviewer's assessment and the opportunity to clarify the contribution beyond the individual components. The central principle of OTT-Vid is to make inter-frame matching account for the loss incurred by a token's own frame when that token is compressed. To our knowledge, no prior Video-LLM token compression method builds this loss into inter-frame matching itself.
Why this principle matters. A strong cross-frame match indicates that similar content exists elsewhere, but does not establish that removing the token leaves its own frame adequately represented. This distinction matters when the answer depends on when content is present, rather than merely whether it appears somewhere in the video. OTT-Vid incorporates the estimated loss of within-frame representativeness directly into inter-frame matching through each token's leave-one-out contribution to saliency-weighted frame coverage.
A unified formulation. We encode this contribution inversely in the transport marginals. Tokens whose removal causes greater estimated coverage loss receive smaller masses, which limits their coupling values and reduces their compression priority. OT jointly coordinates correspondences within each frame pair under these marginal constraints. The transport cost of the resulting plan then serves as a proxy for alignment difficulty, assigning larger compression budgets to easier frame pairs. Token-level compression priorities and frame-pair budget allocation thus derive from the same transport plan.
Controlled evidence. Table 4 isolates these choices on Qwen3-VL-8B at 10% token retention. With other components fixed, LOO-based masses improve VQA/VTG retention by 1.2/1.4 percentage points, globally coordinated OT matching improves over independent Local Gibbs matching by 0.9/2.0 points, and transport-derived budgets add a further 0.5/0.9 points. These controlled comparisons support each part of the formulation, with consistently larger gains on temporal grounding. Our response to W4 provides targeted analyses of persistence-related reasoning, boundary localization, and duration estimation.
[W2, Q1]. Raw versus marginal-normalized coupling ranking
The marginal dependence of raw couplings is an intended part of our compression criterion. OTT-Vid assigns smaller masses to tokens with greater within-frame importance, thereby limiting their compression scores.
Why normalization can be counterproductive. Source normalization changes the score from transported mass, Pij, to the fraction of a source's mass assigned to a destination, Pij/bj. A small-mass, important token can receive a high normalized score when most of its mass matches one destination. This can give it higher compression priority despite its importance.
Theoretical justification. Appendix G formalizes this effect in an n×n setting with uniform destination masses and one preferred destination per source. As ε→0, the maximum raw and source-normalized scores approach min(bj,1/n) and min(1,1/(nbj)), respectively. For bj<1/n<bk, normalization therefore prioritizes the more important source j for compression, reversing the raw ranking. Proposition G.1 provides an explicit sufficient condition for this reversal at positive ε. Both-marginal normalization yields the same ordering in this setting.
Controlled evaluation. We changed only the ranking scores, keeping the transport computation, transport-derived budgets, and compression procedure unchanged (Qwen3-VL-8B, 10% token retention).
Both normalized alternatives reduce retention on both tasks. Together, the theoretical example and controlled results support retaining the marginal weighting in compression ranking.