acceptodds
Under review as a conference paper at ICLR 2027

OTT-Vid: Optimal Transport Temporal Token Compression for Video-LLMs

Abstract

As Video Large Language Models (Video-LLMs) scale to longer and more complex videos, their inference cost grows rapidly due to visual tokens accumulated across frames. Training-free token compression reduces this overhead in part by exploiting repetition across frames, often using cross-frame similarity to identify redundant tokens or group similar frames for compression. However, cross-frame similarity and segmentation alone do not capture each token's importance within its frame or directly determine how much compression to apply to each frame pair. In this work, we propose OTT-Vid, a training-free outer-LLM compression framework that incorporates intra-frame token importance into temporal matching and derives frame-pair budgets from the resulting transport difficulty. After spatial pruning retains representative tokens, we formulate optimal transport (OT) between neighboring frames using importance-aware masses and a locality-aware cost combining feature dissimilarity with spatial distance. Smaller masses limit the coupling values of important tokens, reducing their priority in compression candidate ranking. The transport plan guides token-level compression, while its total cost serves as a proxy for compression difficulty, assigning larger compression budgets to lower-cost frame pairs. Experiments on six benchmarks spanning video question answering and temporal grounding across four backbones demonstrate strong performance at multiple token retention ratios, with gains in temporal grounding supporting the preservation of temporal evidence under compression.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video large language models, token compression, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zdgLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OTT-Vid, a training-free framework that uses importance-aware optimal transport to guide temporal token compression and allocate compression budgets across adjacent frames. Evaluation on four Video-LLM backbones and six benchmarks shows improved performance retention over competing compression methods and inference speedups over uncompressed models.

Critical strengths
  1. Coherent compression framework. The method connects importance-aware temporal matching and frame-pair budget allocation through a shared transport formulation, addressing both which tokens to compress and where to allocate compression.
  2. Broad empirical validation. Evaluation across four backbones, six benchmarks, and multiple retention ratios demonstrates consistent performance retention and practical inference speedups. Including temporal grounding makes the evaluation particularly relevant to preserving temporal evidence.
Critical weaknesses
  1. Limited conceptual novelty. The method appears to be a well-engineered combination of existing heuristics, rather than a sufficiently distinct methodological advance over prior compression approaches.
  2. Insufficient justification of coupling-based ranking. Raw OT couplings depend on both marginal masses and matching costs, but their advantage over marginal-normalized scores is not established.
  3. Unvalidated compression-difficulty proxy. The paper does not directly establish that transport cost predicts compression-induced degradation or outperforms simpler similarity-based budget allocation.
  4. Indirect evidence for temporal preservation. Aggregate temporal-grounding results do not isolate the claimed preservation of persistence, duration, and event boundaries. Targeted evaluation is needed to substantiate this central motivation.
  5. Incomplete baseline coverage. Standalone comparisons with DivPrune (CVPR 2025), FLoC (ICLR 2026), STTM (ICCV 2025), PruMerge (ICCV 2025), and PruneVid (ACL 2025) are needed.
  6. Limited benchmark coverage. Additional evaluation on LVBench, EgoSchema, and NExT-QA would strengthen the evidence for generalization, particularly for long-video understanding and temporal reasoning.
Questions for authors
  1. Clarification: Coupling ranking. Why are raw OT couplings appropriate for ranking compression candidates, given their dependence on marginal masses? Have marginal-normalized alternatives been evaluated?
  2. Evidence request: Budget allocation. Does transport cost correlate with actual compression-induced performance degradation, and does it outperform simpler similarity-based allocation signals?
  3. Evidence request: Temporal preservation. Can the authors provide targeted results on persistence, duration, or event-boundary questions to substantiate the central motivation?
  4. Additional evaluation request. Can the authors provide matched-budget comparisons with standalone DivPrune, FLoC, PruMerge, and PruneVid, and results on LVBench, EgoSchema, and NExT-QA? These would help assess whether the reported advantages generalize beyond the current comparisons.
7 comments

[W1]. Contribution beyond individual components

We appreciate the reviewer's assessment and the opportunity to clarify the contribution beyond the individual components. The central principle of OTT-Vid is to make inter-frame matching account for the loss incurred by a token's own frame when that token is compressed. To our knowledge, no prior Video-LLM token compression method builds this loss into inter-frame matching itself.

Why this principle matters. A strong cross-frame match indicates that similar content exists elsewhere, but does not establish that removing the token leaves its own frame adequately represented. This distinction matters when the answer depends on when content is present, rather than merely whether it appears somewhere in the video. OTT-Vid incorporates the estimated loss of within-frame representativeness directly into inter-frame matching through each token's leave-one-out contribution to saliency-weighted frame coverage.

A unified formulation. We encode this contribution inversely in the transport marginals. Tokens whose removal causes greater estimated coverage loss receive smaller masses, which limits their coupling values and reduces their compression priority. OT jointly coordinates correspondences within each frame pair under these marginal constraints. The transport cost of the resulting plan then serves as a proxy for alignment difficulty, assigning larger compression budgets to easier frame pairs. Token-level compression priorities and frame-pair budget allocation thus derive from the same transport plan.

Controlled evidence. Table 4 isolates these choices on Qwen3-VL-8B at 10% token retention. With other components fixed, LOO-based masses improve VQA/VTG retention by 1.2/1.4 percentage points, globally coordinated OT matching improves over independent Local Gibbs matching by 0.9/2.0 points, and transport-derived budgets add a further 0.5/0.9 points. These controlled comparisons support each part of the formulation, with consistently larger gains on temporal grounding. Our response to W4 provides targeted analyses of persistence-related reasoning, boundary localization, and duration estimation.


[W2, Q1]. Raw versus marginal-normalized coupling ranking

The marginal dependence of raw couplings is an intended part of our compression criterion. OTT-Vid assigns smaller masses to tokens with greater within-frame importance, thereby limiting their compression scores.

Why normalization can be counterproductive. Source normalization changes the score from transported mass, PijP_{ij}, to the fraction of a source's mass assigned to a destination, Pij/bjP_{ij}/b_j. A small-mass, important token can receive a high normalized score when most of its mass matches one destination. This can give it higher compression priority despite its importance.

Theoretical justification. Appendix G formalizes this effect in an n×nn\times n setting with uniform destination masses and one preferred destination per source. As ε→0\varepsilon\to0, the maximum raw and source-normalized scores approach min⁡(bj,1/n)\min(b_j,1/n) and min⁡(1,1/(nbj))\min(1,1/(nb_j)), respectively. For bj<1/n<bkb_j<1/n<b_k, normalization therefore prioritizes the more important source jj for compression, reversing the raw ranking. Proposition G.1 provides an explicit sufficient condition for this reversal at positive ε\varepsilon. Both-marginal normalization yields the same ordering in this setting.

Controlled evaluation. We changed only the ranking scores, keeping the transport computation, transport-derived budgets, and compression procedure unchanged (Qwen3-VL-8B, 10% token retention).

Ranking scoreVQA retention (%)VTG retention (%)
Raw coupling PijP_{ij}94.888.2
Source-normalized Pij/bjP_{ij}/b_j92.684.9
Both-marginal-normalized Pij/(aibj)P_{ij}/(a_i b_j)92.484.3

Both normalized alternatives reduce retention on both tasks. Together, the theoretical example and controlled results support retaining the marginal weighting in compression ranking.


[W3, Q2]. Transport cost as a budget-allocation signal

Why transport cost. Our choice follows directly from the OT objective. The transport plan is optimized to minimize token-level alignment cost under importance-aware marginal constraints, with entropy regularization. The resulting cost, Wt=∑i,jPt,ijCt,ijW_t=\sum_{i,j}P_{t,ij}C_{t,ij}, quantifies alignment difficulty using the same plan and cost matrix that guide compression. We use it as a proxy for compressibility, assigning more compression to lower-cost frame pairs.

Our manuscript does not claim an established correlation between transport cost and task degradation. Directly establishing that relationship is challenging because the effect of compression depends on the question, the existing allocation, and the amount of additional compression. We instead assess its practical utility through controlled comparisons with alternative allocation signals.

Why compare with pooled similarity. Frame-level similarity is used for temporal segmentation in FastVID and frame grouping and allocation in UniComp. Motivated by these approaches, we use pooled-frame dissimilarity as a simpler allocation signal. Pooling can capture broad changes in frame content, such as scene transitions, but local changes may be diluted or canceled in the frame average. Transport cost instead evaluates token-level correspondences under marginal constraints and incorporates spatial distance.

Controlled comparison. On Qwen3-VL-8B at 10% token retention, we keep the importance-aware masses, OT matching, compression procedure, and total token budget fixed, changing only the frame-pair allocation rule. We compare uniform allocation, pooled-frame dissimilarity, and pooled-frame dissimilarity computed after spatial pruning.Both similarity-based alternatives use cosine dissimilarity between mean-pooled frame features, with a budget temperature of 0.3.

Allocation ruleMVBenchVideoMMELongVideoBenchMLVUVQA Avg. (%)Charades-TimeLensActivityNet-TimeLensVTG Avg. (%)
Uniform60.060.058.860.794.342.639.187.3
Frame similarity60.260.058.860.894.442.938.987.4
Frame similarity (after pruning)60.159.859.260.394.342.339.086.9
Transport cost59.960.459.561.194.843.039.588.2

The gains are modest, but transport-derived allocation achieves the highest average retention on both tasks and outperforms both similarity-based alternatives on five of the six benchmarks. Computing pooled similarity after spatial pruning does not improve aggregate retention, suggesting that the advantage is not explained solely by using the retained token set.

Where allocation helps. Appendix C reports the per-video coefficient of variation of frame-pair budgets, CV=σ(Bt)/μ(Bt)\mathrm{CV}=\sigma(B_t)/\mu(B_t). MVBench and Charades-TimeLens have the lowest median CVs (0.095 and 0.113), indicating nearly uniform allocations. Relative to pooled-frame dissimilarity, transport-derived allocation provides no gain on MVBench and only a small gain on Charades-TimeLens. LongVideoBench and ActivityNet-TimeLens have higher CVs (0.174 and 0.234), with gains of 0.6 accuracy points and 0.6 mIoU points, respectively. This pattern is consistent with adaptive allocation being more useful when alignment difficulty varies across frame pairs.

[W4, Q3]. Targeted evaluation of persistence, boundaries, and duration

We agree with the reviewer that temporal-grounding mIoU scores alone are insufficient to substantiate our claims about preserving persistence, duration, and event boundaries. We therefore examine persistence-related question subsets and separately measure boundary and duration errors at 10% token retention.

Persistence-related reasoning. On Qwen3-VL-8B, we evaluate two subsets defined by the benchmarks’ original task labels: object existence, object shuffle, and state change in MVBench. Object tracking and attribute change in LongVideoBench. These questions assess reasoning related to persistence, rather than directly measuring persistence preservation.

Each column reports accuracy retention (%) relative to Vanilla on the same subset. Vanilla correctly answers 370/600 MVBench questions and 152/270 LongVideoBench questions.

MethodMVBench (600 questions)LongVideoBench (270 questions)
FastVID91.995.4
UniComp84.394.1
ST-SimDiff86.295.4
SemVID91.196.7
OTT-Vid98.998.0

OTT-Vid achieves the highest subset retention on both benchmarks, correctly answering 366/600 and 149/270 questions, respectively. We therefore interpret these results as supportive evidence for persistence-related reasoning.

Boundary localization and duration estimation. We additionally decompose grounding errors on Charades-TimeLens and ActivityNet-TimeLens into two metrics in Qwen3-VL 10% retention setting.

  • Boundary error: the mean absolute error of the predicted start and end times, in seconds.
  • Duration error: the absolute log ratio of predicted to ground-truth interval length. Each cell reports Charades-TimeLens / ActivityNet-TimeLens. Lower is better; bold indicates the best compressed result.
MethodBoundary error (s)Duration error
Vanilla2.88 / 19.900.497 / 0.784
FastVID4.44 / 26.700.608 / 0.919
UniComp4.94 / 31.160.587 / 0.879
ST-SimDiff4.44 / 26.530.609 / 0.881
SemVID4.02 / 25.080.611 / 0.832
OTT-Vid3.38 / 23.750.539 / 0.799

Together, these analyses provide targeted evidence supporting our motivation: persistence-related question performance complements direct measurements of boundary localization and duration estimation, going beyond aggregate temporal-grounding mIoU.

Reviewer_kMxEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free token compression framework for video understanding named OTT-Vid. Specifically, OTT-Vid consists of three core components: Saliency-weighted Spatial Compression, Importance-aware Temporal Optimal Transport, and Transport-derived Budget Allocation. The Saliency-weighted Spatial Compression and Importance-aware Temporal Optimal Transport modules make compression decisions on which visual tokens to merge, retain, or discard. The Transport-derived Budget Allocation module dynamically adapts the pairwise compression budgets to assign more compression operations to frame pairs with lower transport difficulty.

The experimental evaluation in this paper assesses the effectiveness of OTT-Vid across multiple benchmark datasets, including MVBench, VideoMME, LongVideoBench, MLVU, Charades-TimeLens, and ActivityNet-TimeLens. Results demonstrate that OTT-Vid achieves substantially improved inference efficiency and high performance retention, outperforming several training-free token compression methods.

Critical strengths
  • The idea of converting intra-frame representativeness into compression priorities through leave-one-out coverage loss is considered novel.
  • The authors’ focus on linking intra-frame importance, cross-frame matching, and compression budget allocation is reasonable.
Critical weaknesses
  • In Eq. (2), the importance score measures how much saliency-weighted feature coverage within the current frame would be lost if a token were removed from its representative set. This does not seem fully equivalent to preserving the evidence needed to infer duration, continuity, and event boundaries. This correspondence may not hold when a change has little effect on visual features or scene salience but is nevertheless crucial for determining whether an action has been completed.

  • The paper independently computes OT plans for all adjacent frame pairs based on the initial spatially pruned token sets. It then connects the accepted merge edges into components and uses the mean of the features in each component as a single output token. However, passing local matching checks does not guarantee that the final merging across multiple frames remains reasonable. Repeated small changes may accumulate into substantial semantic changes.

  • Before temporal OT begins, the authors independently compress each frame to the same number of tokens, then adaptively allocate the temporal compression budget across adjacent frame pairs. However, information density can vary across frames. Even if the subsequent budget allocation mechanism determines that less compression should be applied around key frames, it can only preserve the content that remains after the spatial stage. Errors introduced by the initial spatial selection cannot be corrected by temporal period.

Questions for authors
  • How do the authors handle positional encoding after compression? How are positional encodings handled when the token arrangement is modified in a training-free setting?
  • Have the authors considered comparing against and discussing more vision-side training-free compression methods, such as LLaVA-Scissor [1] and VisionZip [2]?

[1] LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

[2] VisionZip: Longer is Better but Not Necessary in Vision Language Models

5 comments

[W1]. LOO-based importance and temporal evidence preservation

As discussed in the Introduction, OTT-Vid uses within-frame coverage as a complementary signal to cross-frame similarity, and we agree that it is not equivalent to temporal importance. The LOO-based score estimates how much coverage would be lost if a token were removed, allowing temporal compression to consider both cross-frame similarity and each token's contribution to its frame.

Targeted evidence. We refer the reviewer to our response to Reviewer zdgL [W4, Q3] for separate analyses of persistence-related reasoning, boundary localization, and duration estimation. At 10% token retention, OTT-Vid achieves the highest persistence-subset retention on both MVBench and LongVideoBench with Qwen3-VL-8B, and the lowest boundary and duration errors among the compared compressed methods on both grounding benchmarks. These results provide evidence of temporal preservation. Nevertheless, we agree that a visually subtle change can be decisive for judging whether an action has been completed. This case is challenging for query-agnostic outer-LLM compression, including OTT-Vid, which relies on visual signals such as LOO-based coverage and cross-frame similarity to determine what to preserve.

Complementarity with inner-LLM pruning. Inner-LLM pruning can condition on the question and may therefore partly mitigate such cases. One way to combine the two is to retain more tokens before the LLM and defer part of the compression to the inner stage. In Table A8, we combine OTT-Vid with FlashVID's inner-LLM pruning [1]. OTT-Vid initially retains 12.5% of the visual tokens, and FlashVID subsequently prunes tokens within the decoder. The inner stage matches the layer-average visual-token retention of the outer-only 10% setting, providing a comparable computation budget while allowing more tokens to reach the LLM before further selection. Each cell in the table below reports VQA / VTG performance retention (%).

ConfigurationQwen3-VL-8BQwen2.5-VL-7B
OTT-Vid, outer only (10%)94.8 / 88.295.4 / 73.9
OTT-Vid (12.5%) + FlashVID inner pruning96.0 / 90.496.4 / 78.4

The hybrid improves retention on both tasks and backbones, supporting the complementarity of outer and inner compression. Although these results do not directly evaluate the subtle-change cases described above, they motivate retaining more visual evidence until query-aware selection can be applied. We view this as a promising direction for addressing such cases, while recognizing that inner pruning cannot recover evidence already removed by the outer stage.

[1] FlashVID: Efficient Video Large Language Models via Training-Free Tree-Based Spatiotemporal Token Merging. ICLR, 2026.

[W2]. Accumulated change across chained merges

Thank you for this observation. We agree that passing local matching checks does not guarantee that the final merge across multiple frames remains reasonable. To address the concern directly, we measured how often accumulated change occurs and evaluated an ablation that guards against it, on Qwen3-VL-8B at 10% token retention over the six benchmarks.

How often it occurs. As a diagnostic of accumulated change, we call a merged token drifted when the matching cost used for merging, with its feature and position terms, exceeds the merge threshold τ_c between the token and the root of its component. Overall, 11.0% of merged tokens are drifted, and the components containing them account for 3.0% of the output tokens. Tokens with smaller LOO-based masses, which have higher estimated coverage loss, drift less often. Deciles are formed within each frame.

LOO-based mass decile (1 = smallest, 10 = largest)12345678910
Drifted among merged tokens (%)4.06.87.99.010.011.011.912.713.414.5

Root-consistency guard. Frame pairs are processed in temporal order, so the root of a component is already fixed when a source token is checked against it. Without re-solving OT, the guard rejects a merge when the source exceeds τ_c against the original feature and position of that root, which leaves no drifted token by construction. The rejected token is kept and the next candidate of the same frame pair is compressed instead.

MethodMVBenchVideoMMELongVideoBenchMLVUCharades-STAActivityNetVQA ret. (%)VTG ret. (%)
OTT-Vid59.960.459.561.143.039.594.888.2
+ root guard60.060.558.361.343.239.494.588.3

VQA benchmarks report accuracy and grounding benchmarks report mIoU. The guard has mixed effects across benchmarks. Mean VQA retention decreases by about 0.3 points, and mean VTG retention increases by 0.1 points. Two observations help contextualize this result.

  1. Tokens with high estimated coverage loss receive small transport masses. Only 1.1% of the smallest-mass decile is merged into another token, and this decile also shows the lowest drift rate.
  2. Under a fixed budget, each rejected merge is replaced by a candidate that the transport plan ranks lower. Rejected merges amount to 3.6% to 7.2% of the temporal compression budget across the six benchmarks.

These results confirm that accumulated mismatch can occur, as the reviewer noted. However, eliminating the measured mismatch changes average retention only slightly at the same token budget, suggesting that its impact on aggregate performance is limited in the evaluated setting.

[W3]. Uniform spatial budget before temporal compression

We appreciate the reviewer’s concern about assigning the same spatial budget to frames with different compression needs. To assess whether accounting for these differences improves performance, we evaluated two adaptive allocations of the per-frame token count KtK_t on Qwen3-VL-8B at 10% token retention.

  • Effective rank: We allocate K_t proportional to exp(H_t), where H_t is the entropy of the normalized Gram-matrix eigenvalues of each frame’s centered token features. Frames with greater feature diversity therefore retain more tokens.
  • Frame uniqueness: Following UniComp’s [2] allocation rule, we compute each frame’s mean cosine dissimilarity to all frames using mean token features, and allocate K_t via a softmax of the centered scores scaled by √T.

Both allocations preserve the total spatial token budget and respect per-frame capacity. We retain the same within-frame selection rule and temporal compression algorithm.

Spatial allocationMVBenchVideoMMELongVideoBenchMLVUCharades-STAActivityNetVQA ret. (%)VTG ret. (%)
Uniform (OTT-Vid)59.960.459.561.143.039.594.888.2
Effective rank60.260.458.561.043.139.694.588.4
Frame uniqueness60.260.357.761.242.435.694.283.3

These results show no consistent improvement from these allocations. Two factors may contribute to this outcome:

Effective rank and frame uniqueness may not reliably identify key frames that require less compression, and non-uniform per-frame token counts due to adaptive spatial pruning can affect LOO-based mass assignment and transport-derived temporal budgets. While these findings do not support directly adopting either allocation in our current framework, we consider adaptive spatial allocation a promising direction and plan to explore its joint design with temporal compression to better preserve key-frame evidence.

[2] Unicomp: Rethinking video compression through informational uniqueness. CVPR 2026


[Q1]. Positional encoding after compression

Retained tokens preserve their original position identifiers. For backbones with explicit temporal and spatial coordinates, these coordinates are retained, while a merged token inherits those of its component root. We use the backbone’s native positional encoding without introducing new positional parameters.

For rotary encodings such as M-RoPE, retaining the original position identifiers preserves the positional rotations assigned to surviving tokens rather than renumbering them according to the compressed sequence. A merged token is represented at its root’s position. This allows us to reuse the pretrained positional encoding without additional training, although merging does not preserve a separate positional representation for every constituent token.

Reviewer_VSEuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free framework for visual token reduction in video large language models (Video LLMs), with optimal transport (OT) as its core mechanism. The method has three stages. First, at the intra-frame (spatial) level, the authors score visual tokens with two complementary criteria. A saliency metric measures the importance of each token, and a coverage metric encourages the retained tokens to represent the diverse content of the frame. Together they drive spatial token compression within each frame. Second, at the inter-frame (temporal) level, the paper models adjacent frame pairs as an OT problem. It computes token masses and pairwise transport costs between neighboring frames and converts the result into a “transport difficulty” score, which reflects how much a frame’s content changes relative to its neighbor. This score determines a dynamic, frame-adaptive token budget. Frames with more novel or changing content receive more tokens, while static or redundant frames are compressed more aggressively, and the overall token budget stays fixed. Third, the method builds a graph over the retained tokens and merges redundant ones by similarity, so that their information is aggregated rather than discarded outright. Because the framework requires no additional training, it can be plugged into different Video LLMs. The authors evaluate it on four widely used video question answering benchmarks [benchmark names] and two video grounding benchmarks [benchmark names], using several representative backbones including Qwen2.5-VL, Qwen3-VL, InternVideo3, and InternVL3.5. Across different token retention ratios, they report [competitive / superior] performance compared with existing token reduction methods.

Critical strengths
  1. The evaluation benchmarks are extensive, and the token reduction results sound very competitive. Meanwhile, this paper applies its method to a series of advanced Video LLMs or MLLMs, which showcases its effectiveness.

  2. The paper writing and presentation are clear and easy to follow.

  3. This paper also makes comparisons on latency or memory savings with existing vanilla models.

Critical weaknesses
  1. The main concern of this paper lies in the limited novelty of optimal transport utility on this task. There are some recent papers that have explored the optimal transport mechanism to address visual token reduction for VideoLLM, like [1] and [2]. This raises a significant limitation of this paper's novelty, as there are very few new insights or in-depth analysis/findings of this framework. I think the authors need to clearly and in detail explain and clarify the essential differences from these works.

    [1] OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport, CVPR2026

    [2] Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models, CVPR2026

  2. I am confused about whether the authors apply some existing works, like FastVID, SemVID, etc., to the very recently advanced Video LLMs? This needs to be clarified to ensure fair comparisons.

  3. There are common comparisons on VQA benchmarks built on the LLaVA-Video model by abating the token budget; why not include a main table like this in the main paper?

  4. Since the optimal transport method is very similar to [1] and [2], I am also curious about the visualization of samples in terms of the removed and kept visual tokens among the intra-frame and inter-frame comparisons with these existing works.

  5. In the Compression execution section, the authors state the union-find method to merge the token, what if removing the optimal transport with graph-based? So the complex optimal transport does not need to be leveraged?

Questions for authors
  1. I think the authors need to explain in detail the essential differences from existing work.

  2. Some confusing experiments, setups, or further visualization comparisons need to be presented and demonstrated.

  3. I would suggest putting a main table of Video LLaVA comparisons in the main paper.

4 comments

[W1, Q1]. Novelty relative to OTPrune and AOT

We appreciate the reviewer’s emphasis on distinguishing OTT-Vid from prior OT-based compression methods, as this is central to clarifying the novelty of our formulation. OT has already been explored for visual token compression in OTPrune and AOT. Our contribution lies in what inter-frame matching accounts for, rather than in the use of OT itself.

How OTPrune and AOT use OT. OTPrune selects a token subset that minimizes the Wasserstein distance to the full token distribution, using a log-determinant surrogate with uniform token masses within a single image. AOT uses uniform-marginal OT to aggregate tokens into attention-selected anchors that are updated across frames. Token importance is used only to select the anchors and does not enter the transport problem.

The problem we address. Token similarity-based temporal compression can preserve visual content while weakening evidence of when that content was present. That evidence is carried by each frame's own tokens, and once a token is merged into a neighboring frame, the LLM receives no explicit indication that the source frame has changed. A good feature match therefore does not establish that the source frame remains adequately represented. This distinction matters for tasks such as temporal grounding, where recognizing an action and locating its occurrence are different requirements. OTT-Vid thus distinguishes cross-frame substitutability from within-frame dispensability. Alongside matching similarity, it considers the estimated loss of saliency-weighted coverage within a token's own frame if that token is removed.

What is new. To our knowledge, OTT-Vid is the first Video-LLM token compression method whose inter-frame matching itself accounts for how much compressing each token degrades the representation of its own frame. Whereas AOT keeps transport marginals uniform, we derive non-uniform marginals from each token's leave-one-out contribution to saliency-weighted frame coverage.

Why OT. OT is the means of realizing this principle. Its objective weighs token masses and matching costs together under marginal constraints, and the transport cost of the resulting plan measures how difficult a frame pair is to align. We use this cost to assign more compression to lower-cost pairs, so the same plan determines both which tokens are compressed and how the temporal budget is distributed.

Evidence on temporal grounding. The problem above is why we evaluate VTG in addition to VQA. Strong VQA retention does not imply that temporal evidence is preserved. At 10% token retention, averaged over the four backbones in Tables 1 and 2, FastVID retains 93.1% of VQA performance but only 61.6% of VTG performance, whereas OTT-Vid attains the highest retention on both tasks on every backbone (94.7% and 85.3% on average). Under identical OT matching and uniform budget allocation, replacing uniform masses with LOO-based masses improves VQA/VTG retention from 93.1%/85.9% to 94.3%/87.3% (Table 4, rows c and e).

We have revised the Related Work section to include a dedicated subsection on OT-based visual token compression (Sec. 2.3), introducing OTPrune and AOT and clarifying how our use of OT differs.


[W2, Q2]. Baseline setup and fairness of comparison

Yes. All baseline results in Tables 1 and 2 were obtained by running each method on the target backbones ourselves. The implementation details were in Appendix A, but the main text did not point to them when introducing the compared methods, which could cause this confusion. We have revised the main text accordingly.

Shared across methods. Within each backbone and benchmark, all methods use the same sampled frames, prompts, decoding settings, scoring, and target retention ratios, defined as final visual tokens relative to the same uncompressed input. We adapt backbone interfaces and budget controls to the common final-token target.

Released implementations and settings. We start from each method's released implementation and use the authors' settings, including backbone- and task-specific profiles where provided, without additional tuning on our evaluation benchmarks. OTT-Vid uses one shared hyperparameter configuration across backbones.

  • FastVID. DySeg_c=8, DySeg_tau=0.84, DySeg_ignore=0.95, STPrune_d=0.4, DTM_p=4, DTM_beta=0.6, from the authors' Qwen2.5-VL implementation.
  • UniComp. Frame and spatial uniqueness thresholds of 0.005 and 0.2.
  • ST-SimDiff. Released new_topk strategy-3, with similarity threshold 0.8 and event upper bound 0.2.
  • SemVID. Released Qwen3 profile (object ratio 0.6, MMR coefficient 0.9). The Qwen2.5-VL profiles are listed in Appendix A.

Method-specific adaptations, such as inputs unavailable on a backbone, are documented in Appendix A.

[W3, Q3]. Evaluation on LLaVA-Video-7B

Thank you for this suggestion. Our main experiments focused on backbones capable of temporal grounding since our method targets the preservation of temporal evidence. We agree that comparison on LLaVA-Video is important and have added the following results to the main paper as Table 3.

Expanded evaluation. We evaluate LLaVA-Video-7B with 64 frames at a nominal 10% compression budget. All methods were evaluated using implementations based on the authors’ released code and configurations. Beyond the original four VQA benchmarks, we include NExT-QA (multiple-choice test), LVBench, and EgoSchema (subset) in response to the Reviewer zdgL’s requests. We also include AOT and OTPrune to compare with prior OT-based compression methods.

MethodMVBenchVideoMMELongVideoBenchMLVUNExT-QALVBenchEgoSchemaAvg. retention (%)
Vanilla61.964.360.068.581.041.758.8100.0
FastVID59.960.956.863.577.139.356.695.0
UniComp56.459.257.764.874.938.253.492.7
ST-SimDiff56.658.254.862.177.338.354.392.0
SemVID59.259.056.764.177.038.952.293.2
AOT59.661.254.562.478.139.456.694.4
OTPrune55.956.956.559.376.637.156.691.3
OTT-Vid59.460.955.464.777.039.556.094.6

Scores are accuracy (%). Avg. retention is the mean of the seven benchmark-level ratios to Vanilla, calculated before rounding. Bold indicates the best compressed result.

Results. OTT-Vid ranks second in average performance retention (94.6%), within 0.4 percentage points of FastVID (95.0%). It also exceeds AOT (94.4%) and OTPrune (91.3%) on average. These results demonstrate competitive general video understanding performance beyond the temporal-grounding-capable backbones used in our main experiments.

Extended comparison with OT-based methods. We further compare OTT-Vid with AOT and OTPrune on Qwen3-VL-8B at a nominal 10% token retention, covering 7 VQA and 2 VTG benchmarks. The baselines use implementations based on the authors’ released code and configurations, with backbone adaptations where required. VQA scores are accuracy (%) and VTG scores are mIoU (%).

MethodMVBenchVideoMMELongVideoBenchMLVUNExT-QALVBenchEgoSchemaVQA Avg. (%)
Vanilla65.863.760.664.281.338.771.8100.0
AOT60.460.357.460.278.837.165.694.2
OTPrune61.358.857.861.279.336.866.294.5
OTT-Vid59.960.459.561.179.237.465.895.0
MethodCharades-TimeLensActivityNet-TimeLensVTG Avg. (%)
Vanilla47.446.1100.0
AOT41.437.584.3
OTPrune42.438.386.2
OTT-Vid43.039.588.2

OTT-Vid achieves the highest average retention on both tasks: 95.0% for VQA and 88.2% for VTG. Its gains over AOT and OTPrune are larger on VTG, supporting the benefit of incorporating within-frame representativeness into temporal matching.


[W4, Q2]. Qualitative comparison with OT-based methods

To make the differences in token selection more concrete, we have added a qualitative comparison with AOT and OTPrune in Figure 5, accompanied by a new Section 4.4, “Qualitative Analysis.” Figure 5 compares the tokens retained (bright) and removed (dark) by the three methods on the same video with Qwen3-VL-8B at 10% token retention. Each column shows a frame, allowing comparison of both within-frame selection and its evolution across frames.

The clearest contrast appears in Frame 4 (yellow dashed circles), where a cup newly appears while the knife remains on the table. OTPrune distributes retained tokens broadly across the frame. AOT concentrates on the cup without retaining tokens on the knife, whereas OTT-Vid retains tokens on both objects.

Although the knife is visually similar across preceding frames, its continued presence remains part of what Frame 4 shows. This example illustrates the distinction discussed in our response to W1: cross-frame similarity does not establish within-frame dispensability. It provides a qualitative illustration of how OTT-Vid retains evidence of both newly appearing and persistent content.

[W5]. Comparison with a simpler graph-based alternative

Table 4 already includes a simpler graph-based alternative without OT, Local Gibbs. We apologize for not making its graph-based interpretation clear in the original manuscript.

Graph-based matching without OT. Local Gibbs constructs weighted edges from each source token to destination tokens using the same matching costs, importance-aware masses, and temperature as OTT-Vid (Eq. 9). Each source distributes its mass independently, without enforcing destination marginal constraints. Both methods rank compression candidates by their respective edge weights and execute selected merges using the same union-find procedure. The comparison therefore isolates the benefit of globally coordinating assignments through destination marginal constraints.

Controlled comparison and motivation for OT. With identical masses and uniform budget allocation, replacing Local Gibbs with OT improves VQA/VTG performance retention from 93.4%/85.3% to 94.3%/87.3% (Table 4, rows a and e), corresponding to gains of 0.9 and 2.0 percentage points. Unlike independent edge scoring, OT jointly coordinates all correspondences within a frame pair under both source and destination marginal constraints, accounting for competition among source tokens for shared destinations. These results support the benefit of this global coordination, particularly for preserving temporal-grounding performance.

We have revised the ablation discussion of Table 4 to explicitly identify Local Gibbs as an independent edge-based graph matching baseline without OT.

Dear Reviewer,

We sincerely appreciate the time and effort you have invested in reviewing our work. We understand that reviewing and discussing papers can be quite demanding, especially given your other academic and professional commitments.

As the discussion period is nearing its end, we wanted to kindly reach out once more. If you happen to have some time, we would greatly appreciate the opportunity to hear your thoughts on our responses and revised manuscript, and to further discuss our work with you. We would be happy to clarify any points that remain unclear or address any concerns that you may still have. We truly value your feedback and would welcome any further discussion if your schedule allows. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.