acceptodds
Under review as a conference paper at ICLR 2027

LTC: Risk-Gated Lagrangian Token Compression for Video Large Language Models

Abstract

Video Large Language Models (video-LLMs) process large numbers of visual tokens, incurring substantial computational and memory overhead. Existing training-free compression methods reduce spatiotemporal redundancy through importance-based selection, cross-frame matching, and structured aggregation. However, these methods neglect the local motion structure of neighboring correspondences, which may lead to the loss of key dynamic information during compression. Based on this insight, we propose risk-gated Lagrangian Token Compression (LTC), a training-free inference framework for video-LLMs. Specifically, Token Flow Estimation (TFE) uses local feature matching to estimate token displacements and matching confidence. Eulerian Risk Diagnosis (ERD) then combines neighborhood motion structure with active low-confidence signals to construct an aggregation-risk proxy. Guided by this risk proxy, Risk-Gated Lagrangian Aggregation (RGLA) keeps protected tokens as independent candidates, aggregates the remaining tokens that satisfy feature-consistency constraints into compact representatives along temporal trajectories, and selects the final visual tokens by importance under a token budget. The framework directly reuses pretrained visual features without training an additional network. Experiments on three representative video-LLMs and three video understanding benchmarks validate LTC's effectiveness and applicability across models. Retaining only 25% of visual tokens, LTC achieves 101.3% of the vanilla model's average performance on LLaVA-OneVision. Code is available at https://anonymous.4open.science/r/LTC-75E4.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video large language models, Visual token compression, Training-free inference, Token merging, Video understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eArAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents LTC, a training-free framework for visual token compression in video language models. It estimates local token correspondences and matching confidence, then uses neighborhood displacement structure and active low-confidence signals to identify tokens that should remain independent during aggregation. Eligible tokens are merged along estimated trajectories, followed by representative selection under a token budget. Experiments cover three backbones and three video understanding benchmarks, with additional component ablations, sensitivity analyses, and inference efficiency measurements.

Critical strengths
  1. A well-motivated distinction between token importance and aggregation risk. LTC recognizes that feature similarity alone may not determine whether tokens can be safely merged, particularly when motion structure and correspondence uncertainty carry information relevant to temporal understanding. Using local displacement patterns and matching uncertainty to guide merge protection before representative selection provides a coherent motivation for the framework. The training-free design is also a practical advantage.

  2. Competitive results supported by targeted ablations. On LLaVA-OneVision, LTC achieves 52.0% average accuracy at 25% token retention, slightly exceeding the 51.4% of the uncompressed model (Table 1). The component ablations, count-matched random gate, and comparisons with alternative risk statistics help clarify the contributions of individual design choices (Tables 4–5). The Qwen3-VL experiments further account for compression overhead and demonstrate reduced TTFT, providing initial evidence of practical efficiency.

Critical weaknesses
  1. Limited and inconsistent benefits from risk-aware aggregation. My main concern is whether the additional complexity of the proposed mechanism is sufficiently justified. Full LTC improves average accuracy over ERD-only selection by just 0.3 percentage points (Table 4), while the proposed divergence, curl, and shear statistic outperforms simple motion magnitude by only 0.2 points (Table 5). More importantly, Appendix C.3 shows that ERD alone achieves lower compression-induced error than full LTC in the two highest-risk quintiles. This observation raises questions about the effectiveness of the protection mechanism in precisely the challenging regions it aims to address. A clearer explanation of this behavior, together with uncertainty estimates, would help establish whether the proposed mechanism offers a consistent advantage over simpler alternatives.

  2. Insufficient evidence of a superior accuracy-efficiency tradeoff. The improvements over existing compression methods are not consistent across settings. For instance, on 32-frame LLaVA-Video at 25% token retention, LTC achieves 51.9% accuracy compared with 52.3% for FastVID (Table 6). More importantly, the latency and memory evaluations on Qwen3-VL only compare LTC against uncompressed inference, while comparisons with competing compression methods are conducted on the LLaVA backbones. Although the results demonstrate acceleration over vanilla inference, they do not establish whether LTC provides a more favorable accuracy-efficiency tradeoff than existing approaches under comparable conditions. A discussion of existing related video-LLM acceleration methods [1] would further strengthen your core contribution.

[1] Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior.

Questions for authors

Overall, I currently lean toward Weak Rejection. While the method is reasonably motivated and achieves competitive results, I remain concerned about its incremental benefits over simpler alternatives and its practical advantages over existing compression methods. Addressing the following questions with additional evidence could help improve my assessment.

  1. Could the authors further explain the high-risk behavior observed in Figure 6 and Appendix C.3? In particular, under what conditions does risk-aware aggregation help or hurt performance? Paired confidence intervals comparing LTC with ERD-only selection and simpler risk statistics would also help clarify the robustness of the reported gains.

  2. Could the authors provide a controlled comparison with the strongest compression baselines using the same backbone, input-frame count, and token budget? Reporting both accuracy and measured TTFT, including compression overhead, would help establish the practical value of LTC.

5 comments
AuthorsOpenReview ↗

Response to Reviewer eArA [1/2]

We sincerely appreciate your constructive feedback and your recognition of the motivation for distinguishing token importance from aggregation risk, the targeted ablations, and the practical potential of the training-free framework. Following your suggestions, we have further examined the robustness of the component gains, investigated the high-risk behavior, and expanded the controlled accuracy–efficiency comparisons. We have also supplemented the discussion of recent video-LLM acceleration methods. Our point-by-point response follows.

To Weakness 1 & Question 1: Incremental Benefits and High-Risk Behavior

Thank you for raising these concerns. Our additional analyses support modest positive accuracy gains from DCS and merge protection, with DCS requiring comparable measured risk-diagnosis time to motion magnitude. They also identify an allocation trade-off in high-risk videos: increasing the risk-category budget can reduce Q4–Q5 CIER, but may lower overall accuracy. We therefore do not claim uniform improvements over ERD-only selection in these groups.

For the ablations in Section 4.3, Table 4, and Appendix B.5, Table 6, we report dataset-stratified, paired video-level percentile bootstrap intervals. Videos are resampled within each benchmark, retaining all associated questions and sharing sampled IDs and multiplicities across methods. We pool 250,000 replicates—50,000 from each of five fixed seeds—and use the 2.5th/97.5th percentiles of paired differences. Point estimates use the original evaluation set. Accuracy gains are proposed minus baseline; CIER reductions are baseline minus proposed, in percentage points.

ComparisonAvg. accuracy gain [95% CI]Mean CIER reduction [95% CI]
Full LTC vs. ERD-only+0.38 [0.07, 0.82]+0.75 [−0.25, 1.76]
Full LTC vs. ERD + LA+0.38 [0.13, 0.75]+1.37 [0.36, 2.38]
DCS vs. motion magnitude+0.23 [0.02, 0.44]+0.76 [−0.16, 1.68]
DCS vs. flow-gradient norm+0.33 [0.14, 0.53]+1.15 [0.41, 1.91]

All accuracy intervals exclude zero. Importantly, the control isolating merge protection—Full LTC versus ERD + LA—also shows a positive CIER reduction. However, the CIER intervals for Full LTC versus ERD-only and DCS versus motion magnitude include zero, so we qualify those conclusions accordingly.

The component profile in Appendix C.4, Table 15 shows that DCS and motion magnitude both require approximately 8.9 ms for ERD. The following measurements use LLaVA-OneVision with 32 frames and 25% retention. Times are independently rounded to one decimal place.

Risk statisticTFE (ms)ERD (ms)RGLA (ms)Compression (ms)
DCS21.78.945.676.2
Motion magnitude21.98.948.679.4
Flow-gradient norm21.89.346.877.8

These profiles quantify the component costs; the end-to-end comparisons in Part 2 include the full compression overhead.

To investigate the Q4–Q5 behavior, we added source-token visualizations and a fixed-budget allocation study (Appendix D.2, Figures 9–10). In the selected examples, the same 20.01% singleton retention share coexists with 58.66–59.53% aggregation participation in low-risk clips, versus 22.91–23.25% in high-risk clips. These describe source-token states, not final output-token counts.

Protection prevents merging but does not guarantee final retention. Increasing the risk-category share α\alpha from 0.30 to 0.45 at a fixed output budget lowers Q4 CIER from 11.8% to 10.9% and Q5 CIER from 13.4% to 12.6%, while average accuracy changes from 52.0 to 51.9 (Appendix C.3, Table 9). This shows a trade-off between high-risk preservation and overall accuracy. The results support final allocation as one contributor to high-risk errors, while correspondence and merging errors remain possible additional factors.

We hope these uncertainty estimates, timing measurements, and high-risk analyses address your concerns and clarify the mechanism’s benefits and limitations.

AuthorsOpenReview ↗

Response to Reviewer eArA [2/2]

To Weakness 2 & Question 2: Controlled Accuracy–Efficiency Comparisons and Related Work

Thank you for this suggestion. The added controlled comparisons show that LTC achieves higher average accuracy and lower compression overhead at comparable end-to-end latency to FlashVID and OTT-Vid on LLaVA-OneVision. On Qwen3-VL, LTC improves average accuracy over FlashVID by 0.4 percentage points, with approximately 13.4% lower TTFT and 11.1% lower total latency.

Both comparisons are reported in Section 4.4, Table 5. Experiments use batch size one on an NVIDIA A6000, with results macro-averaged over VideoMME, LVBench, and LongVideoBench. TTFT and total latency include compression overhead.

On LLaVA-OneVision with 32 input frames and 25% configured retention, the realized budgets and measured results are:

MethodVisual slots (realized retention)Avg. accuracyCompression (ms)TTFT (ms)Total latency (ms)
Vanilla6,304 (100.0%)51.4—1499.31659.3
FlashVID1,582 (25.1%)51.384.2905.11026.3
OTT-Vid1,562 (24.8%)50.886.7904.21024.2
LTC1,568 (24.9%)52.076.2894.61018.9

All four configurations have the same overall PyTorch peak allocated memory of 19.082 GiB, because the peak occurs during multimodal visual preparation before the first LLM forward. During prefill, peak allocated memory decreases from 16.21 GiB for Vanilla to 15.33 GiB for all three compression methods.

On Qwen3-VL with 64 input frames and 50% configured retention, LTC and FlashVID also have closely matched realized budgets:

MethodVisual slots (realized retention)Avg. accuracyCompression (ms)TTFT (ms)Total latency (ms)
Vanilla17,257 (100.0%)57.4—5158.65774.4
FlashVID8,641 (50.1%)55.5266.23645.73869.6
LTC8,595 (49.8%)55.9153.63157.73439.8

Peak allocated memory is 20.193 GiB for both compression methods, compared with 21.453 GiB for Vanilla. LTC remains below the same-frame full-token accuracy reference of 57.4, but improves both accuracy and measured latency relative to FlashVID in this setting.

In the setting you highlighted—32-frame LLaVA-Video at 25% retention—FastVID achieves 52.3, compared with 51.9 for LTC (Appendix C.2, Table 7). We retain this result and do not claim that LTC leads every configuration. The additional measurements establish favorable accuracy–efficiency trade-offs in the tested settings. Due to rebuttal-time constraints, the Qwen3-VL comparison does not yet include OTT-Vid; we will consider this extension in future work.

Thank you also for highlighting Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior. We have added DyToK to Section 2.2. DyToK uses LLM-guided keyframe priors for dynamic frame-budget allocation, while LTC diagnoses local aggregation risk to guide protection before merging. The revised discussion clarifies these different compression decisions in the context of recent video-LLM acceleration methods.

We hope these controlled comparisons and the expanded related-work discussion address your concerns about LTC’s practical accuracy–efficiency trade-off and clarify its contribution.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you for your thoughtful comments and questions. We hope our response above has addressed the points you raised. As the discussion period draws to a close, we would be grateful for any further feedback you may have. We would be happy to clarify any remaining points.

Thank you sincerely for your time and effort in reviewing our paper.

Best regards,
The Authors

Reviewer_M9vxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LTC is a training-free framework for compressing video tokens. TFE estimates displacement and confidence through local feature matching across adjacent encoded temporal positions. ERD uses divergence, curl, and shear of the displacement field after dominant-translation removal, together with active low-confidence signals, to identify candidates that should avoid merging. RGLA follows absolute displacement to construct feature-consistent, length-constrained trajectories, merges unprotected tokens, and selects representatives under a global budget. Protection means remaining independent during aggregation, not guaranteed inclusion in the final output. Experiments cover three backbones and three benchmarks. At 25% retention, LLaVA-OneVision reaches 101.3% of full-token mean accuracy. On Qwen3-VL, 128-frame input at 25% retention improves mean accuracy over the 32-frame full-token reference, while reducing TTFT from 11.93 s to 4.99 s relative to full-token 128-frame inference.

Critical strengths

Diagnosing merge risk before aggregation is well motivated and supported by substantive ablations. Rather than linking frames by similarity alone, LTC explicitly considers instability in local correspondences and distinguishes tokens that should remain independent from trajectories suitable for aggregation. Table 4 separates risk scoring, trajectory aggregation, and merge protection, and includes a random gate with a matched number of additional protected tokens. Full LTC improves the mean from 51.7 to 52.0 and reduces mean CIER from 11.5% to 10.1% relative to ERD+LA, while also improving both metrics over the random gate. These controls support the value of choosing protection locations rather than merely protecting more tokens.

Compression quality transfers across backbones, with measurable efficiency gains. At 25% retention, LLaVA-OneVision obtains a three-benchmark mean of 52.0 versus 51.4 for the full-token model. Several LLaVA-Video settings also remain close to full-token performance, although LTC does not lead every configuration. Same-frame measurements on Qwen3-VL include compression overhead and show a 2.39× TTFT speedup with approximately 9.4% lower peak memory at 128 frames (Appendix Table 12). These results demonstrate practical value in preserving dynamic information while reducing decoder cost.

Critical weaknesses

Accuracy–latency comparisons against the main compression baselines are missing. The quality tables compare FastVID, FlashVID, and OTT-Vid, but efficiency measurements mainly compare LTC with full-token Qwen3-VL; Appendix B.3 explicitly states that FlashVID and OTT-Vid were not evaluated on Qwen. Local matching, smoothing, risk-field computation, and trajectory construction can introduce material overhead, so matching retention or token–layer budgets does not establish superior latency. Moreover, 128-frame LTC at 4.99 s TTFT is still slower than the 32-frame full-token reference at approximately 2.50 s; higher accuracy and lower cost at matched frame counts are valid but distinct findings. Same-hardware, same-frame baseline timings and component profiles would strengthen the efficiency contribution substantially.

The specific advantage of the risk mechanism needs more robust evidence, especially on high-risk videos. The paper already includes valuable controls, but DCS improves the mean over motion magnitude and flow-gradient norm by only 0.2 and 0.3 points, respectively (Table 5), while merge protection adds 0.3 points over ERD+LA. Paired uncertainty for these differences is not reported, and the random gate uses one seed. More notably, Appendix Figure 6 and Section C.3 show that ERD alone has lower CIER than full LTC in Q4–Q5, with the strongest protection gains occurring in low- and intermediate-risk groups. This does not invalidate the overall method, but leaves the precise failure modes addressed by risk diagnosis incompletely established.

Questions for authors

Could LTC, FlashVID, and OTT-Vid be compared on at least one shared backbone with the same frame count and explicit realized budgets, reporting accuracy, TTFT, total latency, and peak memory, together with TFE/ERD/RGLA timing breakdowns? Please also specify where attention saliency is extracted and whether its cost is included.

Could video-level paired bootstrap intervals be provided for the key differences in Tables 4–5, with multiple random-gate seeds? In Q4–Q5, why does full LTC lose more previously correct predictions than ERD alone: correspondence errors, trajectory merging, or budget competition?

Could Table 3's F × r = 32 comparison include other compressors at the same higher frame counts and output budgets, reporting realized token counts and TTFT? This would separate the benefit of denser sampling from the benefit specific to LTC.

4 comments
AuthorsOpenReview ↗

Response to Reviewer M9vx [1/3]

We sincerely appreciate your constructive feedback and your recognition of the motivation for diagnosis-before-aggregation, the controlled component ablations, and the practical value of the training-free framework. Following your suggestions, we have expanded the accuracy–efficiency comparisons, evaluated multiple random-gate seeds, and further investigated the behavior on high-risk videos. We also report paired bootstrap confidence intervals for key differences. Our point-by-point response follows.

To Weakness 1 & Question 1: Accuracy–Efficiency Comparisons and Component Overhead

Thank you for your insightful suggestion.

LTC achieves practical acceleration with competitive accuracy, including the overhead of TFE, ERD, and RGLA. On LLaVA-OneVision, LTC achieves 1.68× TTFT speedup and 1.63× total-latency speedup over vanilla inference, while obtaining higher average accuracy at comparable end-to-end latency to FlashVID and OTT-Vid.

To address your concern, we have added efficiency comparisons on LLaVA-OneVision using a single NVIDIA A6000 GPU, with 32 input frames, a configured retention ratio of 25%, and batch size one. As reported in Section 4.4, Table 5, the following results compare accuracy and measured latency at closely matched realized budgets. Results are macro-averaged over VideoMME, LVBench, and LongVideoBench, and all latencies are in milliseconds. The Qwen3-VL comparisons are presented in Part 3 of this response.

MethodVisual slots (realized retention)Avg. accuracyCompressionTTFTTotal latency
Vanilla6,304 (100.0%)51.4—1499.31659.3
FlashVID1,582 (25.1%)51.384.2905.11026.3
OTT-Vid1,562 (24.8%)50.886.7904.21024.2
LTC1,568 (24.9%)52.076.2894.61018.9

LTC achieves an average accuracy of 52.0, compared with 51.3 for FlashVID and 50.8 for OTT-Vid, while requiring less compression time. Its TTFT and total latency are 894.6 ms and 1018.9 ms, respectively, both including compression overhead. These results demonstrate improved accuracy at comparable end-to-end latency to the compression baselines.

Regarding memory, all four configurations have the same overall PyTorch peak allocated memory of 19.082 GiB, because the overall peak occurs during multimodal visual preparation before the first LLM forward. During prefill, peak allocated memory decreases from 16.21 GiB for Vanilla to 15.33 GiB for all three compression methods, a reduction of 0.88 GiB, or approximately 5.4%. Table 5 reports the overall peak.

We further provide the component-level timing breakdown to make LTC’s overhead explicit:

TFEERDRGLATotal compression
21.69 ms8.90 ms45.64 ms76.23 ms

The same breakdown, rounded to one decimal place, is reported in Appendix C.4, Table 15. For this OneVision configuration, attention saliency is extracted from the final self-attention layer of the SigLIP vision encoder. Its extraction cost is included in vision-encoder timing and TTFT, without an additional LLM attention pass. The component breakdown measures the subsequent compression stages.

We hope the above baseline comparisons and component-level analysis address your concern regarding LTC’s practical efficiency and compression overhead.

AuthorsOpenReview ↗

Response to Reviewer M9vx [2/3]

To Weakness 2 & Question 2: Robustness of the Risk Mechanism

Thank you for emphasizing the need for stronger evidence. Our additional analyses support the average-accuracy gains of DCS and merge protection: all four confidence intervals for accuracy gains exclude zero, and Full LTC outperforms all three count-matched random-gate runs. The CIER benefits are less uniform, and our high-risk analysis shows that final budget allocation contributes to the Q4–Q5 behavior.

For the key ablations in Section 4.3, Table 4, and Appendix B.5, Table 6, we use a dataset-stratified, paired video-level percentile bootstrap. Videos are resampled within each benchmark, retaining all associated questions and sharing sampled IDs and multiplicities across methods. We pool 250,000 replicates—50,000 from each of five fixed seeds—and use the 2.5th/97.5th percentiles for the intervals. Point estimates are computed on the original evaluation set. Accuracy gains are proposed minus baseline; CIER reductions are baseline minus proposed, in percentage points.

ComparisonAvg. accuracy gain [95% CI]Mean CIER reduction [95% CI]
Full LTC vs. ERD-only+0.38 [0.07, 0.82]+0.75 [−0.25, 1.76]
Full LTC vs. ERD + LA+0.38 [0.13, 0.75]+1.37 [0.36, 2.38]
DCS vs. motion magnitude+0.23 [0.02, 0.44]+0.76 [−0.16, 1.68]
DCS vs. flow-gradient norm+0.33 [0.14, 0.53]+1.15 [0.41, 1.91]

The CIER intervals are positive for Full LTC versus ERD + LA and DCS versus flow-gradient norm, but include zero for the other two comparisons. We therefore qualify the CIER conclusions accordingly. We also extend the random-gate control to three seeds, fixing protection counts and other settings per video. The runs achieve 51.45, 51.59, and 51.65, yielding 51.56 ± 0.10, compared with 52.03 for Full LTC (Appendix C.3, Table 8).

To investigate why ERD-only preserves more previously correct answers in Q4–Q5, we add source-token visualizations and a fixed-budget allocation study in Appendix D.2, Figures 9–10. In the selected examples, singleton retention is 20.01% in both risk groups, while aggregation participation is 58.66–59.53% in low-risk clips and 22.91–23.25% in high-risk clips. These are source-token states, not final output-token counts, and illustrate different aggregation behavior despite the same singleton retention share.

Crucially, protection keeps candidates independent during aggregation but does not guarantee their final inclusion. Increasing the risk-category budget share α\alpha from 0.30 to 0.45 lowers Q4 CIER from 11.8 to 10.9 and Q5 CIER from 13.4 to 12.6, while overall average accuracy changes from 52.0 to 51.9. Increasing α\alpha further to 0.60 yields CIERs of 11.4 and 12.5, with average accuracy decreasing to 51.5 (Appendix C.3, Table 9). This demonstrates a trade-off between high-risk preservation and overall performance. Aggregation can consolidate stable, redundant content, but may lose informative variations through merging or final budgeted selection. The allocation study supports a role for final budget allocation in high-risk errors. Correspondence and merging errors may also contribute—for example, through ambiguous associations between distinct regions or dilution of brief motion cues during aggregation. These remain possible explanations, as the current analysis does not isolate their individual effects.

We hope these uncertainty estimates, multi-seed comparisons, and high-risk analyses address your concerns and clarify both the benefits and limitations of the risk mechanism.

AuthorsOpenReview ↗

Response to Reviewer M9vx [3/3]

To Question 3: Higher-Frame Comparisons with FlashVID

Thank you for this suggestion. At the same 64-frame input and closely matched realized retention ratios, LTC achieves higher average accuracy than FlashVID (55.9 vs. 55.5), with approximately 13.4% lower TTFT and 11.1% lower total latency. This provides a direct comparison of compressor-specific benefits under matched input conditions. The accuracy advantage remains configuration-dependent, as shown by the broader higher-frame results below.

For the 64-frame setting at 50% configured retention, we report the following realized budgets and efficiency measurements in Section 4.4, Table 5. Latencies are in milliseconds.

MethodVisual slots (realized retention)Avg. accuracyCompressionTTFTTotal latency
Vanilla17,257 (100.0%)57.4—5158.65774.4
FlashVID8,641 (50.1%)55.5266.23645.73869.6
LTC8,595 (49.8%)55.9153.63157.73439.8

Peak allocated GPU memory is 20.193 GiB for both compression methods, compared with 21.453 GiB for Vanilla.

We additionally compare FlashVID and LTC at the same higher input-frame counts and configured retention ratios, using the nominal frame-equivalent budget F×r=32F\times r=32. These accuracy results are reported in Section 4.2, Table 3:

Input framesConfigured retentionFlashVID Avg.LTC Avg.
6450%55.555.9
8040%56.156.3
12825%57.256.9

LTC achieves higher average accuracy at 64 and 80 frames, while FlashVID achieves a higher average at 128 frames. Both methods exceed the 32-frame full-token reference of 54.5, confirming that gains from accommodating denser temporal inputs should not be attributed entirely to LTC.

We also clarify the efficiency reference conditions. LTC’s 4.99 s TTFT at 128 frames is compared with 11.93 s for full-token inference at the same 128-frame input, not with the approximately 2.50 s TTFT of the 32-frame full-token model. Figure 4 and Table 14 report these vanilla references. Thus, higher-frame accuracy gains and same-frame acceleration are distinct findings. Moreover, F×r=32F\times r=32 specifies a nominal frame-equivalent retention budget; it does not imply identical realized token counts, FLOPs, or end-to-end latency.

Due to time constraints during the rebuttal period, the Qwen3-VL comparison does not yet include OTT-Vid. We will consider this extension in future work.

We hope these additional comparisons and explicit reference conditions address your concern about distinguishing denser-sampling gains from the benefits specific to LTC.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you for your thoughtful comments and questions. We hope our response above has addressed the points you raised. As the discussion period draws to a close, we would be grateful for any further feedback you may have. We would be happy to clarify any remaining points.

Thank you sincerely for your time and effort in reviewing our paper.

Best regards,
The Authors

Reviewer_NRHMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

To tackle the video LLM visual token compression task, this paper tries to model the motion dynamics and Eulerian risk diagnosis mechanism to compute the kept and merged tokens across neighboring frames. Built upon such an importance selection, this paper utilizes the pretrained features to achieve this compression without any further training. In the end, the authors conduct experiments on three common video LLMs and employ various evaluations on three video understanding benchmarks, which demonstrates their effectiveness and applicability. Meanwhile, they also provide reproducible code.

Critical strengths
  1. The paper writing is clear and easy to follow.

  2. The motivation sounds reasonable, and the proposed method to model the motion structures among neighboring frames makes sense.

  3. The utility of the Eulerian and Lagrangian perspectives sounds interesting for this task.

Critical weaknesses
  1. How to understand the figure illustration in Fig.1(b) between the two different subfigures?

  2. I am curious that the feature from the visual encoder is usually high-level, abstract, and smooth, while the correspondence computation by the local motion estimation method proposed in this paper may produce noisy results among tokens rather than from low-level features, like RGB space. Can the authors explain this in detail or provide some demonstrations?

  3. There are some recent and meaningful related works should be included in the paper to make discussions or experimental comparisons which will strengthen this paper.

    [1] OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport, CVPR2026

    [2] Token reduction via local and global contexts optimization for efficient video large language models, CVPR2026

  4. Existing methods on VideoLLM token compression usually provide results on very aggressive token reduction budgets, like 15% or 10%, but this paper ignores such aggressive comparisons. I suggest the authors conduct evaluations and comparisons in this setting.

  5. How to understand the final kept and merged visual tokens? I suggest the authors provide a visualization demonstration or comparisons with existing methods by illustrating the effectiveness of the token compression in terms of the motion modeling in this paper.

Questions for authors

I would like to see more explanations and demonstrations in the authors' response, and please refer to my detailed questions above. I will consider raising my score if the authors address my questions and concerns well.

4 comments
AuthorsOpenReview ↗

Response to Reviewer NRHM [1/3]

We sincerely appreciate your constructive feedback and your recognition of the clear presentation, the motivation for modeling neighborhood motion structure, and the relevance of the Eulerian and Lagrangian perspectives. Following your suggestions, we have revised the conceptual illustration, expanded the related-work discussion and baseline comparisons, evaluated more aggressive compression budgets, and provided token-level visualizations. We also present RGB-based and feature-based matching results below to address your concern about correspondence estimation. Our point-by-point response follows.

To Weakness 1: Clarification of Figure 1(b)

Thank you for pointing out the ambiguity. We have revised Figure 1(b), its caption, and the accompanying Introduction text to better explain the two perspectives and their roles in LTC.

The Eulerian perspective observes temporal changes at fixed spatial locations. In the left panel, p1p_1 and p2p_2 denote fixed grid positions, while the arrows illustrate local displacements. In LTC, this perspective motivates diagnosing neighborhood displacement structure on the fixed token grid.

The Lagrangian perspective follows corresponding elements as their positions change over time. In the right panel, matching shapes and colors indicate element identity across the displayed time steps, and the dashed trajectories illustrate their movement. This perspective motivates aggregation along estimated cross-frame correspondence trajectories.

These perspectives serve complementary roles in LTC: Eulerian diagnosis identifies candidates that should remain independent during aggregation, while Lagrangian aggregation combines eligible cross-frame tokens along estimated trajectories. We hope the revised illustration and accompanying text make this distinction clearer and help explain how the two perspectives motivate LTC’s diagnosis-before-aggregation design.

To Weakness 2: Reliability of Correspondence Estimation from Visual Features

Thank you for raising this important concern. We agree that matching high-level visual features can be ambiguous, particularly in repetitive or semantically similar regions. To address this concern, we clarify how LTC handles unreliable matches and provide an additional comparison with RGB-based matching.

LTC uses estimated displacements as coarse guidance for token compression, with uncertainty handled at three stages:

  • TFE: Estimates matching confidence and applies confidence-weighted displacement smoothing.

  • ERD: Downweights unreliable derivative evidence and protects active low-confidence candidates.

  • RGLA: Constrains aggregation through protection masks, feature-consistency checks, and trajectory-length limits.

To empirically examine the matching representation, we compare RGB-based and feature-based matching on LLaVA-OneVision with 32 input frames at 25% visual-token retention:

ConfigurationRetentionVideoMME OverallLVBenchLongVideoBenchAvg.
Vanilla100%58.638.956.651.4
RGB-based matching25%58.540.055.751.4
Feature-based matching25%59.140.556.652.0

Feature-based matching achieves higher accuracy on all three benchmarks, improving VideoMME, LVBench, and LongVideoBench by 0.6, 0.5, and 0.9 percentage points, respectively. These results support its usefulness for compression in the evaluated setting. The uncertainty-handling mechanisms are detailed in Sections 3.2–3.4 and Appendix B.4.

We hope this additional comparison and the above clarifications address your concern regarding the reliability of feature-based correspondence estimation.

AuthorsOpenReview ↗

Response to Reviewer NRHM [2/3]

To Weakness 3: Inclusion of Recent Related Work

Thank you for highlighting these relevant studies. We agree that incorporating recent advances and additional experimental comparisons can strengthen the support for LTC’s core contribution: using neighborhood motion structure and matching uncertainty to guide risk-gated cross-frame aggregation under a limited token budget.

We have added OTPrune and AOT (“Token reduction via local and global contexts optimization for efficient video large language models”) to Section 2.2. We have additionally included AOT experiments on both LLaVA-OneVision and LLaVA-Video in Tables 1–2. Below, we summarize its results alongside LTC at all retention ratios included in these comparisons.

LLaVA-OneVision-Qwen2-7B-OV, 32 input frames

RetentionAOT VideoMMELTC VideoMMEAOT LongVideoBenchLTC LongVideoBench
50%58.158.456.157.1
30%57.758.555.756.8
25%57.459.155.556.6
20%57.158.155.256.8

LLaVA-Video-7B-Qwen2, 64 input frames

RetentionAOT VideoMMELTC VideoMMEAOT LongVideoBenchLTC LongVideoBench
50%61.763.658.359.6
30%58.262.757.259.1
15%59.761.856.357.6
10%58.360.455.956.7

LTC achieves higher reported accuracy than AOT on both evaluated benchmarks across all eight settings. Appendix B.3 provides the AOT configuration and explains why we omit its LVBench results: the official repository lists these results as pending, and the released evaluation script provides no LVBench-specific configuration.

We hope the expanded discussion and additional comparisons strengthen the empirical support for LTC’s core contribution and provide a more comprehensive evaluation alongside recent methods.

To Weakness 4: Comparisons under Aggressive Compression

Thank you for this suggestion. The added comparisons show that LTC achieves the highest displayed average among methods with complete three-benchmark results at 15% retention, scoring 53.2. At 10% retention, LTC scores 52.1, exceeding FastVID and OTT-Vid, while FlashVID achieves a higher average of 52.8.

To address your concern, we have added comparisons at both aggressive budgets on LLaVA-Video with 64 input frames, reported in Table 2. “–” indicates unavailable results; AOT’s average is omitted because its LVBench result is unavailable.

RetentionMethodVideoMME OverallLVBenchLongVideoBenchAvg.
15%FastVID61.139.857.852.9
15%FlashVID61.339.458.753.1
15%OTT-Vid61.238.658.152.6
15%AOT59.7–56.3–
15%LTC61.840.257.653.2
10%FastVID59.538.356.851.5
10%FlashVID60.840.557.152.8
10%OTT-Vid59.538.057.851.8
10%AOT58.3–55.9–
10%LTC60.439.256.752.1

Appendix C.1 and Figure 5 additionally report LTC’s LLaVA-OneVision performance across retention ratios, including 15% and 10%. We hope these additional comparisons address your concern and provide a more complete picture of LTC’s strengths and limitations under aggressive compression.

AuthorsOpenReview ↗

Response to Reviewer NRHM [3/3]

To Weakness 5: Visualization of Kept and Merged Tokens

Thank you for this valuable suggestion. To provide a more intuitive view of what is retained, what is aggregated, and how motion-related evidence contributes to the compressed representation, we have added two complementary visualizations, discussed in Appendix D.1:

  • Figure 7: Motion-related token retention. This figure compares LTC and FlashVID on the same three displayed frames of an action-ordering example. Red outlines indicate tokens retained as singletons by each method. The highlighted LTC tokens cover the lower limbs and step platform, providing local evidence relevant to recognizing the stepping action after squatting. LTC correctly answers “C. Step-ups,” whereas FlashVID answers “B. Plank.” We provide this example to illustrate how the retained representations differ between the methods.

  • Figure 8: Risk-gated trajectory aggregation. Red outlines identify ERD-protected candidates that remain independent during aggregation. Differently colored links identify eligible cross-frame correspondence groups. Each group contributes to one merged representative, constructed by blending its similarity-weighted mean with an observed member feature. Multiple source tokens in a group therefore contribute jointly to a compact representation.

We would also like to clarify the distinction between protection during aggregation and final retention. Protection preserves a token’s independence while trajectories are constructed; it does not guarantee that the token will appear in the final output. The final visual sequence contains singleton candidates and merged representatives selected under the common token budget. Appendix D.2 provides further illustrations of source-token states across risk levels and examines how risk-category budget allocation affects high-risk compression errors. We include these analyses to complement the retention and aggregation examples and to explain how aggregation and final budget allocation affect the compressed representation.

We hope that the explanations, experimental results, and visualizations above address your questions and concerns. We would be happy to provide any further clarification and welcome any additional questions or suggestions. Thank you again for your time and constructive feedback.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you for your thoughtful comments and questions. We hope our response above has addressed the points you raised. As the discussion period draws to a close, we would be grateful for any further feedback you may have. We would be happy to clarify any remaining points.

Thank you sincerely for your time and effort in reviewing our paper.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for the constructive feedback and recognition of our work. Reviewer NRHM acknowledges the clear presentation and the motivation for modeling neighborhood motion structure. Reviewers M9vx and eArA recognize the motivation for diagnosis before aggregation, the targeted ablations, and the practical potential of our training-free framework. Following these suggestions, we have strengthened the manuscript and supplemented our rebuttal as summarized below.

Reviewer NRHM

Concerns: Conceptual clarity, correspondence reliability, recent baselines, aggressive compression, and token-level visualizations.

Response: We revised the conceptual illustration to clarify the complementary Eulerian and Lagrangian perspectives, and added token-level visualizations of retention and trajectory aggregation. We clarified uncertainty handling and compared feature-based with RGB-based matching on LLaVA-OneVision. We also expanded the related-work discussion to include OTPrune and AOT, added AOT comparisons on LLaVA-OneVision and LLaVA-Video, and evaluated aggressive compression at 15% and 10% retention on LLaVA-Video.

Reviewer M9vx

Concerns: Practical efficiency, risk-mechanism robustness, high-risk behavior, and attribution of higher-frame gains.

Response: We added controlled accuracy–efficiency comparisons against FlashVID and OTT-Vid on LLaVA-OneVision and against FlashVID on Qwen3-VL, reporting realized budgets and including compression overhead. We profiled LTC’s individual components and clarified attention-saliency extraction and cost accounting for the OneVision setup. On LLaVA-OneVision, we conducted paired video-level bootstrap analyses and count-matched random-gate experiments with three seeds. We further added risk-stratified visualizations and fixed-budget allocation studies to explain why full LTC loses more previously correct predictions than ERD alone in Q4–Q5. We also compared LTC with FlashVID under matched higher-frame Qwen3-VL settings to distinguish denser-input effects from compressor-specific benefits.

Reviewer eArA

Concerns: Incremental component benefits, high-risk behavior, controlled efficiency comparisons, and related work.

Response: On LLaVA-OneVision, we supplemented the DCS and merge-protection ablations with uncertainty estimates and compared the computational costs of alternative risk statistics. We also provided risk-stratified visualizations and fixed-budget allocation studies to explain why full LTC loses more previously correct predictions than ERD alone in Q4–Q5. We expanded controlled baseline comparisons on LLaVA-OneVision and Qwen3-VL, and added a discussion distinguishing DyToK’s frame-budget allocation from LTC’s local aggregation-risk diagnosis.

Detailed evidence and manuscript references are provided in the individual reviewer responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.