- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents LTC, a training-free framework for visual token compression in video language models. It estimates local token correspondences and matching confidence, then uses neighborhood displacement structure and active low-confidence signals to identify tokens that should remain independent during aggregation. Eligible tokens are merged along estimated trajectories, followed by representative selection under a token budget. Experiments cover three backbones and three video understanding benchmarks, with additional component ablations, sensitivity analyses, and inference efficiency measurements.
Critical strengths
-
A well-motivated distinction between token importance and aggregation risk. LTC recognizes that feature similarity alone may not determine whether tokens can be safely merged, particularly when motion structure and correspondence uncertainty carry information relevant to temporal understanding. Using local displacement patterns and matching uncertainty to guide merge protection before representative selection provides a coherent motivation for the framework. The training-free design is also a practical advantage.
-
Competitive results supported by targeted ablations. On LLaVA-OneVision, LTC achieves 52.0% average accuracy at 25% token retention, slightly exceeding the 51.4% of the uncompressed model (Table 1). The component ablations, count-matched random gate, and comparisons with alternative risk statistics help clarify the contributions of individual design choices (Tables 4–5). The Qwen3-VL experiments further account for compression overhead and demonstrate reduced TTFT, providing initial evidence of practical efficiency.
Critical weaknesses
-
Limited and inconsistent benefits from risk-aware aggregation. My main concern is whether the additional complexity of the proposed mechanism is sufficiently justified. Full LTC improves average accuracy over ERD-only selection by just 0.3 percentage points (Table 4), while the proposed divergence, curl, and shear statistic outperforms simple motion magnitude by only 0.2 points (Table 5). More importantly, Appendix C.3 shows that ERD alone achieves lower compression-induced error than full LTC in the two highest-risk quintiles. This observation raises questions about the effectiveness of the protection mechanism in precisely the challenging regions it aims to address. A clearer explanation of this behavior, together with uncertainty estimates, would help establish whether the proposed mechanism offers a consistent advantage over simpler alternatives.
-
Insufficient evidence of a superior accuracy-efficiency tradeoff. The improvements over existing compression methods are not consistent across settings. For instance, on 32-frame LLaVA-Video at 25% token retention, LTC achieves 51.9% accuracy compared with 52.3% for FastVID (Table 6). More importantly, the latency and memory evaluations on Qwen3-VL only compare LTC against uncompressed inference, while comparisons with competing compression methods are conducted on the LLaVA backbones. Although the results demonstrate acceleration over vanilla inference, they do not establish whether LTC provides a more favorable accuracy-efficiency tradeoff than existing approaches under comparable conditions. A discussion of existing related video-LLM acceleration methods [1] would further strengthen your core contribution.
[1] Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior.
Questions for authors
Overall, I currently lean toward Weak Rejection. While the method is reasonably motivated and achieves competitive results, I remain concerned about its incremental benefits over simpler alternatives and its practical advantages over existing compression methods. Addressing the following questions with additional evidence could help improve my assessment.
-
Could the authors further explain the high-risk behavior observed in Figure 6 and Appendix C.3? In particular, under what conditions does risk-aware aggregation help or hurt performance? Paired confidence intervals comparing LTC with ERD-only selection and simpler risk statistics would also help clarify the robustness of the reported gains.
-
Could the authors provide a controlled comparison with the strongest compression baselines using the same backbone, input-frame count, and token budget? Reporting both accuracy and measured TTFT, including compression overhead, would help establish the practical value of LTC.
5 comments
Response to Reviewer eArA [2/2]
To Weakness 2 & Question 2: Controlled Accuracy–Efficiency Comparisons and Related Work
Thank you for this suggestion. The added controlled comparisons show that LTC achieves higher average accuracy and lower compression overhead at comparable end-to-end latency to FlashVID and OTT-Vid on LLaVA-OneVision. On Qwen3-VL, LTC improves average accuracy over FlashVID by 0.4 percentage points, with approximately 13.4% lower TTFT and 11.1% lower total latency.
Both comparisons are reported in Section 4.4, Table 5. Experiments use batch size one on an NVIDIA A6000, with results macro-averaged over VideoMME, LVBench, and LongVideoBench. TTFT and total latency include compression overhead.
On LLaVA-OneVision with 32 input frames and 25% configured retention, the realized budgets and measured results are:
| Method | Visual slots (realized retention) | Avg. accuracy | Compression (ms) | TTFT (ms) | Total latency (ms) |
|---|---|---|---|---|---|
| Vanilla | 6,304 (100.0%) | 51.4 | — | 1499.3 | 1659.3 |
| FlashVID | 1,582 (25.1%) | 51.3 | 84.2 | 905.1 | 1026.3 |
| OTT-Vid | 1,562 (24.8%) | 50.8 | 86.7 | 904.2 | 1024.2 |
| LTC | 1,568 (24.9%) | 52.0 | 76.2 | 894.6 | 1018.9 |
All four configurations have the same overall PyTorch peak allocated memory of 19.082 GiB, because the peak occurs during multimodal visual preparation before the first LLM forward. During prefill, peak allocated memory decreases from 16.21 GiB for Vanilla to 15.33 GiB for all three compression methods.
On Qwen3-VL with 64 input frames and 50% configured retention, LTC and FlashVID also have closely matched realized budgets:
| Method | Visual slots (realized retention) | Avg. accuracy | Compression (ms) | TTFT (ms) | Total latency (ms) |
|---|---|---|---|---|---|
| Vanilla | 17,257 (100.0%) | 57.4 | — | 5158.6 | 5774.4 |
| FlashVID | 8,641 (50.1%) | 55.5 | 266.2 | 3645.7 | 3869.6 |
| LTC | 8,595 (49.8%) | 55.9 | 153.6 | 3157.7 | 3439.8 |
Peak allocated memory is 20.193 GiB for both compression methods, compared with 21.453 GiB for Vanilla. LTC remains below the same-frame full-token accuracy reference of 57.4, but improves both accuracy and measured latency relative to FlashVID in this setting.
In the setting you highlighted—32-frame LLaVA-Video at 25% retention—FastVID achieves 52.3, compared with 51.9 for LTC (Appendix C.2, Table 7). We retain this result and do not claim that LTC leads every configuration. The additional measurements establish favorable accuracy–efficiency trade-offs in the tested settings. Due to rebuttal-time constraints, the Qwen3-VL comparison does not yet include OTT-Vid; we will consider this extension in future work.
Thank you also for highlighting Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior. We have added DyToK to Section 2.2. DyToK uses LLM-guided keyframe priors for dynamic frame-budget allocation, while LTC diagnoses local aggregation risk to guide protection before merging. The revised discussion clarifies these different compression decisions in the context of recent video-LLM acceleration methods.
We hope these controlled comparisons and the expanded related-work discussion address your concerns about LTC’s practical accuracy–efficiency trade-off and clarify its contribution.
Follow-up on Our Response
Dear Reviewer,
Thank you for your thoughtful comments and questions. We hope our response above has addressed the points you raised. As the discussion period draws to a close, we would be grateful for any further feedback you may have. We would be happy to clarify any remaining points.
Thank you sincerely for your time and effort in reviewing our paper.
Best regards,
The Authors
Response to Reviewer eArA [1/2]
We sincerely appreciate your constructive feedback and your recognition of the motivation for distinguishing token importance from aggregation risk, the targeted ablations, and the practical potential of the training-free framework. Following your suggestions, we have further examined the robustness of the component gains, investigated the high-risk behavior, and expanded the controlled accuracy–efficiency comparisons. We have also supplemented the discussion of recent video-LLM acceleration methods. Our point-by-point response follows.
To Weakness 1 & Question 1: Incremental Benefits and High-Risk Behavior
Thank you for raising these concerns. Our additional analyses support modest positive accuracy gains from DCS and merge protection, with DCS requiring comparable measured risk-diagnosis time to motion magnitude. They also identify an allocation trade-off in high-risk videos: increasing the risk-category budget can reduce Q4–Q5 CIER, but may lower overall accuracy. We therefore do not claim uniform improvements over ERD-only selection in these groups.
For the ablations in Section 4.3, Table 4, and Appendix B.5, Table 6, we report dataset-stratified, paired video-level percentile bootstrap intervals. Videos are resampled within each benchmark, retaining all associated questions and sharing sampled IDs and multiplicities across methods. We pool 250,000 replicates—50,000 from each of five fixed seeds—and use the 2.5th/97.5th percentiles of paired differences. Point estimates use the original evaluation set. Accuracy gains are proposed minus baseline; CIER reductions are baseline minus proposed, in percentage points.
All accuracy intervals exclude zero. Importantly, the control isolating merge protection—Full LTC versus ERD + LA—also shows a positive CIER reduction. However, the CIER intervals for Full LTC versus ERD-only and DCS versus motion magnitude include zero, so we qualify those conclusions accordingly.
The component profile in Appendix C.4, Table 15 shows that DCS and motion magnitude both require approximately 8.9 ms for ERD. The following measurements use LLaVA-OneVision with 32 frames and 25% retention. Times are independently rounded to one decimal place.
These profiles quantify the component costs; the end-to-end comparisons in Part 2 include the full compression overhead.
To investigate the Q4–Q5 behavior, we added source-token visualizations and a fixed-budget allocation study (Appendix D.2, Figures 9–10). In the selected examples, the same 20.01% singleton retention share coexists with 58.66–59.53% aggregation participation in low-risk clips, versus 22.91–23.25% in high-risk clips. These describe source-token states, not final output-token counts.
Protection prevents merging but does not guarantee final retention. Increasing the risk-category share α from 0.30 to 0.45 at a fixed output budget lowers Q4 CIER from 11.8% to 10.9% and Q5 CIER from 13.4% to 12.6%, while average accuracy changes from 52.0 to 51.9 (Appendix C.3, Table 9). This shows a trade-off between high-risk preservation and overall accuracy. The results support final allocation as one contributor to high-risk errors, while correspondence and merging errors remain possible additional factors.
We hope these uncertainty estimates, timing measurements, and high-risk analyses address your concerns and clarify the mechanism’s benefits and limitations.