- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper one efficient VLM for video understanding, where an efficient video encoder compress the token within a lightweight visual backbone. CTD, a new training framework is proposed to transfer spatio-temporally dense information from a teacher to a compact student. The expriments on effectiveness and efficiency demonstrate the superiority of the proposed method.
Critical strengths
This may be the first work that integrates spatio‑temporal compression natively inside the vision backbone.
Compressed Token Distillation is proposed to transfer the spatio-temporally dense information from a teacher to a compact student.
The motivation and writing is clear.
The experiments demonstrate the effectivenesses and efficiency of the proposed model.
Critical weaknesses
The evaluations should be more solid, Only video benchmarks are evaluated. What about the performances on the other benchmark datasets, such as images, or document video benchmark dataset.
Traditional frame sampling strategy differs from the proposed litframe. Can the two strategies considerded togehter?
All main experiments are confined to the InternVL model family. What about the performances on other VLMs?
Questions for authors
Check the detailed comments in the weaknesses part.
4 comments
Comments after reading the rebuttal
The first two comments are well addressed.
The comments about the performances on the other VLMs are not well addressed with no experimental results provided. InternVL3-8B is one baseline VLM, which performs inferiorly to current modern VLMs. The performance gains over InternVL3-8B may not be obtained based on other VLMs. The experiments are highly suggested to demonstrate the effectiveness of the proposed LiteFrame.
Thank you for taking the time to read our response and for confirming that our responses addressed your first two concerns. We agree that validating LiteFrame beyond the InternVL family is important for establishing the generalizability of our method. That being said, full validation of LiteFrame on a new VLM family requires distilling a new student encoder followed by Language Model Adaptation, which is infeasible within the rebuttal period given our computational resources.
Nevertheless, LiteFrame's gains rely on two critical components: (1) WAP serving as an effective training objective for distillation and (2) the LLM benefiting from frame scaling under compressed visual tokens. As a result, we believe showing that these two findings transfer to other VLM families is a partial, yet strong indicator of whether the full method would transfer. In the tables below, we test both properties on Qwen3-VL-8B-Instruct. Please check the plots (Fig. 14) in Appendix K.
The evaluations demonstrate the following:
(1) WAP consistently outperforms average pooling. Under a 16$\times$ compression ratio, WAP achieves higher accuracy across four benchmarks, increasing average accuracy from 54.5% to 55.4% (+0.9%p) at 64 frames and from 58.1% to 59.1% (+1.0%p) at 128 frames.
| Method | Frames | Visual Tokens | Video-MME (w/o subs) | Video-MME (w/ subs) | MLVU | LongVideoBench | Average |
|---|---|---|---|---|---|---|---|
| Qwen3-VL-8B-Instruct (No comp.) | 64 | 8192 | 66.8 | 69.7 | 68.6 | 61.2 | 66.6 |
| + Average Pooling (16$\times$) | 64 | 512 | 52.0 | 59.6 | 54.0 | 52.4 | 54.5 |
| + WAP (Ours, 16$\times$) | 64 | 512 | 53.3 | 60.7 | 54.8 | 52.7 | 55.4 |
| Qwen3-VL-8B-Instruct (No comp.) | 128 | 16384 | 69.1 | 73.1 | 72.7 | 62.6 | 69.4 |
| + Average Pooling (16$\times$) | 128 | 1024 | 55.0 | 64.0 | 57.3 | 56.0 | 58.1 |
| + WAP (Ours, 16$\times$) | 128 | 1024 | 56.3 | 65.0 | 58.7 | 56.4 | 59.1 |
(2) WAP enables effective frame scaling under fixed visual token budgets. Across budgets of 1024, 2048, 4096, and 8192 visual tokens, WAP (16$\times$) allows 16$\times$ more input frames and improves average accuracy by 2.0-4.0%p over the uncompressed baseline. Notably, WAP (16$\times$) with 256 frames (2048 tokens) already matches the uncompressed baseline with 32-frame (4096 tokens) using half the visual tokens.
| Method | Frames at Each Budget | 1024 Visual Tokens | 2048 Visual Tokens | 4096 Visual Tokens | 8192 Visual Tokens |
|---|---|---|---|---|---|
| Qwen3-VL-8B-Instruct (No comp.) | 8 / 16 / 32 / 64 | 55.1 | 59.1 | 62.8 | 66.6 |
| + WAP (16$\times$) | 128 / 256 / 512 / 1024 | 59.1 (+4.0) | 62.8 (+3.7) | 66.5 (+3.7) | 68.6 (+2.0) |
In summary, these results confirm that LiteFrame’s core compression primitive and frame scaling behavior generalize well to Qwen3-VL-8B-Instruct; while this still does not show whether the full LiteFrame method transfers, we think it is a strongly positive indicator and hope that it at least partially addresses the reviewer’s concern. We are also extending these evaluations to other VLM families. We will report the results here as they become available.
Following up on our Qwen3-VL-8B results, we have now completed evaluations on Gemma4-E4B-it, which show the same two findings.
(1) WAP consistently outperforms average pooling. At 64 frames and 16× compression, WAP improves average accuracy from 49.0% to 50.1% (+1.1 percentage points), with improvements in all four evaluation settings.
| Method | Frames | Visual Tokens | Video-MME (w/o subs) | Video-MME (w/ subs) | MLVU | LongVideoBench | Average |
|---|---|---|---|---|---|---|---|
| Gemma4-E4B-it (No comp.) | 64 | 16384 | 56.1 | 59.9 | 60.7 | 54.1 | 57.7 |
| + Average Pooling (16$\times$) | 64 | 1024 | 46.1 | 53.5 | 48.7 | 47.6 | 49.0 |
| + WAP (Ours, 16$\times$) | 64 | 1024 | 48.3 | 54.9 | 49.7 | 47.7 | 50.1 |
(2) WAP enables effective frame scaling under fixed visual-token budgets. Across the four evaluated budgets, 16$\times$ WAP processes 16$\times$ more frames and improves average accuracy by 2.2–6.1 percentage points over the uncompressed baseline.
| Method | Frames at Each Budget | 1024 Visual Tokens | 2048 Visual Tokens | 4096 Visual Tokens | 8192 Visual Tokens |
|---|---|---|---|---|---|
| Gemma4-E4B-it (No comp.) | 4 / 8 / 16 / 32 | 44.0 | 47.9 | 51.4 | 55.4 |
| + WAP (16$\times$) | 64 / 128 / 256 / 512 | 50.1 (+6.1) | 53.3 (+5.4) | 56.0 (+4.6) | 57.6 (+2.2) |
Notably, WAP with 128 frames achieves 53.3% using 2,048 visual tokens, exceeding the uncompressed baseline with 16 frames (51.4% using 4,096 tokens) while using half the visual tokens. These results provide further empirical support for LiteFrame’s two design principles beyond InternVL. The updated table and plots are included in Appendix K.
We hope these additional results help address your remaining concerns. If you have any further questions, we would be happy to clarify them before the discussion period ends. We sincerely appreciate your time and thoughtful consideration.
We sincerely thank Reviewer 9tiH for carefully evaluating our work and providing constructive feedback. We are particularly grateful that you recognized (1) the novelty of integrating spatio-temporal compression directly into the vision backbone, (2) the strong experimental results and (3) our clear motivation and writing.
[R3W1] Evaluation Scope
Thank you for the suggestion and we appreciate the sentiment that evaluations should be as comprehensive as possible. That being said, we believe our experiments are quite comprehensive in targeting our main research area of “efficient video understanding” and cover 8 strong video benchmarks: Video-MME, MLVU, LongVideoBench, MMBench-Video, LVBench, MVBench, TVBench, and HLVid. Together, these evaluate short and long video understanding, multiple-choice and free-form QA, and high-resolution video perception. There is some coverage of text reading through OCR questions in Video-MME. However, we report aggregate benchmark results rather than a dedicated analysis of document-video understanding. We agree that document-centric evaluation is a valuable extension but we leave that as future work.
Since LiteFrame was trained on video data, as acknowledged in the Limitations section, image evaluation is outside the current experimental scope. Extending our training and evaluation to image understanding is a promising direction, yet the current results do not establish whether replacing the original encoder with LiteFrame preserves image-understanding performance. We would like to emphasize that LiteFrame achieves a state-of-the-art score on HLVid, which requires fine-grained spatial perception via zero-shot spatial resolution scaling (see Section 5.2.3).
[R3W2] LiteFrame with Uniform Frame Sampling
Thank you for raising this question. Yes, LiteFrame supports both standard uniform frame sampling and the proposed dense clip sampling. We originally used dense clip sampling to help our video encoder exploit temporal continuity between frames.
To directly evaluate uniform frame sampling, we apply it to LiteFrame with 64, 128, and 256 input frames. As shown in the table below, uniform sampling performs comparably to dense clip sampling, with a small improvement of 0.3-0.4 percentage points in average accuracy.
These results show that LiteFrame works well with both sampling strategies, suggesting robustness to the larger inter-frame intervals encountered when uniformly sampling long videos. We have added further results and discussion to Appendix I.
[R3W3] Other VLMs
Thank you for raising this concern and we do agree that evaluation in multiple VLM families would be nice. We focused our experiments on InternVL3-8B because (1) it represents a standard, widely adopted architecture for modern Video LLMs (frame-wise ViT + MLP+LLM) and (2) distilling the student on large-scale video datasets is computationally demanding. Appendix F provides some form of generalization, transferring to InternVL3-2B using the LiteFrame encoder, already distilled from InternVL3-8B. Note that these two models have different ViT weights each. Extending our method to other model families (e.g. Qwen, Molmo) is a promising way to strengthen our paper, which we intend to explore in future work, but due to compute limitations, we are unable to provide this analysis in this rebuttal.