acceptodds
Under review as a conference paper at ICLR 2027

Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction

12 views

Abstract

Visual token reduction is an effective way to accelerate multimodal large language models (MLLMs), but performance deteriorates rapidly under extremely low token budgets. Existing work has explored both visual-token selection and training-based adaptation to reduced visual inputs. We take a step further by asking how a heavily compressed MLLM should learn from the states induced by its own generations. This setting naturally calls for on-policy self-distillation: a heavily compressed model is supervised on the states induced by its own generations, while its full-token counterpart serves as an information-rich teacher. Based on this insight, we propose LT-OPD, a training framework for extreme visual-token reduction. The student rolls out responses with only a small fraction of visual tokens, and a frozen full-token copy of the same MLLM provides distributional supervision along these student-generated trajectories. To stabilize on-policy learning when visual evidence is severely limited, we further introduce a budget-level curriculum that progressively decreases the token budget during training. Across nine benchmarks on Qwen3.5-4B, LT-OPD raises average retained performance under 5% visual-token retention from 68.6% to 82.3%, outperforming training-free, training-based, and reinforcement-learning baselines at the same budget. The gains transfer consistently to Qwen3.5-9B, GLM-4.6V-9B, and LLaVA-OV-1.5-4B. LT-OPD also reduces KV-cache usage by 85.2% and prefill FLOPs by 85.4% without additional inference overhead, demonstrating that on-policy learning can substantially recover capabilities lost to extreme visual-token reduction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, token reduction, OPSD

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_prnPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates performance recovery for multimodal large language models under extremely low visual-token budgets. The authors propose LT-OPD, in which a low-token student generates its own response trajectories and a frozen teacher with full visual tokens provides distributional supervision along these trajectories. The authors also introduce a curriculum that gradually reduces the visual-token budget during training. Experimental results show that LT-OPD substantially recovers model performance while retaining only 5 percent of the visual tokens.

Critical strengths
  1. The paper addresses the practically important problem of extreme visual-token reduction with clear motivation.

  2. LT-OPD has a simple and intuitive design and introduces no additional modules or computational overhead during inference.

  3. The method produces consistent improvements on Qwen3.5-4B, Qwen3.5-9B, GLM-4.6V-9B, and LLaVA-OV-1.5-4B. These results demonstrate a degree of generalization across model scales and architectures.

Critical weaknesses
  1. The methodological novelty is relatively limited. The main approach applies existing on-policy distillation techniques to extreme visual-token reduction and uses a same-model teacher with full visual inputs. The budget curriculum is also based on a standard curriculum-learning strategy. Therefore, the primary contribution lies in the application setting and empirical evaluation rather than a fundamentally new learning algorithm.

  2. The experiments focus almost exclusively on a final visual-token budget of 5 percent. Although results at 10, 15, and 20 percent are reported for training-free methods, LT-OPD itself is trained and evaluated only at the 5 percent target budget.

  3. The comparison of training costs is incomplete. The paper does not report wall-clock time, GPU-hours, or peak memory usage for SFT, fixed-prefix JSD, EPIC, and the reinforcement-learning baselines under the same hardware and data configurations. This makes the additional training cost of LT-OPD difficult to assess.

  4. The specific design of the budget-level curriculum appears somewhat hand-crafted. The paper does not sufficiently justify the three-stage training procedure or the selection of its associated hyperparameters.

Questions for authors

Please see weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer prnP

Thank you for the careful review. We appreciate your positive comments on the motivation, simplicity, and experimental results of LT-OPD. Below we address your concerns and provide additional experiments.

W1. Novelty and Contributions

We would like to further clarify the contributions of LT-OPD.

First, we identify a learning challenge in extreme visual-token reduction. Extreme compression limits the available visual evidence and changes the generation states encountered by MLLMs. This motivates us to study how a heavily compressed model can adapt to its own generation trajectories and recover capabilities under strict visual-token constraints.

Second, we develop a training framework suitable for this setting. LT-OPD allows a low-token student to learn from a full-token self-teacher along student-generated trajectories. We further introduce a budget-level curriculum that gradually reduces visual-token retention, helping the student adapt to increasingly limited visual evidence during training.

Third, our experiments provide useful empirical findings. At 5% visual-token retention, on-policy distillation achieves 81.5% average retained performance, compared with 70.4% for SFT and 77.6% for fixed-prefix distillation with curriculum. Adding the budget-level curriculum further improves performance to 82.3%, while removing its initial high-budget stage reduces performance to 78.0%. These controlled comparisons show the value of adapting to student-generated states and progressively reducing the visual-token budget. We also observe consistent improvements across four MLLM backbones.

We believe these findings contribute useful empirical insights to efficient MLLM training, particularly on how models can recover their capabilities under extreme visual compression. We have further clarified these contributions in the revised manuscript.

W2. Performance at Different Visual-Token Budgets

Our main experiments focus on 5% visual-token retention, where the limited visual evidence makes performance recovery particularly challenging. As discussed in the original manuscript (Lines 344–346), LT-OPD at 5% already outperforms all evaluated training-free methods at 10% retention in average retained performance (82.3% vs. at most 80.2%), and approaches the strongest results at 15%–20% retention.

Following the reviewer's suggestion, we further evaluate LT-OPD at 10% and 20% visual token budgets. The results are shown below.

MethodV*BenchHR-4KGQAMMMUMMBMMEPOPETextVQAOCRBAvg.
LT-OPD(5%)74.973.054.850.078.92077.086.356.053582.3%
LT-OPD(10%)75.473.057.650.780.22212.788.670.860286.9%
LT-OPD(20%)75.973.055.150.980.82185.488.675.163587.5%

LT-OPD achieves 86.9% and 87.5% average retained performance at 10% and 20%. The improvements are obvious on TextVQA and OCRBench as more visual tokens become available. These results show that LT-OPD remains effective (with further performance gains) at larger budgets. We also revise Section 4.7 to add some results and visualizations.

W3. Training Cost and GPU Memory

Section 4.5 already reports that LT-OPD takes 15.1 hours on 8 A100 GPUs. We now provide a detailed comparison of training time, total GPU-hours, and peak GPU memory for all training-based baselines.

All experiments use A100 80GB GPUs, with different GPU counts depending on the method. We report both wall-clock time and total GPU-hours for a clearer comparison. Model loading and initialization take only a few minutes, so most of the reported time is spent on training.

MethodGPUsWallTime(h)GPU-hoursPeakAllocated/Reserved(GiB)
LT-OPD815.1120.867.85/76.22
Fixed-prefixJSD44.919.662.62/75.13
GRPO812.499.255.50/62.96
GSPO812.196.855.44/62.96
DAPO813.1104.865.88/73.04
SFT22.95.854.24/68.40
EPIC86.350.462.89/76.01

SFT, fixed-prefix JSD, and EPIC require less training time as they do not involve on-policy rollouts. LT-OPD takes 120.8 GPU-hours, compared with 96.8–104.8 GPU-hours for GRPO, GSPO, and DAPO. These methods all require rollout generation, while LT-OPD also computes teacher distributions for distillation. Its peak reserved GPU memory (76.22 GiB) is close to that of EPIC (76.01 GiB).

The additional training cost brings substantial performance gains. At 5% visual-token retention, LT-OPD achieves 82.3% average retained performance, compared with 70.4% for SFT, 77.6% for fixed-prefix JSD with curriculum, and 73.7% for EPIC. It also outperforms the evaluated policy optimization baselines.

Importantly, LT-OPD requires no additional modules or teacher model during inference. It preserves the efficiency benefits of visual-token compression, reducing prefill FLOPs by 85.4% and KV-cache usage by 85.2%. We consider the additional training cost worthwhile given the performance recovered at such a limited visual-token budget.

AuthorsOpenReview ↗

Response to Reviewer prnP (following)

W4. Design of Budget-Level Curriculum Learning

We would like to clarify that Appendix H already includes ablations of the three-stage curriculum. The design follows a simple intuition: the initial high-budget stage provides more visual evidence for early on-policy learning, the intermediate stage gradually increases compression difficulty, and the final stage allows the student to adapt to the target 5% budget.

Our ablations support the role of these stages. Removing both the initial holding stage and the final target stage causes a slight performance drop. These results show that both stages contribute to the effectiveness of the curriculum.

We further evaluate curriculum learning on Qwen3.5-9B. The results are shown below.

MethodV*BenchHR-4KGQAMMMUMMBMMEPOPETextVQAOCRBAvg.
Qwen3.5-9B(5%)54.559.649.451.777.71809.482.346.643071.1%
+LT-OPD(w/o curriculum)70.266.953.851.079.62131.386.260.153080.3%
+LT-OPD70.768.153.652.980.52147.986.261.553981.4%

Adding curriculum improves average retained performance from 80.3% to 81.4% on Qwen3.5-9B, with gains on most benchmarks. The additional training cost is small, as shown below.

MethodGPUsTrainingTime(h)GPU-hoursPeakAllocated/Reserved(GiB)
LT-OPD(9B,w/o curriculum)814.511668.99/78.46
LT-OPD(9B,w/ curriculum)815.1120.869.30/78.46

Curriculum adds only 0.6 training hours on 8 A100 GPUs, while peak reserved GPU memory remains unchanged. These results further support the benefit of progressive budget adaptation across model scales.

We did not extensively tune the stage boundaries, and our aim is to establish the effectiveness of the curriculum design. To further examine its sensitivity to hyperparameters, we vary the initial budget, decay schedule, and stage boundary while keeping the other training settings unchanged. For brevity, we report only the average retained performance across the nine benchmarks.

SettingInitialBudgets1s2ScheduleAvg.Retain
Original25%14100Cosine82.3%
Setting120%14100Cosine81.4%
Setting225%14100Linear81.9%
Setting325%14125Cosine82.4%

Average retained performance remains between 81.4% and 82.4% across these settings. The original configuration is also not the best-performing one, suggesting that LT-OPD does not depend on a narrowly tuned curriculum schedule.

Together with the stage ablations and the Qwen3.5-9B results, these experiments support the three-stage design and show that its benefits extend beyond a particular choice of hyperparameters.

Conclusion

We have clarified the motivation and contributions of LT-OPD, and added evaluations at 10% and 20% visual-token budgets. We have also provided a detailed training-cost comparison and further examined curriculum learning through cross-model experiments and hyperparameter sensitivity analysis. We hope these additional results address the reviewer's concerns and provide stronger experimental support for our work.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your positive assessment of our work and your constructive suggestions. We truly appreciate your recognition of our motivation, simple design, and cross-model results. We would like to briefly summarize the additional evidence and clarifications we provided in response to your concerns.

  1. Novelty and contributions (W1): We further clarified the motivation and contributions of LT-OPD, supported by controlled experiments showing the importance of on-policy adaptation under extreme visual compression (77.6% → 82.3% compared with fixed-prefix distillation).

  2. Different visual token budgets (W2): We added experiments at 10% and 20% token retention, extending our evaluation beyond the original 5% setting and showing the effectiveness of LT-OPD across different budgets.

  3. Training cost (W3): We provided detailed comparisons of training time, GPU-hours, and peak memory with SFT, fixed-prefix JSD, EPIC, and RL baselines, clarifying the training cost and performance trade-off.

  4. Curriculum design (W4): We further explained the three-stage curriculum and added hyperparameter sensitivity experiments, showing stable performance (81.2%–82.4%) across different settings.

We have also incorporated important additional results and clarifications into our revised manuscript. We sincerely hope they help address your concerns and strengthen the presentation of our work.

We would be very grateful if you could take a moment to review our response. If you have any remaining concerns or further questions, we would be more than happy to discuss them and provide additional clarification.

Thank you very much for your time, encouragement, and consideration.

Best regards,
The Authors

Reviewer_USsCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses performance degradation of MLLMs under extreme visual-token budgets (e.g., 5%). The authors observe that severe compression not only removes visual evidence but also shifts the generation-state distribution away from that induced by full-token inputs. They propose LT-OPD: a low-token student rolls out its own trajectories, a frozen full-token copy of the same MLLM provides next-token distributional supervision on these student-generated prefixes, and a budget-level curriculum stabilizes training.

Critical strengths
  1. Well-controlled attribution of the on-policy effect. The ablation ladder (SFT 70.4% → +curriculum 73.1% → +curriculum+fixed-prefix JSD 77.6% → OPD w/o curriculum 81.5% → full 82.3%) cleanly isolates the contribution of on-policy states vs. distillation vs. curriculum, which is good and clear.
  2. Comprehensive benchmarks. Nine benchmarks across four task categories, three model families, two scales, and a comparison against three modern RL algorithms. Reproducibility details (seeds, batch composition, prompts, splits) are complete.
Critical weaknesses

1. Limited novelty. The proposed method mainly applies existing On-Policy Distillation (OPD) techniques to visual token compression, together with curriculum learning. Both components have been explored in prior work. In particular, Vision-OPD also uses a teacher with richer visual information for on-policy distillation. The key differences between LT-OPD and Vision-OPD remain unclear, making the novelty of the proposed method somewhat limited.

2. Limited generalizability evaluation. Although LT-OPD is evaluated on multiple MLLMs, the experiments mainly use CDPruner for visual token compression. Since different pruning methods preserve different types of visual information, it is unclear whether LT-OPD can achieve similar improvements with other compression methods.

3. Marginal benefits of curriculum learning. Curriculum learning improves the average retained performance by only 0.8 percentage points. Moreover, it slightly reduces performance on POPE (86.5 → 86.3) and TextVQA (56.3 → 56.0). These results raise questions about the necessity of this component.

4. Missing related work. Several relevant studies on visual token compression are not discussed, including:

  • SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
  • VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration
  • TimeChat-Online: 80% Visual Tokens Are Naturally Redundant in Streaming Videos
  • StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding
Questions for authors

What are the key methodological differences between LT-OPD and Vision-OPD beyond the visual token compression setting?

Could the authors provide a more detailed analysis of end-to-end inference speedup under different input lengths and generation settings?

8 comments
AuthorsOpenReview ↗

Response to Reviewer USsC

Thank you for the careful review. We appreciate your positive comments on our controlled ablations and comprehensive evaluation. Below, we address your concerns and questions.

W1&Q1. Novelty and Differences from Vision-OPD

We would like to clarify the differences between LT-OPD and Vision-OPD in their motivations, learning settings, and training designs.

Different research motivations. Vision-OPD investigates the regional-to-global perception gap, where MLLMs can recognize fine-grained details more accurately from evidence-centered crops than from full images. LT-OPD studies a different phenomenon under extreme visual compression: severe token reduction also changes the generation trajectories encountered by the model. As Reviewer k5fS also noted, this is an interesting observation that has received limited attention to previous visual-token reduction methods. Our work studies how MLLMs can adapt to the generation states induced by severely compressed visual inputs.

We further examine this effect through controlled comparisons with fixed-prefix training. At the same 5% budget, fixed-prefix distillation with curriculum achieves 77.6% average retained performance, while LT-OPD reaches 82.3%. This improvement shows the value of learning from student-generated trajectories under extreme visual compression.

Different teacher–student information settings. Vision-OPD uses a crop-conditioned teacher to transfer fine-grained visual knowledge to a full-image student. In LT-OPD, the teacher and student receive the same image-question pair, with the difference in visual information introduced through token reduction. The frozen teacher receives all visual tokens, while the student operates with only a small fraction of them. In short, the student learns to operate with much less visual information, while Vision-OPD uses richer, localized visual supervision to improve perception. LT-OPD also does not require explicit identification of important image regions to construct its distillation supervision. This creates a different training challenge: recovering full-token behavior under strict visual-token constraints.

Budget-level curriculum learning. LT-OPD introduces a training schedule that gradually reduces visual-token retention from 25% to 5%. This allows the student to start with more visual evidence and form useful on-policy trajectories before reaching the target budget. Our ablations show that removing either the initial high-budget holding stage or the final target-budget holding stage reduces performance. These results show the role of visual-token availability as a factor in training difficulty.

Overall, LT-OPD studies an important learning challenge under extreme visual compression and develops a training framework for this setting. Across nine benchmarks, LT-OPD improves average retained performance from 68.6% to 82.3% at 5% visual-token retention, with consistent gains across different MLLM architectures and model scales. These gains preserve the efficiency benefits of compression, including an 85.4% reduction in prefill FLOPs and an 85.2% reduction in KV-cache usage. We have added a dedicated discussion of these differences in the revised manuscript (Section 3.4).

W2. Generality Across Pruning Methods.

LT-OPD can be applied to different visual-token pruning methods. We chose CDPruner mainly for easy integration into our training pipeline. At 5% retention, its average retained performance (68.6%) is below DivPrune (77.6%) and ZOO-Prune (72.7%). As detailed in Appendix B, CDPruner selects tokens after the native visual merger and before the language model. This makes it easy to integrate with our training pipeline and compatible with FlashAttention. It also reduces prefill FLOPs from 6.002 T to 0.878 T (85.4%, Table 5). To test LT-OPD with other pruning methods, we further apply it to DART and DivPrune. We have added these results to the revised main table (Section 4.2). All methods below retain 5% visual tokens.

MethodV*BenchHR-4KGQAMMMUMMBMMEPOPETextVQAOCRBenchAvg.
DART52.957.651.142.871.81,771.381.752.347771.4%
DivPrune70.774.454.643.777.21,834.886.048.646777.6%
CDPruner52.968.450.445.774.01,531.870.243.240468.6%
EPIC68.672.843.147.377.81,781.981.140.547073.7%
LearnPruner66.061.448.747.173.31,534.973.935.040368.5%
LT-OPD(DART)71.271.553.151.477.11,999.386.254.052180.4%
LT-OPD(DivPrune)74.975.955.949.080.22,030.787.561.158684.1%
LT-OPD(CDPruner,paper)74.973.054.850.078.92,077.086.356.053582.3%

All LT-OPD variants outperform their training-free counterparts and training-based methods. Notably, LT-OPD with DivPrune reaches 84.1%, exceeding our original CDPruner-based result. These results show that LT-OPD is effective with different pruning strategies.

AuthorsOpenReview ↗

Response to Reviewer USsC (following)

W3. Effectiveness of Budget-Level Curriculum Learning

We thank the reviewer for raising this point. The main purpose of our curriculum is to help the student form useful on-policy trajectories at the beginning of training, when the target 5% budget may leave too little visual evidence.

As shown in Table 2 (main text), adding curriculum to OPD improves average retained performance from 81.5% to 82.3%. The gain can be seen on most benchmarks. We further examine the three-stage schedule in Appendix H, where removing either holding stage leads to a drop. These results suggest that both stages play a useful role: the initial stage provides more visual evidence for early rollouts, and the final stage allows the model to adapt to the target budget.

We also observe that curriculum improves vanilla SFT from 70.4% to 73.1%, showing that progressive budget reduction can benefit other training objectives as well.

We further evaluate curriculum learning on Qwen3.5-9B, where it improves average retained performance from 80.3% to 81.4%, with gains on most benchmarks. The tables below show the results.

MethodV*BenchHR-4KGQAMMMUMMBMMEPOPETextVQAOCRBAvg.
Qwen3.5-9B(5%visual token)54.559.649.451.777.71,809.4082.346.643071.1%
+LT-OPD(w/o curriculum)70.266.953.85179.62,131.3086.260.153080.3%
+LT-OPD70.768.153.652.980.52,147.9086.261.553981.4%
MethodGPUsTraining time(h)GPU-hoursPeak allocated/reserved memory(GiB)
LT-OPD(9B,w/o curriculum)814.5116.068.99/78.46
LT-OPD(9B,w/ curriculum)815.1120.869.30/78.46

This improvement requires only 0.6 additional training hours on 8 A100 GPUs (14.5 vs. 15.1 hours), while peak GPU memory remains nearly unchanged. This also suggests that curriculum learning improves performance at the target 5% token budget with minimal additional training cost.

W4. Missing Related Work

Thank you for pointing out these relevant studies. We have revised Section 2.1 to include all four papers.

We added SparseVLM and VisionTrim to the discussion of training-free visual-token reduction, covering text-guided token selection, sparsification, and merging. We also added TimeChat-Online and StreamingAssistant to the discussion of visual-token reduction for streaming video understanding.

We have updated the section to better cover both training-free and training-based methods, and to explain how LT-OPD fits into this line of work. The changes are included in the revised PDF and will be kept in the final version.

AuthorsOpenReview ↗

Response to Reviewer USsC (following)

Q2: End-to-End Inference Speedup

Thank you for the suggestion. We extend our efficiency evaluation to different visual input lengths and output lengths using 128 questions from HR-Bench 4K. The short and long visual inputs contain 783 and 12,075 tokens on average before pruning, respectively. We evaluate output lengths of 32, 128, and 1,024 maximum output tokens. Each setting is repeated 5 times, giving a total of 7,680 end-to-end latency measurements across the two methods. The results are shown below.

Computational cost:

InputMethodKV Cache(MB)↓FLOPs Ratio↓Prefill FLOPs(T)↓
Short (783)Full-token28.9616.541
LT-OPD (5%)4.610.1581.035
Long (12,075)Full-token398.991111.222
LT-OPD (5%)23.110.0475.213

End-to-end latency (ms/sample ↓) and speedup:

InputMethod32 tokens128 tokens1,024 tokens
ShortFull-token391.321328.0410137.36
LT-OPD (5%)354.23 (1.105×)1285.83 (1.033×)10033.41 (1.010×)
LongFull-token2904.393919.413461.28
LT-OPD (5%)2360.60 (1.230×)3289.38 (1.192×)12107.66 (1.112×)

LT-OPD achieves 1.105×–1.230× end-to-end speedup with 32-token outputs, with larger gains on high-resolution inputs. For the longer visual inputs, it maintains 1.192× speedup at 128 maximum output tokens and 1.112× at 1,024 maximum output tokens, while reducing prefill FLOPs by 95.3% and KV-cache usage by 94.2%.

The end-to-end speedup becomes smaller as output length increases, since autoregressive decoding takes up a larger share of the total inference time. Overall, LT-OPD brings the clearest latency gains with high-resolution visual inputs and shorter outputs. These results also help explain the speedup reported under the original evaluation setting in Table 5.

Conclusion

We have clarified the differences from Vision-OPD, added experiments with two more pruning methods, and further examined the role of curriculum learning. We have also expanded the related work and provided a more detailed analysis of inference efficiency. We hope these results help address the reviewer's concerns and strengthen the experimental support for our work.

Reviewer_k5fSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets extreme visual token reduction for multimodal large language models (MLLMs), where only a tiny fraction (e.g., 5%) of visual tokens are retained for speeding up inference. Existing token pruning and training‑based adaptation approaches suffer severe performance degradation under such extreme token budgets. The authors point out a observation: heavy visual compression not only discards visual information but also alters the model’s generation trajectories at inference. The authors propose LT‑OPD. It has two core components: On‑policy self‑distillation and Budget‑level curriculum learning Experiments are conducted across multiple MLLM backbones over 9 single‑image benchmarks, plus multi‑image/video benchmarks.

Critical strengths
  1. A interesting insight: prior works ignore that compressed models produce different generation trajectories.
  2. Propose two effective component.
Critical weaknesses
  1. The proposed LT-OPD essentially performs conditional post-training for multimodal large language models, specifically adapted to a particular pruning operator (CDPruner). Consequently, it is no longer a general-purpose visual token selection approach. Its practical performance is limited by several interconnected constraints: it relies heavily on the distribution of the in-house LT-14K training dataset, is tied to the specific behavior of the underlying token-pruning algorithm, and requires significant computational resources for full-parameter fine-tuning. When switching to alternative pruning methods, migrating to unseen data distributions, or scaling to larger model sizes, additional retraining is necessary, which limits its ease of use compared to purely visual-token reduction techniques.
  2. Image-based training may negatively impact other potential capabilities, such as long video understand, visual-based multi-turn, GUI.
  3. The paper does not conduct dedicated diagnostic analysis to examine whether its post‑training adaptation encourages the compressed student model to overly rely on textual language priors while underutilizing scarce visual evidence
Questions for authors

Is it better to use this strategy to train a visual token compression module?

7 comments
AuthorsOpenReview ↗

Response to Reviewer k5fS

Thank you for your careful review and comments. We are very glad that you recognize our interesting insight and proposed effective components. Here are responses to your concerns and questions:

W1. Generality across pruning methods, training data, and model scales.
LT-OPD does not depend on a specific pruning method. We choose CDPruner for its simple integration into our training pipeline, and do not choose "the strongest pruning method" to help LT-OPD's performance. Its performance at 5% budget (68.6%) is actually lower than DivPrune (77.6%) and ZOO-Prune (72.7%). As detailed in Appendix B, CDPruner selects visual tokens after the native visual merger and before the language model. This is compatible with modern training and inference implementations, including FlashAttention. It reduces prefill FLOPs from 6.002 T to 0.878 T.

To examine whether LT-OPD also works with other pruning methods, we further apply it to DART and DivPrune. The results are shown below, and we have added them to the revised main table (Section 4.2).

MethodV*BenchHR-4KGQAMMMUMMBMMEPOPETextVQAOCRBenchAvg.
DART52.957.651.142.871.81,771.381.752.347771.4%
DivPrune70.774.454.643.777.21,834.886.048.646777.6%
CDPruner52.968.450.445.774.01,531.870.243.240468.6%
EPIC68.672.843.147.377.81,781.981.140.547073.7%
LearnPruner66.061.448.747.173.31,534.973.935.040368.5%
LT-OPD(DART)71.271.553.151.477.11,999.386.254.052180.4%
LT-OPD(DivPrune)74.975.955.949.080.22,030.787.561.158684.1%
LT-OPD(CDPruner,paper)74.973.054.850.078.92,077.086.356.053582.3%

LT-OPD improves the average retained performance of DART from 71.4% to 80.4%, and DivPrune from 77.6% to 84.1%. Both variants also outperform the other training-based baselines. In particular, LT-OPD with DivPrune achieves 84.1%, even higher than our original CDPruner-based result (82.3%). These results show that the effectiveness of LT-OPD extends to different token pruning strategies.

LT-OPD does not depend on the LT-14K training set. We constructed LT-14K to help preserve MLLM capabilities across different domains under extreme visual compression. It includes fine-grained perception, general VQA, and reasoning data, without tuning the data mixture for specific evaluation benchmarks. To examine the dependence on training data, we randomly sample 11K questions from PixMo Ask Model Anything, one of the sources used in LT-14K, and train LT-OPD using only this subset. The results are shown below.

MethodV*BenchHR-4KGQAMMMUMMBMMEPOPETVQAOCRBAvg.
CDPruner52.968.450.445.774.01,531.870.243.240468.6%
EPIC68.672.843.147.377.81,781.981.140.547073.7%
LearnPruner66.061.448.747.173.31,534.973.935.040368.5%
LT-OPD+PixMo71.170.453.748.477.61,940.585.253.251079.3%

With fewer training samples from a single source, LT-OPD still achieves 79.3% average retained performance, outperforming CDPruner (68.6%), EPIC (73.7%), and LearnPruner (68.5%). Notably, EPIC and LearnPruner are trained on the full LT-14K dataset. This result shows that LT-OPD remains effective with a smaller, single-source training set, without relying on the full LT-14K mixture.

Effectiveness on larger model scales. We have already evaluated LT-OPD on two 9B MLLMs, Qwen3.5-9B and GLM-4.6V-9B, as reported in Tables 3 and 4 of our original submission. Both models are trained with full-parameter updates.

MethodV*BenchHR-4KGQAMMMUMMBMMEPOPETVQAOCRBAvg.
Qwen3.5-9B(5%)54.559.649.451.777.71,809.482.346.643071.1%
Qwen3.5-9B(5%)+LT-OPD70.768.153.652.980.52,147.986.261.553981.4%
GLM-4.6V-9B(5%)45.060.546.842.369.21,646.076.356.134673.8%
GLM-4.6V-9B(5%)+LT-OPD51.363.453.848.383.42,146.383.364.849786.3%

LT-OPD improves average retained performance by 10.3 percentage points on Qwen3.5-9B and 12.5 percentage points on GLM-4.6V-9B. These results show that the gains also hold for larger models and different architectures. Training on 20B or 32B models exceeds the memory capacity of our current 8×A100 80GB setup, so we cannot do it now. We consider extending LT-OPD to these larger models an interesting direction for future work.

Overall, these experiments show that LT-OPD can be trained with different pruning methods, remains effective with a smaller single-source training set, and provides consistent gains across model architectures and scales. While each backbone and visual token reduction method requires adaptation (also consistent with EPIC, one baseline), the trained model can be used across downstream tasks without further task-specific training.

AuthorsOpenReview ↗

Response to Reviewer k5fS (following)

W2. Image-based training, Evaluation Scope and Other Capabilities

Thank you for the suggestion. Our evaluation is consistent with prior work and still outperforms baselines in the suggested tasks. Our original evaluation follows common practice in visual token reduction studies, including CDPruner, DivPrune, EPIC, HiPrune, and ZOO-Prune, where single-image VQA, OCR, and visual perception benchmarks are widely used. We also include perception-heavy benchmarks to examine performance under extreme visual token reduction.

We would also like to clarify that our original submission already includes evaluations on MuirBench (multi-image understanding) and MVBench (video understanding) in Appendix I. LT-OPD achieves 52.0 and 55.6, respectively, outperforming CDPruner (39.1/51.6) and EPIC (44.0/52.2).

Following the reviewer's suggestion, we further evaluate LT-OPD on three additional benchmarks: LongVideoBench [1] for long-video understanding, MMDU [2] for multi-turn multimodal conversations, and ScreenSpot-v2 [3] for GUI tasks. All compressed methods retain 5% visual tokens.

MethodLongVideoBenchMMDUScreenSpot-v2
CDPruner49.346.74.2
DivPrune56.647.46.4
LearnPruner49.047.16.4
EPIC55.848.28.0
LT-OPD59.450.014.0

Long-video understanding. We use 32 frames per video. Full-token Qwen3.5-4B achieves 62.30. LT-OPD achieves 59.4 on LongVideoBench (only 2.9 points lower), outperforming the strongest baseline, DivPrune (56.6), by 2.8 points. Notably, in this task with very long visual context, LT-OPD achieves up to 1.85× faster per-token decoding than the full-token model, averaged over 10 runs.

Multi-turn conversations. LT-OPD achieves 50.0 on MMDU, compared with 57.0 for the full-token Qwen3.5-4B. It outperforms all compressed baselines (46.7–48.2) and is the only compressed method to reach 50.0.

GUI tasks. ScreenSpot-v2 is very challenging under the 5% visual-token budget, where fine-grained spatial information and tiny layouts and buttons are heavily compressed. The full-token model achieves 92.1, while the compressed baselines achieve only 4.2–8.0. LT-OPD improves the score to 14.0, achieving the highest performance among all evaluated compressed methods.

Together with our existing MuirBench and MVBench results, these additional experiments show that LT-OPD consistently outperforms training-free and training-based baselines across all evaluated tasks, including long-video understanding, multi-turn interaction, and GUI tasks, despite being trained solely on single-image data. The results also highlight the remaining challenges of extreme visual compression on tasks requiring precise spatial grounding.

[1] Wu H, Li D, Chen B, et al. Longvideobench: A benchmark for long-context interleaved video-language understanding[J]. Advances in Neural Information Processing Systems, 2024, 37: 28828-28857.

[2] Liu Z, Chu T, Zang Y, et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms[J]. Advances in Neural Information Processing Systems, 2024, 37: 8698-8733.

[3] Wu, Zhiyong, et al. "OS-ATLAS: Foundation action model for generalist GUI agents." International Conference on Learning Representations. Vol. 2025. 2025.

W3. Textual Language Prior

Thank you for this thoughtful comment. We have carefully checked LT-14K to avoid overlap with the evaluation benchmarks, as documented in Appendix A (Lines 706–707). To further examine whether LT-OPD uses the remaining visual evidence, we evaluate it on MMVP (CVPR 2024) [1]. MMVP contains paired images with small but important visual differences, and the paired images share the same question. It is useful for testing whether MLLMs can distinguish visual facts from similar-looking inputs, and suitable for testing "language prior".

MethodMMVP(Pair Acc%)
Qwen3.5-4B(Full)59.3
CDPruner(5%)24.7
DivPrune(5%)29.3
LT-OPD(5%)37.3

LT-OPD largely improves MMVP performance from 24.7% to 37.3%. Since MMVP requires models to distinguish fine-grained visual differences, this result provides further evidence that LT-OPD preserves the ability to use visual information under extreme compression. These gains suggest that LT-OPD does not rely on language priors alone.

One possible explanation is that the retained visual tokens still carry information from other image regions through self-attention, as the attention mechanism discussed by Abnar and Zuidema [2]. LT-OPD may help the student make better use of this information through on-policy adaptation.

[1] Tong, Shengbang, et al. "Eyes wide shut? exploring the visual shortcomings of multimodal llms." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Abnar, Samira, and Willem Zuidema. "Quantifying attention flow in transformers." Proceedings of the 58th annual meeting of the association for computational linguistics. 2020.

AuthorsOpenReview ↗

Response to Reviewer k5fS (following)

Question: Can compressor-only training work well?

We'd like to clarify that the LearnPruner baseline in our paper already follows this setting, where only the learnable token scorer is trained. We further adopt its scorer in LT-OPD, freezing the MLLM backbone while applying our on-policy distillation and budget-level curriculum.

MethodV*BenchHR-4KGQAMMMUMMBMMEPOPETextVQAOCRBenchAVG Retain Ratio
scorer-only+OPD+curriculum42.959.948.748.372.71,701.075.826.033564.3%
LearnPruner66.061.448.747.173.31,534.973.935.040368.5%
LT-OPD74.973.054.850.078.92,077.086.356.053582.3%

This variant achieves only 64.3% average retained performance, compared with 68.5% for LearnPruner and 82.3% for full LT-OPD. Consequently, only training a selection module under extreme visual token reduction does not work well.

One possible explanation is that extreme visual token reduction inevitably changes the student's generation trajectories, regardless of the pruning strategy. With a frozen backbone, adapting to these shifted states becomes very challenging, as optimizing the token selection module alone provides limited flexibility to recover the lost capabilities. Nevertheless, we believe jointly learning visual token selection and on-policy adaptation is a promising direction, which we plan to explore in future work.

Conclusion

Overall, these additional experiments show that LT-OPD works with different pruning methods and training data, and improves performance across model scales and tasks beyond single-image understanding. Our MMVP results also show clear gains in fine-grained visual grounding, supporting the use of visual evidence under extreme compression. We have added the new results to the revised manuscript and clarified the raised concerns of our approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.