- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates performance recovery for multimodal large language models under extremely low visual-token budgets. The authors propose LT-OPD, in which a low-token student generates its own response trajectories and a frozen teacher with full visual tokens provides distributional supervision along these trajectories. The authors also introduce a curriculum that gradually reduces the visual-token budget during training. Experimental results show that LT-OPD substantially recovers model performance while retaining only 5 percent of the visual tokens.
Critical strengths
-
The paper addresses the practically important problem of extreme visual-token reduction with clear motivation.
-
LT-OPD has a simple and intuitive design and introduces no additional modules or computational overhead during inference.
-
The method produces consistent improvements on Qwen3.5-4B, Qwen3.5-9B, GLM-4.6V-9B, and LLaVA-OV-1.5-4B. These results demonstrate a degree of generalization across model scales and architectures.
Critical weaknesses
-
The methodological novelty is relatively limited. The main approach applies existing on-policy distillation techniques to extreme visual-token reduction and uses a same-model teacher with full visual inputs. The budget curriculum is also based on a standard curriculum-learning strategy. Therefore, the primary contribution lies in the application setting and empirical evaluation rather than a fundamentally new learning algorithm.
-
The experiments focus almost exclusively on a final visual-token budget of 5 percent. Although results at 10, 15, and 20 percent are reported for training-free methods, LT-OPD itself is trained and evaluated only at the 5 percent target budget.
-
The comparison of training costs is incomplete. The paper does not report wall-clock time, GPU-hours, or peak memory usage for SFT, fixed-prefix JSD, EPIC, and the reinforcement-learning baselines under the same hardware and data configurations. This makes the additional training cost of LT-OPD difficult to assess.
-
The specific design of the budget-level curriculum appears somewhat hand-crafted. The paper does not sufficiently justify the three-stage training procedure or the selection of its associated hyperparameters.
Questions for authors
Please see weakness.
6 comments
Response to Reviewer prnP (following)
W4. Design of Budget-Level Curriculum Learning
We would like to clarify that Appendix H already includes ablations of the three-stage curriculum. The design follows a simple intuition: the initial high-budget stage provides more visual evidence for early on-policy learning, the intermediate stage gradually increases compression difficulty, and the final stage allows the student to adapt to the target 5% budget.
Our ablations support the role of these stages. Removing both the initial holding stage and the final target stage causes a slight performance drop. These results show that both stages contribute to the effectiveness of the curriculum.
We further evaluate curriculum learning on Qwen3.5-9B. The results are shown below.
| Method | V*Bench | HR-4K | GQA | MMMU | MMB | MME | POPE | TextVQA | OCRB | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-9B(5%) | 54.5 | 59.6 | 49.4 | 51.7 | 77.7 | 1809.4 | 82.3 | 46.6 | 430 | 71.1% |
| +LT-OPD(w/o curriculum) | 70.2 | 66.9 | 53.8 | 51.0 | 79.6 | 2131.3 | 86.2 | 60.1 | 530 | 80.3% |
| +LT-OPD | 70.7 | 68.1 | 53.6 | 52.9 | 80.5 | 2147.9 | 86.2 | 61.5 | 539 | 81.4% |
Adding curriculum improves average retained performance from 80.3% to 81.4% on Qwen3.5-9B, with gains on most benchmarks. The additional training cost is small, as shown below.
| Method | GPUs | TrainingTime(h) | GPU-hours | PeakAllocated/Reserved(GiB) |
|---|---|---|---|---|
| LT-OPD(9B,w/o curriculum) | 8 | 14.5 | 116 | 68.99/78.46 |
| LT-OPD(9B,w/ curriculum) | 8 | 15.1 | 120.8 | 69.30/78.46 |
Curriculum adds only 0.6 training hours on 8 A100 GPUs, while peak reserved GPU memory remains unchanged. These results further support the benefit of progressive budget adaptation across model scales.
We did not extensively tune the stage boundaries, and our aim is to establish the effectiveness of the curriculum design. To further examine its sensitivity to hyperparameters, we vary the initial budget, decay schedule, and stage boundary while keeping the other training settings unchanged. For brevity, we report only the average retained performance across the nine benchmarks.
| Setting | InitialBudget | s1 | s2 | Schedule | Avg.Retain |
|---|---|---|---|---|---|
| Original | 25% | 14 | 100 | Cosine | 82.3% |
| Setting1 | 20% | 14 | 100 | Cosine | 81.4% |
| Setting2 | 25% | 14 | 100 | Linear | 81.9% |
| Setting3 | 25% | 14 | 125 | Cosine | 82.4% |
Average retained performance remains between 81.4% and 82.4% across these settings. The original configuration is also not the best-performing one, suggesting that LT-OPD does not depend on a narrowly tuned curriculum schedule.
Together with the stage ablations and the Qwen3.5-9B results, these experiments support the three-stage design and show that its benefits extend beyond a particular choice of hyperparameters.
Conclusion
We have clarified the motivation and contributions of LT-OPD, and added evaluations at 10% and 20% visual-token budgets. We have also provided a detailed training-cost comparison and further examined curriculum learning through cross-model experiments and hyperparameter sensitivity analysis. We hope these additional results address the reviewer's concerns and provide stronger experimental support for our work.
Follow-up on Our Rebuttal
Dear Reviewer,
Thank you again for your positive assessment of our work and your constructive suggestions. We truly appreciate your recognition of our motivation, simple design, and cross-model results. We would like to briefly summarize the additional evidence and clarifications we provided in response to your concerns.
-
Novelty and contributions (W1): We further clarified the motivation and contributions of LT-OPD, supported by controlled experiments showing the importance of on-policy adaptation under extreme visual compression (77.6% → 82.3% compared with fixed-prefix distillation).
-
Different visual token budgets (W2): We added experiments at 10% and 20% token retention, extending our evaluation beyond the original 5% setting and showing the effectiveness of LT-OPD across different budgets.
-
Training cost (W3): We provided detailed comparisons of training time, GPU-hours, and peak memory with SFT, fixed-prefix JSD, EPIC, and RL baselines, clarifying the training cost and performance trade-off.
-
Curriculum design (W4): We further explained the three-stage curriculum and added hyperparameter sensitivity experiments, showing stable performance (81.2%–82.4%) across different settings.
We have also incorporated important additional results and clarifications into our revised manuscript. We sincerely hope they help address your concerns and strengthen the presentation of our work.
We would be very grateful if you could take a moment to review our response. If you have any remaining concerns or further questions, we would be more than happy to discuss them and provide additional clarification.
Thank you very much for your time, encouragement, and consideration.
Best regards,
The Authors
Response to Reviewer prnP
Thank you for the careful review. We appreciate your positive comments on the motivation, simplicity, and experimental results of LT-OPD. Below we address your concerns and provide additional experiments.
W1. Novelty and Contributions
We would like to further clarify the contributions of LT-OPD.
First, we identify a learning challenge in extreme visual-token reduction. Extreme compression limits the available visual evidence and changes the generation states encountered by MLLMs. This motivates us to study how a heavily compressed model can adapt to its own generation trajectories and recover capabilities under strict visual-token constraints.
Second, we develop a training framework suitable for this setting. LT-OPD allows a low-token student to learn from a full-token self-teacher along student-generated trajectories. We further introduce a budget-level curriculum that gradually reduces visual-token retention, helping the student adapt to increasingly limited visual evidence during training.
Third, our experiments provide useful empirical findings. At 5% visual-token retention, on-policy distillation achieves 81.5% average retained performance, compared with 70.4% for SFT and 77.6% for fixed-prefix distillation with curriculum. Adding the budget-level curriculum further improves performance to 82.3%, while removing its initial high-budget stage reduces performance to 78.0%. These controlled comparisons show the value of adapting to student-generated states and progressively reducing the visual-token budget. We also observe consistent improvements across four MLLM backbones.
We believe these findings contribute useful empirical insights to efficient MLLM training, particularly on how models can recover their capabilities under extreme visual compression. We have further clarified these contributions in the revised manuscript.
W2. Performance at Different Visual-Token Budgets
Our main experiments focus on 5% visual-token retention, where the limited visual evidence makes performance recovery particularly challenging. As discussed in the original manuscript (Lines 344–346), LT-OPD at 5% already outperforms all evaluated training-free methods at 10% retention in average retained performance (82.3% vs. at most 80.2%), and approaches the strongest results at 15%–20% retention.
Following the reviewer's suggestion, we further evaluate LT-OPD at 10% and 20% visual token budgets. The results are shown below.
LT-OPD achieves 86.9% and 87.5% average retained performance at 10% and 20%. The improvements are obvious on TextVQA and OCRBench as more visual tokens become available. These results show that LT-OPD remains effective (with further performance gains) at larger budgets. We also revise Section 4.7 to add some results and visualizations.
W3. Training Cost and GPU Memory
Section 4.5 already reports that LT-OPD takes 15.1 hours on 8 A100 GPUs. We now provide a detailed comparison of training time, total GPU-hours, and peak GPU memory for all training-based baselines.
All experiments use A100 80GB GPUs, with different GPU counts depending on the method. We report both wall-clock time and total GPU-hours for a clearer comparison. Model loading and initialization take only a few minutes, so most of the reported time is spent on training.
SFT, fixed-prefix JSD, and EPIC require less training time as they do not involve on-policy rollouts. LT-OPD takes 120.8 GPU-hours, compared with 96.8–104.8 GPU-hours for GRPO, GSPO, and DAPO. These methods all require rollout generation, while LT-OPD also computes teacher distributions for distillation. Its peak reserved GPU memory (76.22 GiB) is close to that of EPIC (76.01 GiB).
The additional training cost brings substantial performance gains. At 5% visual-token retention, LT-OPD achieves 82.3% average retained performance, compared with 70.4% for SFT, 77.6% for fixed-prefix JSD with curriculum, and 73.7% for EPIC. It also outperforms the evaluated policy optimization baselines.
Importantly, LT-OPD requires no additional modules or teacher model during inference. It preserves the efficiency benefits of visual-token compression, reducing prefill FLOPs by 85.4% and KV-cache usage by 85.2%. We consider the additional training cost worthwhile given the performance recovered at such a limited visual-token budget.