- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes InfDistill, an attention-based knowledge distillation method for compact MLLMs. The authors divide model layers into three information-flow stages: text understanding, multimodal interaction, and response generation. Different attention regions are distilled at each stage. Experiments on Qwen1.5 and Qwen2.5 models across eight multimodal benchmarks show modest performance improvements.
Critical strengths
- Improving knowledge transfer to compact MLLMs is an important problem with practical value.
- The proposed stage-wise attention distillation method is simple and easy to implement.
- The paper includes many ablation studies.
Critical weaknesses
- The improvements over LLaVA-KD are small, ranging from −0.2 to +0.6 points.
- The novelty is limited. Attention distillation have already been studied in prior work. This paper mainly changes which attention regions are distilled at different layer depths.
- As a distillation paper, Table 1 compares against too few existing distillation methods. Even cited methods such as Align-KD and CompoDistill are not experimentally compared.
- The related work is incomplete. Several closely related studies on attention-based or token-interaction-based MLLM distillation are missing [1, 2, 3].
- The training efficiency is not supported by wall-clock time and memory measurements.
- All experiments are limited to the Qwen family. In particular, Figure 2 only studies small Qwen models, which is insufficient to support general claims about MLLMs.
- The use of “scalable” in the title is unclear and potentially misleading. The paper only observes approximately proportional stage boundaries across different sizes of Qwen models, but provides no evidence of computational scalability or generalization across model architectures.
- The presentation could be improved. Figures 2 and 3 do not clearly communicate the main idea, and the Introduction lacks an intuitive overview figure. Author-year citations are also frequently inserted directly into the surrounding text. The authors should consistently use
\citepfor citations. And the introduction is too long.
[1] Jang et al. “Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation.” ECCV 2026.
[2] Chen et al. “Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions.” ICML 2026.
[3] Yu et al. “Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation.” arXiv 2026.
Questions for authors
Please see the weaknesses above.
Dear Reviewer 5xMA,
Could you check if the response has addressed your issues? Since the discussion is closing soon, your reply would be very important.
Best,
AC