- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies visual grounding in multimodal on-policy distillation. The authors argue that standard OPD only matches token distributions and does not directly teach the student which visual regions to use. They propose Visual-Align OPD, which adds a visual attention alignment loss between the teacher and student at selected decoder layers. Experiments on InternVL3 and Qwen3-VL across seven multimodal benchmarks show improvements over standard OPD on most tasks.
Critical strengths
- The problem is important since standard OPD may not fully use visual information.
- The proposed method is simple and effective.
- The experiments cover two model families and several visual tasks.
Critical weaknesses
- The novelty is limited. Prior work has already shown that visual attention distillation can improve multimodal students [1, 2, 3]. This paper mainly integrates this established technique into OPD, so the contribution appears incremental.
- The related work is substantially incomplete. A large body of MLLM distillation research is not discussed, including LLaVA-KD [4], Align-TI [5], Hide to See [6], LLaVA-MoD [7], and LLaVADI [8]. This makes it difficult to understand how the proposed method differs from existing MLLM distillation approaches.
- Sections 3.1 and 3.2 repeat the descriptions of response sampling and the OPD objective. Equations 4 and 5 could also be combined into a single definition of the overall OPD loss.
- Equations 6 to 11 use too many equations to explain attention map extraction. A simple figure would make the method easier to understand.
- Algorithm 1 appears before the relevant definitions and can be confusing. It would be clearer to place it after Section 3.2.
[1] Feng et al. "Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement." CVPR 2025.
[2] Kim et al. "CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs." ICLR 2026.
[3] Jang et al. "Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation." ECCV 2026.
[4] Cai et al. "LLaVA-KD: A Framework of Distilling Multimodal Large Language Models." ICCV 2025.
[5] Chen et al. "Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions." ICML 2026.
[6] Yu et al. "Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation." arXiv 2026.
[7] Shu et al. "LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation." ICLR 2025.
[8] Xu et al. "LLAVADI: What Matters For Multimodal Large Language Models Distillation." arXiv 2024.
Questions for authors
Please see the weaknesses.