acceptodds
Under review as a conference paper at ICLR 2027

Seeing Before Distilling: Prefill Visual Alignment for Multimodal On-Policy Distillation

Abstract

On-policy distillation (OPD) mitigates the distribution mismatch of offline distillation by training students on their own generated trajectories. However, we find that vanilla OPD is often insufficient for vision-language models, especially on visually dominated tasks, where students may attend to irrelevant image regions and weaken subsequent token-level distillation. To address this issue, we propose Prefill Visual Alignment On-Policy Distillation, which aligns the student’s visual-token attention with the teacher during the prefill stage. This avoids task-specific decoded-token selection and provides a simple, unified supervision signal across multimodal tasks. Experiments on InternVL3 and Qwen3-VL show that our method improves on most benchmarks over vanilla OPD, e.g., from 44.8 to 51.5 on InfoVQA and from 69.1 to 71.8 on OCRBench. These results demonstrate that prefill-stage visual alignment strengthens visual grounding and enables more effective multimodal knowledge transfer.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
On-Policy Distillation, Vision-Language Models, Visual Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8nUYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual grounding in multimodal on-policy distillation. The authors argue that standard OPD only matches token distributions and does not directly teach the student which visual regions to use. They propose Visual-Align OPD, which adds a visual attention alignment loss between the teacher and student at selected decoder layers. Experiments on InternVL3 and Qwen3-VL across seven multimodal benchmarks show improvements over standard OPD on most tasks.

Critical strengths
  1. The problem is important since standard OPD may not fully use visual information.
  2. The proposed method is simple and effective.
  3. The experiments cover two model families and several visual tasks.
Critical weaknesses
  1. The novelty is limited. Prior work has already shown that visual attention distillation can improve multimodal students [1, 2, 3]. This paper mainly integrates this established technique into OPD, so the contribution appears incremental.
  2. The related work is substantially incomplete. A large body of MLLM distillation research is not discussed, including LLaVA-KD [4], Align-TI [5], Hide to See [6], LLaVA-MoD [7], and LLaVADI [8]. This makes it difficult to understand how the proposed method differs from existing MLLM distillation approaches.
  3. Sections 3.1 and 3.2 repeat the descriptions of response sampling and the OPD objective. Equations 4 and 5 could also be combined into a single definition of the overall OPD loss.
  4. Equations 6 to 11 use too many equations to explain attention map extraction. A simple figure would make the method easier to understand.
  5. Algorithm 1 appears before the relevant definitions and can be confusing. It would be clearer to place it after Section 3.2.

[1] Feng et al. "Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement." CVPR 2025.

[2] Kim et al. "CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs." ICLR 2026.

[3] Jang et al. "Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation." ECCV 2026.

[4] Cai et al. "LLaVA-KD: A Framework of Distilling Multimodal Large Language Models." ICCV 2025.

[5] Chen et al. "Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions." ICML 2026.

[6] Yu et al. "Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation." arXiv 2026.

[7] Shu et al. "LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation." ICLR 2025.

[8] Xu et al. "LLAVADI: What Matters For Multimodal Large Language Models Distillation." arXiv 2024.

Questions for authors

Please see the weaknesses.

Reviewer_uk8aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses on-policy distillation for vision-language models and argues that token-level OPD alone is insufficient because the student attends to task-irrelevant image regions, so the teacher's token-level supervision is applied on a misaligned visual-evidence path. They proposed Visual-Align OPD, adding a KL term that aligns the student's attention distribution over visual tokens with the teacher's.

Critical strengths
  • The problem is stated concretely and overall clearly written, easy-to-follow.
  • The paper shows that vanilla OPD can fall below the base student on ChartQA and both captioning benchmarks for two different model families.
  • On-policy distillation is becoming the default LLM distillation recipe; asking what breaks when it is applied to VLMs, and proposing a modality-specific correction, is instersting and would be beneficial to other researchers
Critical weaknesses
  • Soundness justification. My main concern is that the paper's central design claim is not tested. The paper argues that aligning aggregated prefill attention is preferable to selecting and aligning decoded tokens because it avoids token-selection overhead and is unified across tasks. This is a claim relative to existing attention-distillation objectives developed in the offline setting (e.g., Align-KD, CompoDistill, TRAG), which can be ported into the OPD loss with little effort; the only baseline reported, however, is vanilla OPD. Showing that the proposed prefill-level, single-KL alignment outperforms these alternatives under on-policy training is necessary to support the stated contribution, since that is precisely where the design choices (query tokens, layer range, loss form) differ.

  • §3.3 does not yet support its conclusion, but could with additional analysis. The bound rests on a shared-value assumption that removes exactly the term the loss does not control (value mismatch between models of different width) and on image-renormalized attention that drops the image-mass term, so it currently shows only that aligning attention aligns a hypothetical context vector. The claim that "aligning attention aligns the visual evidence the student uses" would be well supported if the authors (i) measured the actual teacher–student visual-context gap before/after VA, decomposed into attention-shape, value-mismatch, and image-mass terms; and (ii) added a control that aligns to the initial student's own attention map (or a random teacher map), to show the gain comes from the teacher's grounding rather than from drift regularization.

Questions for authors

[questions]

  • Log-PPL analysis (Fig. 3). Since the teacher's log-PPL on human captions appears higher than the base student's, how should the reader interpret vanilla OPD raising log-PPL; as drift away from the human caption distribution, or as the expected effect of becoming more teacher-like?
  • Attention-similarity analysis (Fig. 2, Table 2). The mean cosine-similarity gains toward the teacher are small (+0.01 on two of three benchmarks) and the figure appears to use ~20 samples per benchmark with several samples moving in the opposite direction, so it is hard to judge whether the shift is meaningful. It would also help to see the base student (before training) as a third condition in Fig. 2 and its similarity to the teacher
  • Related work on attention-based distillation should be cited and discussed. Although these methods are developed in the offline (teacher-forced) setting, attention distillation for MLLMs is an established line of work — e.g., Align-KD (CVPR 2025), CompoDistill (ICLR 2026), TRAG (2026), and LLaVA-KD. The observation that students attend to irrelevant image regions also connects to existing analyses which would strengthen the motivation if acknowledged.
  • Given that vanilla OPD falls below the base student on ChartQA and both captioning benchmarks for both model families, how does offline distillation (token-level KD on ground-truth or teacher-generated responses) compare on these benchmarks? This would help readers judge whether on-policy distillation is the right starting point for VLMs.
  • Further ablation. Two further ablations could further clarify what the gain comes from: training with the alignment term alone (λ_OPD = 0), and aligning to the initial student's own attention map instead of the teacher's, to separate teacher grounding from drift regularization.

[minor]

  • Table 3/4 headers and caption say "OCRBench Acc." / "InfoVQA Acc." / "accuracy/EM," but Table 1 and §4.2 report both as EM.
Reviewer_SWCvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Visual-Align OPD, an on-policy distillation method for vision-language models that augments token-level distribution matching with teacher–student visual-attention alignment. The main motivation is that vanilla OPD supervises what the student predicts but only indirectly corrects errors in where the student obtains visual evidence. The proposed method therefore extracts attention distributions over visual tokens from selected decoder layers and minimizes an additional KL divergence between the teacher and student attention maps. Experiments on InternVL3 and Qwen3-VL teacher–student pairs across seven multimodal benchmarks show that the method generally improves over vanilla OPD, with particularly large gains on ChartQA and InfoVQA.

Critical strengths
  • The paper identifies a relevant limitation of applying text-oriented OPD directly to vision-language models: token-level supervision may be insufficient when the student follows an incorrect visual-evidence path. Aligning visual attention is a simple and intuitive extension.
  • The method is evaluated on two model families and a diverse set of visually intensive tasks, including chart reasoning, OCR, document understanding, text-based VQA, and image captioning. The weight and layer-number ablations suggest that the proposed objective is beneficial under several configurations and introduces no inference-time overhead.
Critical weaknesses
  1. The definition and implementation of “prefill visual alignment” are insufficiently specified.

    • Alignment positions: The method is repeatedly described as operating during visual prefill, but the formulation computes attention conditioned on (x,y<t)(x, y_{<t}), and the analysis evaluates generated-token attention maps. The paper does not clearly state whether the query set QQ contains prefill tokens, generated tokens, or both. It is therefore difficult to determine whether the proposed objective is genuinely prefill-only or instead aligns visual attention throughout autoregressive decoding.
    • Implementation details: The construction of QQ, head selection, teacher–student layer mapping, and the exact timing and tensor shapes of the alignment loss are not reported in sufficient detail. These choices affect both the meaning of the method and its computational cost. A more explicit algorithm and implementation description are needed for reproducibility.
  2. The reported hyperparameters and computational setup contain important inconsistencies.

    • Alignment weight: The InternVL3 results in Table 1 (71.8 on OCRBench, 51.5 on InfoVQA, 48.20 on Flickr30k, and 23.35 on VizWiz-Caps) exactly match the λv=0.1\lambda_v=0.1 row in Table 3. However, Appendix Table 8 states that the visual-alignment weight is αv=1.0\alpha_v=1.0, whose results in Table 3 are different. The authors should clarify which weight was used for the main results and how it was selected.
    • GPU configuration: Section 4.1 states that training uses four NVIDIA A100 GPUs, whereas Appendix Table 8 reports three training GPUs and derives an effective batch size of six from this setting. These discrepancies should be resolved, as they currently make the main experiments difficult to reproduce and leave the hyperparameter-selection protocol unclear.
  3. The current comparisons do not isolate the benefit of prefill visual alignment from generic intermediate-representation regularization.

    • The main comparisons include only the base student and vanilla OPD. Additional compute-matched baselines, such as standard attention transfer, hidden-state alignment, offline distillation with visual alignment, or a non-teacher visual-attention regularizer, would help establish whether the gains are specific to the proposed teacher-guided visual alignment.
    • The attention-similarity improvements relative to the teacher are modest (0.01 on Flickr30k and TextVQA and 0.05 on ChartQA), and attention similarity alone does not demonstrate that the aligned regions contain answer-relevant evidence. Grounding-based or intervention-based evaluation, together with multi-seed results and training-time memory/throughput measurements, would provide stronger support for the proposed mechanism.
Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.