acceptodds
Withdrawn from ICLR 2027

PA-OPSD: Privileged-Anchor On-Policy Self-Distillation for Mitigating Hallucinations in Multimodal Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy self-distillation (OPSD) has recently emerged as an effective approach for improving multimodal large language models (MLLMs) initialized from pretrained models. However, existing methods primarily enhance teacher predictions with privileged information or create teacher–student input discrepancies through image degradation, neither of which ensures that the student inherits the teacher’s ability to identify and exploit critical visual evidence. To address this limitation, we propose PA-OPSD (Privileged-Anchor On-Policy Self-Distillation), an on-policy self-distillation framework guided by privileged visual anchors. Specifically, the teacher efficiently aggregates image features conditioned on the question prompt, ground-truth answer, and an initial response prefix to extract privileged visual anchors relevant to the current generation process. These anchors then support two complementary supervision pathways. First, in the contrastive decoding pathway, the teacher compares token distributions obtained from the original image and a perturbed image in which the privileged visual anchors are masked. The resulting distributional divergence reveals hallucination-prone token positions whose predictions are sensitive to critical visual evidence, enabling more precise supervision of the student’s on-policy rollouts. Second, in the visual attention distillation pathway, the teacher transfers its attention over critical visual regions, informed by privileged information, to the student, guiding the student to ground its generation in the appropriate visual evidence. Extensive experiments demonstrate that PA-OPSD consistently improves multimodal reasoning while mitigating visual hallucinations, validating the effectiveness of privileged visual anchors and the proposed dual-path supervision mechanism. Code is available in supplementary material.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multi modal, on-policy distillation, attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_chdeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PA-OPSD, an on-policy self-distillation framework for multimodal large language models. An EMA teacher extracts visual anchors using the question, training-only reference answer, and the student's early response. These anchors guide visual masking to identify sensitive generation positions, where the student receives both contrastive output supervision and spatial attention supervision. Experiments on three backbones and eight evaluation settings show improvements over existing self-distillation baselines without changing the inference pipeline.

Critical strengths

1.Reference answers guide visual evidence localization but are excluded from subsequent teacher prediction passes. This lets the model benefit from training-time answer information while keeping the student's inputs and architecture unchanged.

2.The results show a consistent benefit on high-resolution perception: PA-OPSD obtains the best HR8K score on every evaluated backbone. This is particularly relevant to the method's focus on local visual evidence.

3.The localization analysis goes beyond attention heatmaps. Comparing equal-area masking of high-attention, low-attention, and random regions provides a useful check that the selected regions matter to model predictions.

Critical weaknesses

1.The distinction from related work needs further clarification. Although VA-OPD is discussed, VGS [1] has also studied token-level visual dependence and adaptive supervision, while Reinforced Attention Learning [2] includes on-policy attention distillation alongside output distillation.

2.The paper shows that masking the selected regions affects predictions, but does not directly show that PA-OPSD students use visual evidence more faithfully than OPSD or VCSD students.

3.The ablations do not directly establish the benefit of the shared-anchor design. Removing answer conditioning, replacing regions with random ones, or removing the attention loss shows that individual components are useful, but does not isolate the value of sharing anchors and selected positions across the two paths.

4.Eq. (5) aligns attention distributions normalized over visual tokens, so the attention loss can be small even when total visual attention is low. Spatial alignment alone does not establish stronger visual reliance. Additional evidence is needed to support the grounding interpretation of this objective.

[1] Hee Suk Yoon et al. Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding. ICML, 2026.

[2] Bangzheng Li et al. Reinforced Attention Learning. arXiv:2602.04884, 2026.

Questions for authors

Listed in the "Weaknesses" section.

Reviewer_gNSNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PA-OPSD, an on-policy self-distillation method for reducing hallucinations and improving visual grounding in multimodal large language models. An EMA teacher uses the question, training-time ground-truth answer, and the student's early response to extract privileged visual anchors. These anchors guide local image masking and help identify visually sensitive generation positions through differences in teacher predictions. At the selected positions, the student is trained using contrastively enhanced output targets and spatial attention alignment. Experiments on three model configurations and eight evaluation settings show improvements in average performance. The paper also includes component ablations, masking experiments, and parameter analyses.

Critical strengths
  1. The paper presents a coherent design for improving visual grounding in on-policy self-distillation. The proposed framework uses shared privileged visual anchors to guide both generation-position selection and spatial attention distillation. This provides a reasonable way to focus training on visually relevant decisions rather than treating all generation positions equally. The use of ground-truth answers is restricted to training-time anchor extraction, without introducing additional components at inference time.

  2. The experimental evaluation provides encouraging results and useful ablation evidence. PA-OPSD achieves the highest average performance across three evaluated model configurations. The paper also includes ablations on the conditioning branches, visual regions, attention loss, and masking strategies. In particular, Table 2 supports the contributions of anchor localization and attention supervision, while Table 3 provides additional evidence that the identified regions are important for model predictions.

Critical weaknesses
  1. The independent benefit of critical generation-position selection is not fully established. A central motivation of PA-OPSD is to focus distillation on visually sensitive generation positions. However, the existing experiments do not directly compare the proposed JSD-based position selection with the same number of randomly selected positions under otherwise identical training conditions. The random-region ablation in Table 2 and the masking experiments in Table 3 provide useful evidence for visual localization, but they do not isolate the contribution of generation-position selection. As a result, it remains unclear how much of the improvement comes from this central mechanism rather than the contrastive teacher targets and additional attention supervision.

  2. The evidence for improved visual reliance is not yet conclusive. The attention-loss ablation and the region-masking experiments provide useful evidence that attention supervision is beneficial and that the selected regions affect predictions. However, matching spatial attention distributions does not by itself establish that the trained student relies more faithfully on task-relevant visual evidence. The existing analyses do not clearly separate improved visual reliance from improved attention alignment. A controlled comparison between the full model and the variant without attention loss under task-relevant and task-irrelevant visual changes would help substantiate this stronger interpretation.

Questions for authors
  1. Can the authors compare the proposed JSD-based generation-position selection with count-matched random positions or uniform distillation under otherwise identical training conditions? In particular, keeping the visual anchors, teacher targets, and loss normalization fixed would help clarify the independent contribution of critical position selection.

  2. Does spatial attention distillation improve the trained student's reliance on task-relevant visual evidence? For example, can the authors compare the full model with the variant without attention loss under task-relevant and task-irrelevant visual changes? This would help distinguish improved visual grounding from improved attention alignment alone.

  3. Can the authors clarify whether OPSD, VCSD, and RP-OPSD were trained and evaluated using matched data, training budgets, and evaluation settings? Are repeated-run results available to assess the reliability of the reported improvements?

Reviewer_7LzUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PA-OPSD combines selective output distillation with spatial attention supervision on student-generated multimodal responses. An EMA teacher extracts quality-weighted visual anchors using question, training-only reference-answer and early-response attention. Frozen anchors guide original-image and region-masked teacher predictions. Their JSD selects visually sensitive positions, while a contrastively enhanced full-vocabulary distribution supplies output targets; spatial attention JSD supervises the same positions. Subsequent teacher prediction contexts exclude answer text and privileged cached states, and inference uses the ordinary student. Experiments train three Qwen backbones on ViRL39K and report eight evaluation settings. The reported average exceeds the strongest competing distillation average by 0.69–1.13 points across the three backbones.

Critical strengths
  1. A coherent use of training-only privileged information. The same frozen anchors connect teacher guidance, visual contrast and spatial supervision. Appendix B.1 requires rebuilding prediction contexts without answer text or privileged cached states. This provides a clearly specified information boundary and preserves ordinary student inference.
  2. Positive averages with relevant component evidence. Table 1 reports averages of 66.58, 71.88 and 74.55, the highest among the compared methods for each backbone. In Table 2, random regions lower the 9B average by approximately 0.88 points, and removing attention supervision lowers it by approximately 0.60. These controls support the contribution of localization and spatial supervision, while individual benchmark rankings appropriately qualify the aggregate result.
Critical weaknesses
  1. The central recipe remains incompletely specified. Appendices A.4/B.2 describe sample-quality score η\eta and teacher corrections χlht\chi_{lht}, δlht\delta_{lht} verbally, without specifying their computations and activation thresholds. Defaults for EMA decay, loss weights and response-position selection are also missing. These choices determine supervision eligibility and targets, so they materially affect reproducibility.
  2. The evidence does not isolate selection from target shaping. Figure 3 varies masking and α, but reports neither α=0 nor a fixed-anchor comparison of selected, all and equal-count random response positions. Under identical teacher views, Top-0% gives zero JSD and can suppress both losses; it is not a clean position-selection control. Table 3 supports prompt-conditioned localization without training, rather than demonstrating a change in the trained student's reliance. The modest average margins also lack repeated-training uncertainty and a stated independent tuning protocol.
Questions for authors
  1. Clarification: Please specify η\eta, χlht\chi_{lht}, δlht\delta_{lht}, the core hyperparameters, layer/head selection and masking replacement content. What fractions of examples and response positions remain eligible after gating? If available, the referenced configuration/code could resolve much of this directly.
  2. Additional evidence: With the same anchors and teacher guidance, what are the separate effects of α=0\alpha=0 versus α=2\alpha=2 and selected versus all/equal-count random positions? Please report supervised-token counts and training budgets.
  3. Clarification and evidence: How were masking fraction, α\alpha and layer count selected? Please specify evaluation splits, decoding and judges, and report variability across independent training runs. Paired intervals on fixed predictions would address a separate source of evaluation uncertainty.
  4. Additional evidence: Does a fixed-set, paired post-training region intervention distinguish the full method from output-only distillation? This would test the stronger grounding interpretation. Measured training time/memory would additionally clarify the cost of the extra teacher passes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.