acceptodds
Under review as a conference paper at ICLR 2027

ASYM-OPSD: Mitigating MLLM Hallucinations via Asymmetric On-Policy Self-Distillation

Abstract

Hallucination in Multimodal Large Language Models (MLLMs) remains a critical barrier to trustworthy deployment. Existing mitigation strategies, whether inference-time interventions such as contrastive decoding and attention reallocation, or training-based preference alignment, share a fundamental limitation: they rely on coarse-grained, sequence-level signals that cannot precisely localize the specific tokens responsible for hallucinations. We introduce ASYM-OPSD, an asymmetric on-policy self-distillation framework that derives dense, token-level reward signals from hierarchical textual descriptions of the input image that serve as privileged visual anchors. When a teacher model conditioned on anchored input evaluates the student's own rollouts, the per-token log-probability gap between teacher and student reveals which tokens are visually grounded and which are hallucinated. Our pipeline comprises three stages: visual anchor construction, anchor-guided warm-up that repairs hallucinated responses to initialize a teacher model, and reinforcement learning with group-relative token-level advantages and curriculum scheduling on anchor granularity. Experiments on Qwen3-VL-4B and InternVL3.5-8B show that ASYM-OPSD surpasses training-based and inference-time baselines across four hallucination benchmarks, mitigating long-form hallucinations by up to without compromising general VQA performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Hallucination Mitigation, On-Policy Self-Distillation, Large Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_vAHCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ASYM-OPSD, an asymmetric on-policy distillation framework for mitigating hallucinations in multimodal large language models. External models construct and verify multi-granularity visual anchors and repair model responses to train a teacher. The student generates responses from images and questions alone, while the frozen, anchor-conditioned teacher scores these responses token by token. The teacher–student log-probability gap provides dense rewards, combined with group-relative advantages and a curriculum from detailed to brief anchors. Experiments on Qwen3-VL-4B and InternVL3.5-8B report improvements across four hallucination benchmarks while largely preserving general VQA performance, without additional inference-time components.

Critical strengths

The paper addresses an important problem with fine-grained feedback targeting localized hallucinations.

Scoring student-generated responses through teacher forcing avoids aligning independently generated teacher and student responses.

Results improve across two backbone families, with no additional teacher or anchors required at inference.

Critical weaknesses
  1. Equation (1) measures relative teacher preference and is affected by teacher fine-tuning, confidence, wording, style, and the evolving student distribution. Correct tokens can receive negative rewards, while hallucinated tokens favored by the teacher or erroneous anchors can receive positive rewards. Group centering further disconnects positive advantages from factual correctness.

  2. Loss aggregation is ambiguous. Equation (3) averages within each response, whereas Appendix A.2 (page 19) specifies token-mean loss aggregation. Averaging over all valid batch tokens and averaging responses equally assign different weights to variable-length responses. The authors should clarify the implemented objective and provide matching pseudocode.

  3. Does OPSD use the repaired teacher? How are the GRPO judge’s paired True/False verdicts mapped to rewards?

  4. In Equations (1)–(3), are student log-probabilities computed using the rollout policy or the policy being updated? Are rewards and advantages detached from the computation graph?

  5. The self-distillation terminology needs qualification. Although teacher and student share a base model, external models generate anchors and repair responses used to fine-tune the teacher. The method therefore incorporates substantial external supervision; the authors should clarify its distinction from externally assisted distillation.

Questions for authors

Please address the questions raised in the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer vAHC (1/2)

We thank the reviewer for the careful reading and the questions about the reward, optimization, and supervision sources.

Summary. (i) Human-labeled reward analysis, anchor checks, and entropy/length measurements address grounding and response collapse (W1). (ii) We clarify microbatch token-mean reduction and will align Eq. (3) with the implementation (W2). (iii) Both teachers use repaired targets; their training inputs differ by the anchor. The GRPO reward and PPO gradient paths are specified below (W3–W4). (iv) We will revise the terminology to identify the privileged context and external supervision (W5).


W1. Grounding, anchor quality, and response collapse

  1. Teacher–student signal. We scored identical tokens and prefixes in 23 human-labeled HaloQuest cases at RL initialization. TT is the frozen SFT teacher, SS the original backbone, and A/0A/0 denotes with/without detailed anchors. Teacher comparisons keep weights and the ANALYST prompt fixed; LS,0L_{S,0} uses the student's dataset prompt.
BackboneSignalLog-probability differenceAUROCGrounded − hallucinated meanG>H responses
Qwen3-VL-4BFull SFT-teacher rewardLT,A−LS,0L_{T,A}-L_{S,0}0.646+1.11020/23
Qwen3-VL-4BAnchor input componentLT,A−LT,0L_{T,A}-L_{T,0}0.648+0.97120/23
Qwen3-VL-4BTeacher–student difference, no anchorLT,0−LS,0L_{T,0}-L_{S,0}0.560+0.13915/23
Qwen3-VL-4BOriginal backbone with anchorLS(A)−LS,0L_S(A)-L_{S,0}0.563+0.35018/23
InternVL3.5-8BFull SFT-teacher rewardLT,A−LS,0L_{T,A}-L_{S,0}0.624+0.71223/23
InternVL3.5-8BAnchor input componentLT,A−LT,0L_{T,A}-L_{T,0}0.641+0.66021/23
InternVL3.5-8BTeacher–student difference, no anchorLT,0−LS,0L_{T,0}-L_{S,0}0.517+0.05216/23
InternVL3.5-8BOriginal backbone with anchorLS(A)−LS,0L_S(A)-L_{S,0}0.594+0.34216/23

AUROC ranks hallucinated tokens using the negative signal; G>H counts responses with higher grounded-token means. The anchor input accounts for most of the grounded–hallucinated mean separation on both backbones.

  1. Anchor quality. Our anchor-cleaning pipeline uses image-based checks by Gemini and GPT, followed by repair or removal of rejected anchors (Appendix A.1, Table A3). Residual errors and omissions can make some token rewards noisy. The human-labeled results above and the held-out hallucination reductions support the value of this feedback for learning image-grounded answers.

  2. Entropy and answer length. In the reported Qwen run, actor entropy changes from 0.573 to 0.511. The reported character lengths before and after ASYM-OPSD are:

BenchmarkQwen3-VL-4B: base → ASYM-OPSDInternVL3.5-8B: base → ASYM-OPSD
HallusionBench195 → 22761 → 97
HaloQuest86 → 12426 → 34
MMHal-Bench146 → 14959 → 58

These measurements show no sign of severe collapse in entropy or answer length. On Qwen MMHal, hallucination falls from 15.33% to 5.21% while answer length stays close to its original level (146 → 149 characters).


W2. Loss aggregation and Eq. (3)

The implementation uses token-mean reduction within each microbatch:

LB=∑i∈B,tmi,tℓi,t∑i∈B,tmi,tL_B=\frac{\sum_{i\in B,t}m_{i,t}\ell_{i,t}}{\sum_{i\in B,t}m_{i,t}}

where mm masks valid response tokens and ℓ\ell is the clipped PPO loss. Each fixed-size microbatch loss is divided by the gradient-accumulation factor. Longer responses therefore contribute more token terms within a microbatch. We will revise Eq. (3) and the pseudocode to state this reduction and the gradient-accumulation and distributed-averaging steps explicitly.


W3. Teacher preparation and the GRPO judge

Both teachers use repaired response targets. The OPSD teacher is trained with the original question; the ASYM-OPSD teacher is trained with the anchor and original question. The ASYM-OPSD teacher is frozen during RL.

GRPO calls gpt-5.6-luna from verl's reward manager. The judge receives the question, reference answer, and prediction, and returns a correctness score from 0.0 to 1.0 in 0.1 increments. The score credits correct content in responses that also contain hallucinations. The reward manager normalizes scores within each prompt group and stores the normalized score at the last valid token. The GRPO trainer then computes a response-level advantage and applies the same advantage to all valid tokens in the response.

AuthorsOpenReview ↗

Response to Reviewer vAHC (2/2)

W4. Rollout-policy scores and actor-update gradients

The reward uses cached rollout-policy log-probabilities. Teacher scores, rewards, and advantages remain fixed during actor updates; advantages are computed under torch.no_grad(). Current-policy log-probabilities retain gradients in the PPO ratio.


W5. Distillation terminology and external supervision

We will revise the terminology to “privileged-context on-policy distillation” and clarify the external supervision used to construct anchors and repair responses.


Revision Plan. We will add the reward validation to Section 4.4 and correct Eq. (3), the teacher definitions in Table 3, and the GRPO details in Appendix A.2. We will also update the distillation terminology.

Thank you for the rebuttal. However, I still have several concerns:

  1. W1. Reward validity. The additional analysis does not fully address my concern that relative teacher preference is not equivalent to factual correctness, as it can be affected by teacher's wording and style. I acknowledge that these factors are difficult to quantify, but I encourage the authors to explicitly discuss this limitation in the paper.

  2. W2. Eq. (3). The formulation still needs revision. Calling it the "standard clipped GRPO objective" (line 244) is imprecise, and the group number is not explicitly reflected.

  3. GRPO judge. Please include the exact scoring rubric and judge prompt for reproducibility.

I also suggest highlighting manuscript revisions in blue to make the changes easier to identify.

I encourage the authors to carefully address these remaining issues. I will consider the subsequent discussion and comments from other reviewers before making my final decision.

Thank you for your helpful feedback and additional questions.

  1. We have added the original reward script used by our GRPO baseline to the supplementary material at ./training/verl/workers/reward_manager/graphallu.py, together with the script at ./training/scripts/run_grpo_graphallu.sh.
  2. We have resubmitted the manuscript, with newly added content highlighted in blue. In response to your follow-up:
    • Reward interpretation: We explicitly clarify that the reward captures relative teacher preference rather than directly judging factual correctness, and may also reflect wording and style.
    • Objective notation: We describe the objective as “a PPO-style clipped surrogate with dense group-relative token advantages” and explicitly specify prompt-group, rollout, and token indices, while preserving the implemented microbatch token-mean reduction.
    • GRPO judge: The exact judge prompt, including the scoring instructions and few-shot examples, is now provided in Appendix A.6.

We hope these revisions clarify our method and its implementation. Thank you again for your constructive feedback.

Reviewer_G58GOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper proposes ASYM-OPSD, a training framework for reducing hallucinations in multimodal large language models. The method first constructs hierarchical textual descriptions of training images and uses anchor-guided response repair to initialize a teacher model. During reinforcement learning, the student generates responses without these descriptions, while the frozen teacher evaluates the same responses with access to the visual anchors. The teacher–student log-probability differences provide token-level rewards, combined with group-relative advantages and a curriculum over anchor granularity. Experiments on Qwen3-VL-4B and InternVL3.5-8B compare ASYM-OPSD with training-based and inference-time baselines across hallucination and general VQA benchmarks.

Critical strengths

(i) The paper studies an important problem in multimodal hallucination mitigation: how to provide more precise training feedback when hallucinations occur in only part of a generated response. Exploring token-level supervision is a meaningful direction for improving visual grounding without treating all tokens in a response equally.

(ii) The method achieves improvements on several hallucination metrics across two vision-language model families, while largely preserving general VQA performance. The evaluation covers four hallucination benchmarks and six general VQA benchmarks, with additional ablations examining the effects of teacher repair and anchor-granularity scheduling.

Critical weaknesses

(i) The paper does not sufficiently establish that the teacher–student log-probability gap in Eq. (1) reliably measures token-level visual groundedness. The teacher is fine-tuned and receives visual anchors, while the student has different parameters and no anchors. The probability gap may therefore reflect differences in model behavior or language preferences rather than whether a token is supported by the image. Although Section 4.4 evaluates token-level hallucination detection, it uses the same raw backbone under different input conditions on only 20 annotated responses, rather than directly validating the teacher–student reward used during training.

(ii) The visual anchors are constructed using ground-truth information, making it unclear whether the performance gains come from improved visual grounding or additional answer supervision. Appendix A.1 states that anchor generation uses gold question–answer pairs and, for some datasets, reference data or reasoning chains. These anchors may already contain information about the correct answers, which is then passed to the teacher during training. The paper does not compare against anchors generated without ground-truth information, so the contribution of the proposed visual-grounding mechanism cannot be clearly separated from that of the additional supervision.

Questions for authors

(i) Please provide more explanation about whether the teacher–student log-probability gap in Eq. (1) reliably measures token-level visual groundedness, and how this is validated for the reward actually used during training.

(ii) Please clarify whether the performance improvements come from the proposed visual-grounding mechanism or the additional ground-truth information used to construct the visual anchors.

After Rebuttal

3 comments
AuthorsOpenReview ↗

Response to Reviewer G58G

We thank the reviewer for the detailed questions about the training reward and reference-assisted supervision.

Summary. (i) The new experiment directly evaluates the frozen SFT teacher–student reward at RL initialization and separates the anchor contribution (W1/Q1). (ii) The no-gold audit and length-stratified overlap analysis clarify how references guide task-relevant visual descriptions (W2).


W1/Q1. Evaluating the SFT-teacher reward used for RL

We directly evaluated the frozen SFT teacher–student reward at RL initialization on 23 cases selected from 608 HaloQuest items where both backbones' answers were judged hallucinated. Grounded and hallucinated spans follow the labeling protocol in Fig. 5. Each condition scores the same response tokens and prefixes. To isolate the anchor contribution, we hold the SFT teacher weights and ANALYST system prompt fixed and remove only the anchor block.

Below, TT is the frozen SFT teacher and SS the original backbone used to initialize RL. AA denotes detailed anchors; teacher condition 00 removes the anchor block. LS,0L_{S,0} uses the student's dataset prompt, and LS(A)L_S(A) uses the original backbone with the anchored ANALYST context.

BackboneSignalLog-probability differenceAUROCGrounded − hallucinated meanG>H responses
Qwen3-VL-4BFull SFT-teacher rewardLT,A−LS,0L_{T,A}-L_{S,0}0.646+1.11020/23
Qwen3-VL-4BAnchor input componentLT,A−LT,0L_{T,A}-L_{T,0}0.648+0.97120/23
Qwen3-VL-4BTeacher–student difference, no anchorLT,0−LS,0L_{T,0}-L_{S,0}0.560+0.13915/23
Qwen3-VL-4BOriginal backbone with anchorLS(A)−LS,0L_S(A)-L_{S,0}0.563+0.35018/23
InternVL3.5-8BFull SFT-teacher rewardLT,A−LS,0L_{T,A}-L_{S,0}0.624+0.71223/23
InternVL3.5-8BAnchor input componentLT,A−LT,0L_{T,A}-L_{T,0}0.641+0.66021/23
InternVL3.5-8BTeacher–student difference, no anchorLT,0−LS,0L_{T,0}-L_{S,0}0.517+0.05216/23
InternVL3.5-8BOriginal backbone with anchorLS(A)−LS,0L_S(A)-L_{S,0}0.594+0.34216/23

AUROC uses the negative signal to rank hallucinated tokens; G>H counts responses with a higher grounded-token mean. The anchor component accounts for most of the grounded–hallucinated mean separation on both backbones. The full reward assigns higher means to grounded tokens in 20/23 Qwen responses and all 23 InternVL responses, and its AUROC exceeds the original-backbone reward on both models.


W2. Reference supervision and the no-gold comparison

In our initial design, the anchor generator received only the image and question. After inspecting the generated anchors, we found that many contained unnecessary details unrelated to the question, and some contained hallucinations. We then added the gold answer to help the generator focus on the visual facts needed to answer the question, and used Gemini and GPT to check and repair the anchors against the image.

We also generated 1,500 anchors from the image and question alone, of which 1,495 were nonempty. A text audit by Qwen3.8-Max found that 17.0% missed facts needed to answer the question and 15.4% contradicted the reference answer.

We measured anchor–answer overlap on 288 open-ended cases, separating matches by matched answer length:

Gold-informed anchorLength, charactersAnswer-text match (any length)Answer-text match (≥2 tokens)Answer-text match (≥3 tokens)Token F1: full referenceToken F1: best answer match
Brief23843.8%21.2%13.5%0.2230.206
Medium63546.9%17.4%8.0%0.1230.092
Detailed195348.3%28.1%19.4%0.0700.051

Character lengths and overlap statistics above use the same 288-case subset. The corresponding no-gold results are:

No-gold anchorAnswer-text match (any length)Answer-text match (≥2 tokens)Answer-text match (≥3 tokens)
Brief36.9%13.9%8.0%
Medium33.7%12.5%6.6%
Detailed34.0%10.4%3.5%

No-gold brief anchors cover 287 nonempty cases; medium and detailed anchors cover 288 each. Single-token matches are expected for answers about colors or counts, such as “blue” or “two,” which also appear naturally in visual descriptions.


Q2. Clarification of RAPIC

We thank the reviewer for this question and would appreciate clarification of what “RAPIC” refers to, so we can address the intended point.


Revision Plan. We will add the SFT-teacher reward analysis to Section 4.4 and the no-gold and answer-overlap audits to Appendix A.1.

I appreciate the authors' detailed response and the additional analyses, which have sufficiently addressed my main concerns. I also apologize for the copy-and-paste error in Question (ii) of my original review. My intended question was consistent with Weakness (ii), concerning whether the performance improvements come from improved visual grounding or the additional ground-truth supervision used to construct the visual anchors. The authors have addressed this concern in their response. I will raise my rating and would be glad to see the paper accepted. It's a good paper.

Thanks for reading our response! We appreciate the time you’ve spent on our paper and the helpful feedback you shared.

Reviewer_57K5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles MLLM hallucinations by giving a teacher additional image descriptions and training a student using their token-level log-probability differences. Experiments on Qwen3-VL-4B and InternVL3.5-8B report lower hallucination rates across four benchmarks, with little impact on general VQA performance.But the verification section needs to consider.

Critical strengths

I am pleased to see work utilizing textual hints for VLM self-distillation. This approach may risk reinforcing linguistic priors rather than visual grounding, but the hallucination and general VQA results reported in this paper are encouraging, indicating the method's potential. However, whether it truly avoids linguistic prior reinforcement still requires more direct verification.

Critical weaknesses

It is unclear whether the token-level log-probability gap truly reflects visual grounding, especially since the teacher uses different weights and receives anchors constructed with reference answers.

Questions for authors

1.Since the visual anchors are generated using reference answers, could the authors report results using image-only anchors to clarify where the improvements come from? 2.Does the teacher receive the student's response as part of its prompt before scoring it? If so, how is potential answer-copying bias addressed?And how do you confirm this. 3.Equation (2) uses group-relative normalization. If most sampled responses contain hallucinations, could some incorrect tokens still receive positive advantages? How is this issue handled?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 57K5 (1/2)

Response to Reviewer 57K5

We thank the reviewer for the questions about visual grounding, reference information, and token-level feedback.

Summary. (i) Our reward distinguishes grounded from hallucinated tokens on both backbones, and visual anchors make the main contribution (W1). (ii) Reference answers guide anchors toward facts needed for the question; a text audit finds these facts missing in 17.0% of no-gold anchors (Q1). (iii) The full student response is excluded from the teacher's user prompt. Style-preserving warm-up and a detailed-to-brief curriculum target answer shortcuts (Q2). (iv) A hallucinated token can receive a positive advantage under group normalization. In the rollout audit, about 80% of sign flips involve mildly negative rewards in [−1, 0), while lower rewards retain lower advantage weights within each group (Q3).


W1. Does the reward reflect visual grounding?

The fixed-teacher comparison isolates the anchor's contribution: we keep the SFT teacher weights and ANALYST system prompt unchanged and remove only the anchor block. We evaluate the same 23 HaloQuest cases where both backbones' answers were judged hallucinated, with grounded and hallucinated spans labeled following Fig. 5.

Below, TT is the frozen SFT teacher and SS the original backbone used to initialize RL. AA denotes detailed anchors; teacher condition 00 removes the anchor block. LS,0L_{S,0} uses the student's dataset prompt, and LS(A)L_S(A) uses the original backbone with the anchored ANALYST context.

BackboneSignalLog-probability differenceAUROCGrounded − hallucinated meanG>H responses
Qwen3-VL-4BFull SFT-teacher rewardLT,A−LS,0L_{T,A}-L_{S,0}0.646+1.11020/23
Qwen3-VL-4BAnchor input componentLT,A−LT,0L_{T,A}-L_{T,0}0.648+0.97120/23
Qwen3-VL-4BTeacher–student difference, no anchorLT,0−LS,0L_{T,0}-L_{S,0}0.560+0.13915/23
Qwen3-VL-4BOriginal backbone with anchorLS(A)−LS,0L_S(A)-L_{S,0}0.563+0.35018/23
InternVL3.5-8BFull SFT-teacher rewardLT,A−LS,0L_{T,A}-L_{S,0}0.624+0.71223/23
InternVL3.5-8BAnchor input componentLT,A−LT,0L_{T,A}-L_{T,0}0.641+0.66021/23
InternVL3.5-8BTeacher–student difference, no anchorLT,0−LS,0L_{T,0}-L_{S,0}0.517+0.05216/23
InternVL3.5-8BOriginal backbone with anchorLS(A)−LS,0L_S(A)-L_{S,0}0.594+0.34216/23

AUROC uses the negative signal to rank hallucinated tokens; G>H counts responses with a higher grounded-token mean. The anchor component accounts for most of the grounded–hallucinated mean separation on both backbones. The full reward assigns higher means to grounded tokens in 20/23 Qwen responses and all 23 InternVL responses, and its AUROC exceeds the original-backbone reward on both models.


Q1. Anchors generated without reference answers

In our initial design, the anchor generator received only the image and question. After inspecting the generated anchors, we found that many contained unnecessary details unrelated to the question, and some contained hallucinations. We then added the gold answer to help the generator focus on the visual facts needed to answer the question, and used Gemini and GPT to check and repair the anchors against the image.

We generated 1,500 anchors from the image and question alone, of which 1,495 were nonempty. Qwen3.8-Max judged that 17.0% missed facts needed to answer the question and 15.4% contradicted the reference answer.

AuthorsOpenReview ↗

Response to Reviewer 57K5 (2/2)

Q2. Teacher inputs and answer copying

(1) The teacher's user prompt contains the image, anchor, and question. It scores the student response as an assistant continuation, using only the preceding response tokens. In our tests, adding the full response to the user prompt inflated log-probabilities and destabilized updates; the longer context also slowed scoring and raised cost. (2) To reduce answer shortcuts, teacher warm-up uses repaired student responses that preserve their original style. The curriculum moves from detailed descriptions through medium descriptions to brief anchors, encouraging early learning from richer visual context to reduce direct answer copying.

In the reported Qwen run, actor entropy changes from 0.573 to 0.511. Qwen's HallusionBench answers also increase from 195 to 227 characters after ASYM-OPSD training. These measurements show no sign of severe response collapse in entropy or answer length.

We measured anchor–answer overlap on 288 open-ended cases, separating matches by matched answer length:

Gold-informed anchorLength, charactersAnswer-text match (any length)Answer-text match (≥2 tokens)Answer-text match (≥3 tokens)Token F1: full referenceToken F1: best answer match
Brief23843.8%21.2%13.5%0.2230.206
Medium63546.9%17.4%8.0%0.1230.092
Detailed195348.3%28.1%19.4%0.0700.051

Character lengths and overlap statistics above use the same 288-case subset. The corresponding no-gold results are:

No-gold anchorAnswer-text match (any length)Answer-text match (≥2 tokens)Answer-text match (≥3 tokens)
Brief36.9%13.9%8.0%
Medium33.7%12.5%6.6%
Detailed34.0%10.4%3.5%

No-gold brief anchors cover 287 nonempty cases; medium and detailed anchors cover 288 each. Single-token matches are expected for answers about colors or counts, such as “blue” or “two,” which also appear naturally in visual descriptions.


Q3. Can hallucinated tokens receive positive advantages?

Yes. A hallucinated token can receive a positive advantage. Group normalization assigns positive advantages to rewards above the group mean, including negative rewards.

In the first Qwen3-VL-4B rollout batch (256 ×\times 32 rollouts), 23% of all valid generated tokens had negative raw rewards and positive advantages after normalization. With the same all-token denominator, 4.6% had raw rewards below −1.0 and positive advantages. Thus, about 80% of the observed sign flips involved raw rewards between −1.0 and 0.

Most sign flips occur among mildly negative rewards. Within each prompt group, more negative rewards still receive lower advantage weights. The human-labeled analysis separately supports the reward’s ability to distinguish grounded and hallucinated spans: grounded spans receive higher full-reward means in 20/23 Qwen responses and all 23 InternVL responses.


Revision Plan. We will add the reward analysis to Section 4.4. Appendix A will include the no-gold, overlap, and sign-change results, together with the teacher input template.

Dear Reviewer 57K5,

Thank you again for your feedback.

As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. We are happy to clarify anything further.

Best regards, The Authors

Reviewer_WukuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ASYM-OPSD to mitigate hallucinations in MLLMs. It addresses the limitation of existing methods in accurately identifying hallucinated tokens. The method uses the token-level log-probability difference between a Teacher and a Student, with the Teacher leveraging Visual Anchors, as a reward signal. The authors report the best performance on 13 out of 14 metrics across four hallucination benchmarks using Qwen3-VL-4B and InternVL3.5-8B. The method also maintains the original inference procedure without requiring additional model calls.

Critical strengths
  1. The method shows consistent performance improvements across various hallucination benchmarks.

  2. It is practical at inference time, as it requires no additional models or computation.

Critical weaknesses
  1. It is unclear whether the Teacher–Student log probability difference truly reflects visual grounding. Although Figures 1 and 5 provide some validation, they do not disentangle the effect of Visual Anchors from the distribution shift caused by Teacher SFT. This raises concerns about the validity of the core reward formulation.

  2. The proposed method largely combines existing techniques. Although Table 3 provides ablation studies, it does not sufficiently isolate the contributions of Visual Anchors, Teacher SFT, and token-level rewards. As a result, it is difficult to determine which components are responsible for the reported improvements.

Section 2.1 contains an incomplete citation (?), and the same paper is listed twice in the References. This suggests insufficient manuscript proofreading and raises concerns about the overall quality of the paper.

Questions for authors
  1. Can the authors disentangle the effect of Visual Anchors from the distribution shift caused by Teacher SFT?
  2. Can the authors provide additional ablation studies to independently evaluate the contributions of Visual Anchors, Teacher SFT, and token-level rewards?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Wuku

We thank the reviewer for the comments on reward grounding and method contributions.

Summary. (i) Human-labeled tests show that our reward distinguishes grounded from hallucinated tokens on both backbones. The results show that visual anchors are the main contributor to this separation (W1/Q1). (ii) The sequence-mean comparison supports token-level feedback for reducing hallucinations in descriptive generation (W2/Q2).


W1/Q1. Separating the anchor effect from teacher fine-tuning

From 608 HaloQuest cases, we selected 23 where both backbones' answers were judged hallucinated. We labeled grounded and hallucinated spans following Fig. 5. Anchored conditions use detailed descriptions. Below, TT is the frozen SFT teacher and SS the original backbone used to initialize RL. Teacher inputs share the ANALYST system prompt; AA includes the anchor block and 00 removes it. LS,0L_{S,0} uses the student's dataset prompt, while LS(A)L_S(A) scores with the original backbone under the anchored ANALYST context.

BackboneSignalLog-probability differenceAUROCGrounded − hallucinated meanG>H responses
Qwen3-VL-4BFull SFT-teacher rewardLT,A−LS,0L_{T,A}-L_{S,0}0.646+1.11020/23
Qwen3-VL-4BAnchor input componentLT,A−LT,0L_{T,A}-L_{T,0}0.648+0.97120/23
Qwen3-VL-4BTeacher–student difference, no anchorLT,0−LS,0L_{T,0}-L_{S,0}0.560+0.13915/23
Qwen3-VL-4BOriginal backbone with anchorLS(A)−LS,0L_S(A)-L_{S,0}0.563+0.35018/23
InternVL3.5-8BFull SFT-teacher rewardLT,A−LS,0L_{T,A}-L_{S,0}0.624+0.71223/23
InternVL3.5-8BAnchor input componentLT,A−LT,0L_{T,A}-L_{T,0}0.641+0.66021/23
InternVL3.5-8BTeacher–student difference, no anchorLT,0−LS,0L_{T,0}-L_{S,0}0.517+0.05216/23
InternVL3.5-8BOriginal backbone with anchorLS(A)−LS,0L_S(A)-L_{S,0}0.594+0.34216/23

AUROC ranks hallucinated tokens using the negative signal; the mean-gap column compares grounded and hallucinated tokens. The anchor component accounts for most of this mean separation on both backbones. The full reward gives higher grounded-token means in 20/23 Qwen responses and all 23 InternVL responses. The original backbone also provides a localization signal. Teacher warm-up supports training: Section 3.2 reports that direct distillation from the base model produces extremely short responses and transfers the anchor's descriptive style. Anchor-guided warm-up repairs factual errors while preserving the original answer style.


W2/Q2. Method contributions and token-level feedback

Our contributions are: 1) compared with image-contrast and entropy indicators, our anchor-conditioned log-probability gap provides a more precise signal for locating hallucinated tokens (Fig. 5), which we use as a dense reward for training. ASYM-OPSD achieves the best results on 13 of 14 metrics across four hallucination benchmarks and two backbones (Table 2). On Qwen3-VL-4B, it reduces the MMHal hallucination rate by 38.1% relative to the strongest baseline; 2) compared with existing OPSD, we introduce teacher warm-up and an anchor curriculum to address response collapse and anchor copying. We warm up the teacher on repaired student responses that preserve the student's answer style, and gradually move from detailed anchors to brief facts during RL.

The sequence-mean control averages feedback over each response's TiT_i valid tokens and broadcasts the averaged advantage:

rˉi=1Ti∑t=1Tiri,t\bar r_i=\frac{1}{T_i}\sum_{t=1}^{T_i}r_{i,t},

Aˉi=1Ti∑t=1TiAi,t\bar A_i=\frac{1}{T_i}\sum_{t=1}^{T_i}A_{i,t}.

BackboneFeedbackHB aAcc ↑HB fAcc ↑HB qAcc ↑HQ acc ↑CHAIR_i ↓CHAIR_s ↓MMHal ↓
Qwen3-VL-4BSequence-mean66.4647.4036.7084.509.5037.279.38
Qwen3-VL-4BToken-level, ASYM-OPSD69.8052.3139.5686.506.9038.805.21
InternVL3.5-8BSequence-mean64.2241.0429.2368.678.9630.0015.62
InternVL3.5-8BToken-level, ASYM-OPSD65.8143.9332.9769.838.3027.6015.62

Response averaging retains a response-level learning signal, as reflected in its HB and HaloQuest gains over the base models. Retaining token-level feedback reduces Qwen MMHal hallucination from 9.38% to 5.21% and InternVL CHAIR caption hallucination from 30.00% to 27.60%.


Citation display and duplicate references

Thank you for pointing out the citation display issue. We will complete the citation and merge the duplicate HallusionBench entries, Guan et al. (2024a, 2024b).


Revision Plan. We will add the reward decomposition and sequence-mean comparison to Sections 4.4–4.5, with settings in Appendix A. We will fix the missing citation and duplicate references.

Dear Reviewer Wuku,

Thank you again for your feedback.

As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. We are happy to clarify anything further.

Best regards, The Authors

Thank you for the additional experiments and clarifications. However, I remain unconvinced that the proposed combination of existing techniques constitutes a sufficiently distinct methodological contribution. The core ideas of on-policy self-distillation (OPSD, Vision-OPD), visual description-based hallucination mitigation (VDGD), and token-level visual feedback (Cat-PO, VPPO) have already been explored in prior work. Therefore, I will maintain my original rating.

Thank you for the follow-up. Regarding your remaining concern about novelty, we would like to clarify three distinctions:

  1. Vs. Vision-OPD: Vision-OPD transfers the perceptual advantage of enlarged image crops to a full-image policy, primarily targeting fine-grained visual understanding rather than explicitly localizing hallucinated content in generated responses. Our focus is token-specific feedback for correcting such content.
  2. Vs. VDGD: VDGD uses visual descriptions to guide inference-time decoding, requiring description generation and decoding intervention at inference. Our approach internalizes their benefits through training, without requiring anchors or teacher calls at inference.
  3. Vs. Cat-PO and VPPO: Their token-level signals emphasize visual relevance or dependency rather than directly identifying which generated tokens express visually unsupported content. We focus on hallucination localization and corrective feedback, supported by human-labeled reward analyses.

We have supplemented the relevant citations and revised Section 2 to clarify these distinctions, with the added discussion highlighted in blue. We hope this clarifies our contribution and how it differs from the work you mentioned.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Responses

We sincerely thank all reviewers for their careful reading and constructive feedback. We have revised the manuscript to address the following key questions about ASYM-OPSD.

  1. What is the core contribution, and does the reward capture visual grounding? ASYM-OPSD turns privileged visual anchors into token-level feedback, with style-preserving teacher warm-up and an anchor curriculum making this signal trainable. On 23 human-labeled cases, the full SFT-teacher reward gives grounded spans higher means in 20/23 Qwen responses and 23/23 InternVL responses. Holding the teacher fixed shows that anchors account for most of this separation.

  2. What does token-level feedback add? Compared with broadcasting a response-averaged advantage, token-level feedback reduces Qwen MMHal hallucination from 9.38% to 5.21% and InternVL CHAIR_s from 30.00% to 27.60%. These comparisons support fine-grained feedback for descriptive generation.

  3. Why are reference answers used in anchor construction? Reference answers focus the generator on visual facts needed to answer the question, rather than broad image descriptions. In the no-gold audit, 17.0% of nonempty anchors missed answer-relevant facts, supporting this task-focused design. The generated anchors are checked against the image, and Table A5 reports answer-text overlap for both gold-informed and no-gold anchors.

  4. How are answer copying and response shortening addressed? Teacher scoring uses preceding response tokens, without placing the full answer in the user prompt. Style-preserving warm-up and the anchor curriculum target answer shortcuts, while the overlap audit quantifies shared answer text. Qwen MMHal hallucination falls from 15.33% to 5.21% while mean answer length stays comparable (146 → 149 characters); actor entropy in the reported Qwen run changes from 0.573 to 0.511.

  5. How should reward-to-advantage sign changes be interpreted? Group normalization can assign positive advantages to negative raw rewards. Unlike a response-level correctness score, the dense teacher–student log-probability gap conveys token-specific, graded teacher preferences, which are converted into policy-gradient weights. Normalization preserves reward ordering within each prompt group, so lower-reward tokens receive lower advantage weights rather than all tokens sharing a response-level correctness signal.

  6. What implementation details have been clarified? We clarify teacher inputs and initialization, microbatch token-mean reduction in Eq. (3), the fixed scoring signals used during actor updates, and the GRPO judge. Appendix A.2 collects these implementation details.

Summary of Manuscript Revisions

Revision locationWhat's newRaised by
§4.4, Table 3; Appendix A.3Validate the SFT-teacher reward and separate the anchor contribution from the teacher–student difference.Wuku (W1/Q1); 57K5 (W1); G58G (W1/Q1); vAHC (W1)
§4.5; Appendices A.2–A.3, Table A10Add the sequence-mean control and full two-backbone results.Wuku (W2/Q2)
§3.1; Appendix A.1, Table A5Clarify reference-assisted anchor construction and add no-gold and answer-overlap audits.57K5 (Q1–Q2); G58G (W2)
§3.3, Eq. (3); §4.1; Table 4; Appendix A.2Correct loss reduction and teacher definitions; specify causal teacher inputs, fixed rewards, gradient paths, and the GRPO judge.vAHC (W2–W4); 57K5 (Q2)
Appendix A.3Add answer-length, entropy, and reward-to-advantage sign-change analyses.57K5 (Q2–Q3); vAHC (W1)
Abstract and method terminology; ReferencesIdentify privileged-context distillation and external supervision; consolidate duplicate HallusionBench references.vAHC (W5); Wuku (citation comments)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.