- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes ASYM-OPSD, an asymmetric on-policy distillation framework for mitigating hallucinations in multimodal large language models. External models construct and verify multi-granularity visual anchors and repair model responses to train a teacher. The student generates responses from images and questions alone, while the frozen, anchor-conditioned teacher scores these responses token by token. The teacher–student log-probability gap provides dense rewards, combined with group-relative advantages and a curriculum from detailed to brief anchors. Experiments on Qwen3-VL-4B and InternVL3.5-8B report improvements across four hallucination benchmarks while largely preserving general VQA performance, without additional inference-time components.
Critical strengths
The paper addresses an important problem with fine-grained feedback targeting localized hallucinations.
Scoring student-generated responses through teacher forcing avoids aligning independently generated teacher and student responses.
Results improve across two backbone families, with no additional teacher or anchors required at inference.
Critical weaknesses
-
Equation (1) measures relative teacher preference and is affected by teacher fine-tuning, confidence, wording, style, and the evolving student distribution. Correct tokens can receive negative rewards, while hallucinated tokens favored by the teacher or erroneous anchors can receive positive rewards. Group centering further disconnects positive advantages from factual correctness.
-
Loss aggregation is ambiguous. Equation (3) averages within each response, whereas Appendix A.2 (page 19) specifies token-mean loss aggregation. Averaging over all valid batch tokens and averaging responses equally assign different weights to variable-length responses. The authors should clarify the implemented objective and provide matching pseudocode.
-
Does OPSD use the repaired teacher? How are the GRPO judge’s paired True/False verdicts mapped to rewards?
-
In Equations (1)–(3), are student log-probabilities computed using the rollout policy or the policy being updated? Are rewards and advantages detached from the computation graph?
-
The self-distillation terminology needs qualification. Although teacher and student share a base model, external models generate anchors and repair responses used to fine-tune the teacher. The method therefore incorporates substantial external supervision; the authors should clarify its distinction from externally assisted distillation.
Questions for authors
Please address the questions raised in the weaknesses above.
4 comments
Response to Reviewer vAHC (2/2)
W4. Rollout-policy scores and actor-update gradients
The reward uses cached rollout-policy log-probabilities. Teacher scores, rewards, and advantages remain fixed during actor updates; advantages are computed under torch.no_grad(). Current-policy log-probabilities retain gradients in the PPO ratio.
W5. Distillation terminology and external supervision
We will revise the terminology to “privileged-context on-policy distillation” and clarify the external supervision used to construct anchors and repair responses.
Revision Plan. We will add the reward validation to Section 4.4 and correct Eq. (3), the teacher definitions in Table 3, and the GRPO details in Appendix A.2. We will also update the distillation terminology.
Thank you for the rebuttal. However, I still have several concerns:
-
W1. Reward validity. The additional analysis does not fully address my concern that relative teacher preference is not equivalent to factual correctness, as it can be affected by teacher's wording and style. I acknowledge that these factors are difficult to quantify, but I encourage the authors to explicitly discuss this limitation in the paper.
-
W2. Eq. (3). The formulation still needs revision. Calling it the "standard clipped GRPO objective" (line 244) is imprecise, and the group number is not explicitly reflected.
-
GRPO judge. Please include the exact scoring rubric and judge prompt for reproducibility.
I also suggest highlighting manuscript revisions in blue to make the changes easier to identify.
I encourage the authors to carefully address these remaining issues. I will consider the subsequent discussion and comments from other reviewers before making my final decision.
Thank you for your helpful feedback and additional questions.
- We have added the original reward script used by our GRPO baseline to the supplementary material at ./training/verl/workers/reward_manager/graphallu.py, together with the script at ./training/scripts/run_grpo_graphallu.sh.
- We have resubmitted the manuscript, with newly added content highlighted in blue. In response to your follow-up:
- Reward interpretation: We explicitly clarify that the reward captures relative teacher preference rather than directly judging factual correctness, and may also reflect wording and style.
- Objective notation: We describe the objective as “a PPO-style clipped surrogate with dense group-relative token advantages” and explicitly specify prompt-group, rollout, and token indices, while preserving the implemented microbatch token-mean reduction.
- GRPO judge: The exact judge prompt, including the scoring instructions and few-shot examples, is now provided in Appendix A.6.
We hope these revisions clarify our method and its implementation. Thank you again for your constructive feedback.
Response to Reviewer vAHC (1/2)
We thank the reviewer for the careful reading and the questions about the reward, optimization, and supervision sources.
Summary. (i) Human-labeled reward analysis, anchor checks, and entropy/length measurements address grounding and response collapse (W1). (ii) We clarify microbatch token-mean reduction and will align Eq. (3) with the implementation (W2). (iii) Both teachers use repaired targets; their training inputs differ by the anchor. The GRPO reward and PPO gradient paths are specified below (W3–W4). (iv) We will revise the terminology to identify the privileged context and external supervision (W5).
W1. Grounding, anchor quality, and response collapse
AUROC ranks hallucinated tokens using the negative signal; G>H counts responses with higher grounded-token means. The anchor input accounts for most of the grounded–hallucinated mean separation on both backbones.
Anchor quality. Our anchor-cleaning pipeline uses image-based checks by Gemini and GPT, followed by repair or removal of rejected anchors (Appendix A.1, Table A3). Residual errors and omissions can make some token rewards noisy. The human-labeled results above and the held-out hallucination reductions support the value of this feedback for learning image-grounded answers.
Entropy and answer length. In the reported Qwen run, actor entropy changes from 0.573 to 0.511. The reported character lengths before and after ASYM-OPSD are:
These measurements show no sign of severe collapse in entropy or answer length. On Qwen MMHal, hallucination falls from 15.33% to 5.21% while answer length stays close to its original level (146 → 149 characters).
W2. Loss aggregation and Eq. (3)
The implementation uses token-mean reduction within each microbatch:
LB=∑i∈B,tmi,t∑i∈B,tmi,tℓi,t
where m masks valid response tokens and ℓ is the clipped PPO loss. Each fixed-size microbatch loss is divided by the gradient-accumulation factor. Longer responses therefore contribute more token terms within a microbatch. We will revise Eq. (3) and the pseudocode to state this reduction and the gradient-accumulation and distributed-averaging steps explicitly.
W3. Teacher preparation and the GRPO judge
Both teachers use repaired response targets. The OPSD teacher is trained with the original question; the ASYM-OPSD teacher is trained with the anchor and original question. The ASYM-OPSD teacher is frozen during RL.
GRPO calls gpt-5.6-luna from verl's reward manager. The judge receives the question, reference answer, and prediction, and returns a correctness score from 0.0 to 1.0 in 0.1 increments. The score credits correct content in responses that also contain hallucinations. The reward manager normalizes scores within each prompt group and stores the normalized score at the last valid token. The GRPO trainer then computes a response-level advantage and applies the same advantage to all valid tokens in the response.