acceptodds
Under review as a conference paper at ICLR 2027

DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs

Abstract

Reinforcement learning (RL) is widely used to sharpen reasoning in multimodal large language models (MLLMs), yet its effect on hallucination is uneven. We trace this to two weak points in the correction chain from reward to parameter update. At the rollout level, hard queries—those with high semantic entropy— frequently produce unanimously wrong sample groups, collapsing the group- relative advantage to zero exactly where hallucination risk is highest. At the optimization level, confident-but-wrong tokens are gradient-invisible: a categor- ical policy’s expected score-gradient norm vanishes as its distribution sharpens, so the predictions that most need correction receive the weakest updates. We propose Dual-Entropy Enhanced Policy Optimization (DEEPO), a dual-stage en- hancement combining signal variance regularization with gradient precondition- ing: semantic-entropy-triggered expert prefixes inject grounded continuations on high-uncertainty queries, providing direct supervision and restoring advantage variance, while advantage-sign-aware R´enyi preconditioning counteracts logit- level saturation so correction reaches confident errors in the operational confi- dence regime. Both branches improve over GRPO individually; their interaction is statistically significant on VideoMMMU—the most complex long-horizon task in our evaluation suite (+4.0, 95% CI [1.1, 6.9])—and additive elsewhere. DEEPO reduces hallucination while preserving accuracy and training stability.

Primary area:
reinforcement learning
Keywords:
multimodal, reinforcement learning, hallucinations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Jq37OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DEEPO (Dual-Entropy Enhanced Policy Optimization), a reinforcement learning framework designed to reduce hallucination in multimodal large language models (MLLMs). The authors identify two limitations in existing RL-based optimization approaches: first, high semantic entropy queries may produce unanimously incorrect rollout groups, causing the group-relative advantage signal in GRPO to collapse; second, confident but incorrect predictions receive weak corrective updates due to vanishing policy gradients in low-entropy regimes.

To address these issues, the paper introduces a dual-stage optimization framework. The first stage performs signal repair by activating entropy-triggered expert prefixes on difficult queries, providing grounded continuation signals and restoring advantage variance. The second stage performs delivery repair through an advantage-sign-aware Rényi entropy gradient scaling strategy, aiming to increase corrective updates for confident errors.

The authors evaluate DEEPO on hallucination benchmarks including POPE, VideoHallucer, and HallusionBench, as well as general multimodal reasoning benchmarks such as MMBench, MMSTAR, and VideoMMMU. Results show improvements over GRPO and other RL training strategies on the Qwen2.5-VL-7B backbone.

Critical strengths

Strength 1:

The paper provides a meaningful diagnosis of why RL optimization may fail to correct hallucinations. The observation that difficult queries can produce unanimously wrong rollout groups, causing GRPO-style relative advantages to vanish, is practically relevant and provides a useful explanation for why RL does not always improve hallucination. This diagnosis motivates the proposed intervention naturally and gives the method a clear conceptual foundation.

Strength 2:

The empirical evaluation is comprehensive and carefully designed. The authors compare DEEPO with several RL baselines under the same backbone and report improvements on both hallucination reduction and general reasoning tasks. The ablation study further demonstrates that the two proposed branches provide complementary benefits rather than merely duplicating each other's effects.

Critical weaknesses

Weakness 1:

The expert prefix mechanism introduces additional supervision requirements. Although the authors discuss self-generated hints, the main experiments rely on prefixes obtained from Video-R1 trajectories. This raises questions about scalability and whether the improvement comes primarily from better RL optimization or from additional supervised signals. More experiments with different levels of expert supervision quality and quantity would help clarify this issue.

Weakness 2:

The generalization ability of DEEPO is not sufficiently demonstrated. Most experiments focus on the Qwen2.5-VL-7B backbone and Video-R1 training data. Although the evaluation covers multiple benchmarks, it remains unclear whether the method generalizes to other architectures, larger models, or different RL optimization algorithms.

Questions for authors

Q1:

Can the authors provide empirical evidence that the proposed Rényi gradient scaling actually changes gradient allocation during PPO training, rather than being normalized away by Adam or other optimizer dynamics?

Q2: How sensitive is DEEPO to the quality and amount of expert prefixes? Does the method remain effective when the available prefixes contain incorrect reasoning traces?

Q3: What is the additional computational cost introduced by the entropy-triggered prefix generation compared with standard GRPO training?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Jq37(1/4)

Thank you for your constructive comments and for recognizing the experimental design and component complementarity. We address your concerns in four separate replies. Revisions addressing your comments are highlighted in purple, including the optimizer discussion in Section 4.2, the actual Adam-update diagnostic in Section 5.4 (Table 6) and Appendix A.1, and the discussion of hint controls in Section 5.4 and Appendix A.8. Shared additions on direct hallucination evaluation and the paired VideoHallucer comparison are highlighted in blue. All references below follow the revised numbering.

W1/Q2: Do the gains come from additional supervision, and how does DEEPO depend on expert-prefix quality and availability?

Training design beyond access to expert traces

We compare DEEPO with SFT+GRPO using the same expert traces. SFT+GRPO learns complete trajectories through SFT before RL, whereas DEEPO incorporates selective prefix guidance, auxiliary prefix supervision, and sign-aware gradient scaling within RL.

MethodPOPE ↑VideoHallucer ↑HallusionBench ↑MMBench ↑MMSTAR ↑VideoMMMU ↑
SFT+GRPO, same traces85.056.469.485.959.651.2
DEEPO85.857.669.786.460.853.3
Mean difference+0.8+1.2+0.3+0.5+1.2+2.1

DEEPO achieves higher mean scores on all six benchmarks. The additional paired analysis on VideoHallucer shows improvements in all four seed pairs, with a mean gain of 1.20 percentage points (Appendix A.3, Table 8; Section 5.4).

The component ablations in Table 2 and weighting-rule controls in Table 3 further examine the training design. Together, these results support the value of how expert information is incorporated and how updates are allocated.

Prefix source and quality

Our main experiments reuse existing Video-R1 trajectories without online teacher calls during RL. Video-R1’s CoT annotations are generated by Qwen2.5-VL-72B-Instruct and filtered for quality. This provides a reusable approach to obtaining guidance for other domains: a suitable teacher model can generate CoT trajectories for target-domain training examples, which can then be processed through DEEPO’s prefix truncation and filtering pipeline. The guidance source is therefore not restricted to the existing Video-R1 traces.

Appendix A.8, Tables 13–14 further examine shuffled and self-generated hints:

MethodPOPE ↑HallusionBench ↑VideoMMMU ↑
GRPO83.768.644.7
DEEPO, shuffled hints83.267.744.5
DEEPO-self, self-generated hints84.668.650.8
DEEPO, dataset-derived hints85.869.753.3

Shuffling prefixes across questions removes the gains, supporting the importance of query-relevant guidance. DEEPO-self instead uses prefixes from verified-correct rollouts within the same group, requiring no external expert traces. It retains approximately 71% of the full method’s VideoMMMU gain: (50.8 − 44.7)/(53.3 − 44.7).

Prefix availability

Guidance is selective rather than required for every training instance. Table 18 and Algorithm 1 specify entropy-based triggering and 20% hint dropout. DEEPO-self additionally operates with prefixes available only when the original rollout group contains a verified-correct answer.

These experiments show that useful gains can be obtained with self-generated guidance and selective hint availability, while dataset-derived, question-matched prefixes provide further improvements. The revised discussion appears in Appendix A.8 (p. 19) and Section 5.4 (p. 10).

AuthorsOpenReview ↗

Response to Reviewer Jq37(2/4)

W2: Generalization across model architectures, training data, and RL algorithms

Thank you for raising the question of generalization. Our experiments provide evidence across two MLLM families and demonstrate flexibility in the source of expert guidance.

Cross-family validation

Beyond Qwen2.5-VL-7B, Appendix A.9, Table 16 reports the following four-seed means on InternVL2.5-8B:

MethodPOPE ↑HallusionBench ↑VideoMMMU ↑
InternVL2.5-8B82.155.644.2
+ GRPO83.457.045.3
+ DEEPO85.058.251.0
DEEPO − GRPO+1.6+1.2+5.7

DEEPO improves over GRPO on all three benchmarks, spanning hallucination evaluation and multimodal reasoning. These results demonstrate effectiveness beyond the Qwen model family.

Flexibility of training guidance

DEEPO’s prefix mechanism accepts reasoning trajectories associated with the training questions. As discussed in our previous reply, suitable teacher models can generate trajectories for target-domain examples, which can then enter the same prefix truncation and filtering pipeline.

The existing DEEPO-self experiment in Appendix A.8, Table 14 provides complementary evidence: it uses verified-correct rollouts from the current policy instead of dataset-provided expert trajectories and retains approximately 71% of the full method’s VideoMMMU gain. Thus, the guidance mechanism can operate with both dataset-derived and self-generated prefixes.

Relationship to the RL framework

The Rényi entropy–score-gradient relationship underlying the scaling rule applies to categorical softmax policies and does not depend on a particular model family or GRPO-specific architecture. The scaling factor is computed from token probabilities and advantage signs before gradient aggregation.

The complete DEEPO implementation combines this rule with GRPO’s grouped rollouts and advantage estimation. Extending the design to another RL framework involves integrating expert-conditioned rollouts and token-level weighting with that framework’s sampling and advantage computation.

Together, the cross-family results and alternative guidance-source experiment support the portability of DEEPO’s design and its effectiveness across the evaluated settings.

AuthorsOpenReview ↗

Response to Reviewer Jq37(3/4)

Q1: Does Rényi gradient scaling change actual updates, or is its effect normalized away by Adam?

Thank you for raising this question. We have revised the optimizer discussion in Section 4.2 (pp. 6–7) and added paired Adam-update diagnostics in Section 5.4, Table 6 (p. 10), with the protocol and metric defined in Appendix A.1 (p. 16). These revisions are highlighted in purple.

Token-level reweighting changes gradient aggregation

DEEPO assigns different weights according to token entropy and advantage sign before gradient aggregation. Let hkh_k denote the policy-gradient contribution of token kk. Without and with scaling, the aggregated policy gradients are:

G0=∑khk,Gw=∑kwkhk.G_0=\sum_k h_k,\qquad G_w=\sum_k w_kh_k.

Because the weights vary across tokens, GwG_w is generally not a uniform multiple of G0G_0: both its direction and the relative contributions across parameter coordinates can change. Adam’s normalization therefore does not imply cancellation of this nonuniform reweighting.

Paired diagnostics measure actual Adam parameter updates

We compare three fixed batches at the same checkpoint. Both branches start from identical model parameters and Adam states, reuse the same trajectories, advantages, and other training settings, and differ only in whether Rényi scaling is enabled.

BatchUnscaled update norm ∥Δθ0∥2\lVert\Delta\theta_0\rVert_2DEEPO update norm ∥Δθw∥2\lVert\Delta\theta_w\rVert_2Update cosine similarityRelative update difference
10.018420.018760.987416.13%
20.020670.021050.981719.39%
30.017930.018140.990813.69%

Here, Δθ0\Delta\theta_0 and Δθw\Delta\theta_w are the actual parameter increments after one optimizer step. The relative update difference is:

∥Δθw−Δθ0∥2∥Δθ0∥2.\frac{\lVert\Delta\theta_w-\Delta\theta_0\rVert_2}{\lVert\Delta\theta_0\rVert_2}.

Update norms differ by only 1.17%–1.85%, while the relative differences between update vectors reach 13.69%–19.39%. The cosine similarities also show a change in update direction. Thus, the modulation persists in actual Adam parameter updates and changes their allocation beyond a uniform increase in magnitude.

These parameter-level measurements complement the weighting-rule comparisons in Table 3 and the hallucinated-span analysis in Table 15, connecting the proposed weighting to optimizer behavior, downstream performance, and relative corrective-signal allocation.

Reviewer_ZRuJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies hallucination during reinforcement learning training of multimodal large language models and proposes Dual-Entropy Enhanced Policy Optimization (DEEPO). Built on GRPO, the method uses answer-level semantic entropy to identify questions that need assistance, then generates additional responses using expert reasoning prefixes and provides auxiliary supervision. It also adjusts token-level update weights according to Rényi entropy and the sign of the advantage to address insufficient learning signals and weak updates for confident predictions. The authors evaluate the method on six hallucination and general multimodal reasoning benchmarks and analyze its effects through ablations, controlled comparisons, and results across multiple random seeds. The full method achieves higher scores than the compared training strategies using the same backbone.

Critical strengths
  1. The evaluation has broad coverage: six benchmarks span images, videos, hallucination, and general multimodal reasoning, with a fairly extensive set of comparison methods.
  2. The experimental study is thorough, including branch ablations, alternative designs, and controls for computational budget and expert data, together with results across multiple random seeds and training cost measurements.
Critical weaknesses
  1. The paper emphasizes hallucination mitigation, but the method primarily uses task rewards and model uncertainty for training, without directly distinguishing hallucinations from other errors. The improvements on hallucination evaluations are valuable, but the current analysis does not sufficiently establish that the method specifically targets hallucination. In particular, improvements on general reasoning benchmarks such as VideoMMMU cannot be directly attributed to reduced hallucination.
  2. The justification for the semantic entropy trigger remains insufficient. The authors report a correlation between high semantic entropy and all-wrong groups, but do not compare against directly triggering hints for all-wrong groups based on verifiable rewards, nor sufficiently analyze this correlation. Questions with low semantic entropy but consistently incorrect answers may still be missed.
  3. The differences between the gradient scaling branch and directly related prior work need further explanation. ACPO [1] already emphasizes uncertain decisions under positive advantages and confident decisions under negative advantages; QwenLong-L1.5 [2] also discusses masking negative updates at high-entropy positions. Although their specific formulas differ from DEEPO's, the paper does not sufficiently explain what is added beyond these designs.
  4. The experiments mainly use Qwen2.5-VL-7B, with InternVL2.5-8B as an additional backbone. The validation is concentrated on older 7B–8B models, leaving it unclear whether the method still provides gains on newer and stronger models.

[1] Zijun Xie et al. ACPO: Asymmetric Credit Policy Optimization via Mode-Local Entropy Surrogate. arXiv:2607.03126, 2026. https://arxiv.org/abs/2607.03126
[2] Weizhou Shen et al. QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management. arXiv:2512.12967, 2025. https://arxiv.org/abs/2512.12967

Questions for authors
  1. Have the authors compared against directly triggering expert hints for all-wrong groups or groups with identical rewards based on verifiable rewards? What advantages does the semantic entropy trigger offer over this direct approach?
  2. Beyond final-answer accuracy, is there evidence that further distinguishes gains from reduced hallucination from those due to improvements in general perception or reasoning?
  3. Please explain the main differences between the gradient scaling design and the related designs in ACPO [1] and QwenLong-L1.5 [2], and whether the current experiments sufficiently demonstrate the value of these differences.
  4. Are there results on newer or stronger backbones that support the method's scope of applicability?
  5. The labels “+ w/o Expert Hint / Gradient Scaling” in Table 2 are easy to misread. Please clarify them as “DEEPO with a branch removed” or “with only one branch retained.”
6 comments
AuthorsOpenReview ↗

Response to Reviewer ZRuJ(1/5)

Thank you for your constructive comments. We address your concerns in five separate replies. Revisions made specifically in response to your comments are highlighted in orange: the related-work discussion in Section 2, the trigger comparison in Section 5.4 and Appendix A.2 (Table 7), and the ablation-label clarifications in Section 5.3, Table 2, and Appendix A.8 (Table 12). The shared direct hallucination evaluation in Section 5.4 (Table 5) is highlighted in blue. All manuscript references below follow the revised numbering.

W1/Q2: Does DEEPO reduce hallucinated content beyond improving final-answer accuracy?

We have added a direct evaluation of generated content, assessing hallucination separately from final-answer correctness.

Why can the method help without a dedicated hallucination reward?

DEEPO uses semantic uncertainty to select questions for expert-prefix guidance and combines token entropy with advantage sign to allocate update strength. These mechanisms do not explicitly classify every hallucination or identify every erroneous token. Their intended role is to improve learning on uncertain questions and counteract attenuated corrective signals on highly confident predictions. Whether this reduces hallucinated content must therefore be evaluated directly.

Direct evidence beyond answer accuracy

We annotated responses to the same 200 questions per benchmark across GRPO, SFT+GRPO using the same expert traces, and DEEPO. A response is counted as hallucinated if it contains at least one visual factual claim that contradicts or is unsupported by the provided visual evidence. Rates use all 200 responses, rather than only incorrect answers.

BenchmarkMethodHallucination rate ↓Final-answer accuracy ↑
HallusionBenchGRPO30.5%68.5%
HallusionBenchSFT+GRPO, same traces31.0%69.5%
HallusionBenchDEEPO24.5%70.0%
VideoMMMUGRPO42.5%44.5%
VideoMMMUSFT+GRPO, same traces35.5%51.0%
VideoMMMUDEEPO31.5%53.5%

Relative to same-trace SFT+GRPO, DEEPO reduces observed hallucination rates by 6.5 and 4.0 percentage points, respectively. On HallusionBench, answer accuracy improves by only 0.5 points while the hallucination rate decreases by 6.5 points. The measured benefit thus extends beyond final-answer accuracy. On VideoMMMU, the separate annotation provides direct evidence of fewer responses containing hallucinated claims, rather than attributing the overall benchmark gain to hallucination reduction.

These results are reported in Section 5.4, Table 5 (p. 10). They complement the existing hallucinated-span analysis in Appendix A.9, Table 15: the span analysis examines relative corrective-signal allocation, whereas the new evaluation directly measures hallucination in the trained models’ outputs.

AuthorsOpenReview ↗

Response to Reviewer ZRuJ(2/5)

W2/Q1: Why use semantic-entropy triggering rather than directly providing hints to all-wrong or equal-reward groups?

Thank you for suggesting this comparison. We have added a controlled comparison with all-wrong triggering in Appendix A.2, Table 7 (p. 16), summarized in Section 5.4 (p. 9) and highlighted in orange.

The triggers address different intervention needs

All-wrong triggering identifies groups where every sampled answer is incorrect. Semantic entropy identifies disagreement in answer meaning, including groups that contain some correct answers but still exhibit unstable predictions. Expert guidance can help these uncertain questions even when their original group-relative advantages are nonzero.

Equal-reward groups also include entirely correct groups, so equal rewards alone do not establish a need for guidance. We therefore evaluate the triggers through learning outcomes and actual guidance usage.

Controlled comparison

We compare semantic-entropy and all-wrong triggering using the same training seed and the same fixed prefix rule: both use the first 50% of the expert thought segment, excluding the final answer. All-wrong triggering is determined by task-answer correctness across the original rollout group.

Both configurations retain the same expert trajectories, prefix filtering, hint dropout, auxiliary prefix supervision, gradient scaling, and other training settings. The fixed prefix rule isolates the trigger choice and allows low-entropy questions to receive nonempty prefixes.

TriggerPOPE ↑VideoHallucer ↑HallusionBench ↑VideoMMMU ↑Actual hint usageMean prefix length, tokens
Semantic entropy85.457.069.352.626.8%153.8
All-wrong85.056.569.451.734.6%161.2

Actual hint usage is the fraction of training question instances receiving a nonempty prefix after filtering and dropout. Mean prefix length is calculated over those instances; it differs because the triggers select different questions and expert trajectories.

Semantic-entropy triggering achieves higher scores on three of four benchmarks, with a 0.1-point decrease on HallusionBench, while using hints 7.8 percentage points less frequently.

The result also clarifies our motivation: restoring advantages in all-wrong groups is one benefit of guidance, but does not exhaust its usefulness. Semantic entropy additionally selects uncertain groups containing both correct and incorrect responses.

AuthorsOpenReview ↗

Response to Reviewer ZRuJ(3/5)

W3/Q3: How does DEEPO differ from ACPO and QwenLong-L1.5, and what evidence supports its design?

Thank you for pointing out these related methods. We have added the comparison with ACPO and QwenLong-L1.5 in Section 2 (p. 3), highlighted in orange.

Differences in optimization mechanisms

DEEPO’s design goes beyond distinguishing positive and negative advantages. It uses the relationship between Rényi entropy and the expected squared logit-score gradient norm to construct compensation for attenuated corrective signals.

  • ACPO constructs a mode-local entropy surrogate from the highest token probability and routes gradients according to whether the sampled token matches the policy mode. DEEPO instead applies a stop-gradient Rényi-based factor, using stabilized inverse scaling for negative advantages. The methods therefore differ in their underlying statistics, weighting functions, and gradient propagation.

  • QwenLong-L1.5 uses entropy thresholds to mask selected negative updates. DEEPO continuously adjusts relative token weights. Retaining an unmasked gradient and compensating for its attenuation are different operations: DEEPO explicitly increases the relative corrective weight of low-entropy tokens within negative-advantage trajectories.

DEEPO further combines this token-level modulation with answer-level semantic-entropy-triggered expert conditioning, connecting trajectory generation with update allocation.

Controlled evidence for the weighting design

Table 3 compares alternative weighting rules under the same expert-guidance framework:

Weighting rulePOPE ↑HallusionBench ↑VideoMMMU ↑
Symmetric direct85.069.250.4
Symmetric inverse85.269.351.1
Advantage-sign-only84.969.050.0
DEEPO’s proposed rule85.869.753.3

The proposed rule achieves higher scores on all three benchmarks. Symmetric direct and inverse weighting test whether a single entropy-weighting direction suffices, while advantage-sign-only weighting tests whether distinguishing advantage signs alone explains the gains. These controls support combining entropy-dependent weighting with advantage direction.

The component ablations in Table 2 further examine the combination with expert guidance. Together, these experiments support the value of DEEPO’s specific weighting rule and its integration with the rollout-stage intervention.

References

[1] Xie et al. ACPO: Asymmetric Credit Policy Optimization via Mode-Local Entropy Surrogate. arXiv, 2026.

[2] Shen et al. QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management. arXiv, 2025.

Reviewer_mj35OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DEEPO, which combines semantic-entropy-triggered expert prefixes with advantage-sign-aware Rényi entropy weighting for multimodal reinforcement learning. The method is motivated by two limitations: uninformative relative advantages in unanimously incorrect rollout groups, and weak score gradients for highly confident predictions. The experiments demonstrate improvements over GRPO and several other training strategies across hallucination and reasoning benchmarks.

Critical strengths
  1. The paper addresses an important problem and presents a concrete, implementable training method.

  2. The empirical evaluation includes four training seeds, component ablations, weighting-rule controls, shuffled and self-generated hints, a second backbone, and computational-cost measurements.

  3. The full method consistently improves over GRPO on the evaluated benchmarks. The positive interaction between the two branches on VideoMMMU provides evidence of complementarity in that setting.

  4. The paper explicitly acknowledges several limitations, including the inability to rescue consistently wrong, low-semantic-entropy groups and the local scope of its theoretical analysis.

Critical weaknesses
  1. The loss of learning signal in all-wrong GRPO groups is a well-known limitation; this observation alone does not establish novelty. The substantive question is whether DEEPO’s expert-prefix intervention improves upon established approaches to incorporating external demonstrations.

LUFFY [1] is particularly relevant because it combines off-policy demonstrations with on-policy rollouts in Mixed-Policy GRPO and addresses learning beyond the model’s existing capabilities. However, it is neither discussed nor evaluated in this submission.

A comparison with LUFFY, or a comparably strong baseline, using the same backbone, training prompts, expert-trace source, and comparable supervision and compute budgets is important for assessing the claimed contribution.

  1. The relationship to SEED-GRPO is insufficiently addressed. SEED-GRPO [2] already uses prompt-level semantic entropy to modulate GRPO updates, taking more conservative steps on uncertain questions. The authors should discuss the differences.

  2. The motivations do not establish that the proposed design specifically corrects hallucinations.

The two motivating observations are plausible, but several steps in the argument remain unverified:

  • High semantic entropy is neither equivalent to hallucination nor sufficient to imply an all-wrong group. Consistently incorrect answers may instead have low semantic entropy. The paper acknowledges that both branches can be ineffective in this case.

  • Low token entropy does not identify an erroneous token. Correct facts and structural tokens can also be highly predictable. Broadcasting a negative sequence-level advantage does not resolve which tokens caused the failure.

  • A small gradient does not establish that amplifying it produces more accurate correction. The outcome also depends on reward quality, credit assignment, and interactions through shared parameters.

  1. The strongest controlled results leave the hallucination-specific contribution uncertain. Against SFT+GRPO using the same expert traces, DEEPO improves POPE by 0.8 points and HallusionBench by 0.3 points, but the reported paired 95% confidence intervals are respectively [−0.2,1.4][-0.2,1.4] and [−0.1,0.9][-0.1,0.9]. Both include zero. This means that an additional benefit on these hallucination benchmarks has not been demonstrated convincingly.

[1] Learning to Reason under Off-Policy Guidance. In NeurIPS 2025. [2] SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization. In arXiv 2025.

Questions for authors

See Weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mj35(1/4)

Thank you for your constructive comments. We address your four concerns in four separate replies. Revisions addressing your comments are highlighted in blue in the revised manuscript. We have added the related-work comparisons in Section 2, the multimodal LUFFY comparison in Appendix A.4 (Table 9), the direct hallucination evaluation in Section 5.4 (Table 5), and the paired VideoHallucer analysis in Section 5.4 and Appendix A.3 (Table 8). All manuscript references below follow the revised numbering.

W1: Relationship to LUFFY, direct comparison, and contribution positioning

We agree that the loss of learning signal in all-wrong GRPO groups does not, by itself, establish novelty. The relevant question is whether DEEPO offers an effective way to incorporate expert information beyond existing approaches. We have therefore added a discussion of LUFFY in Section 2 and a direct comparison with its multimodal adaptation in Appendix A.4.

LUFFY combines off-policy demonstrations with on-policy rollouts through Mixed-Policy GRPO and policy shaping. DEEPO instead uses semantic entropy to selectively introduce truncated expert prefixes, applies RL updates to policy-generated continuations with auxiliary supervision on the prefixes, and allocates token-level update strength through advantage-sign-aware Rényi scaling. Our contribution lies in this intervention and optimization design, rather than in identifying the all-wrong-group limitation.

W1.1: How is LUFFY adapted to the multimodal setting?

LUFFY was originally studied in text-only reasoning. Applying it to our image and video tasks requires additional multimodal adaptation. We preserved its core learning mechanisms and made the following adaptations:

  • Aligning teacher and student visual evidence. Both follow the adaptive video-sampling protocol in Appendix G, using 4–64 frames per video, with identical sampled frames, preprocessing settings, and subtitle access. This ensures that the visual evidence underlying teacher guidance is also available to the student.

  • Defining the optimization scope. We freeze the visual encoder and retain LUFFY’s original policy-shaping mechanism, restricting this adaptation to policy learning over existing visual representations. Since low-probability tokens may reflect either perceptual or reasoning difficulties, we do not interpret all such tokens as reasoning deficiencies or introduce additional per-token reliability weights.

  • Preserving conditioning consistency for policy ratios. Although frame counts adapt across videos, the sampled frames, crops, and temporal information are fixed and reused between each rollout and its corresponding policy updates. This prevents changes in visual input from confounding the policy ratio.

These choices address the requirements of multimodal training while preserving LUFFY’s core learning mechanisms.

W1.2: How does DEEPO perform against the adapted LUFFY baseline?

The comparison below is included in Appendix A.4, Table 9 (p. 17), with a summary in Section 5.2 (p. 8). The adapted LUFFY uses the same initial checkpoint, training prompts, expert-trace source, and shared training settings as DEEPO. LUFFY is trained with a single seed; GRPO and DEEPO scores are the four-seed means from Table 1.

MethodPOPE ↑VideoHallucer ↑HallusionBench ↑MMBench ↑MMSTAR ↑VideoMMMU ↑
GRPO83.752.068.684.557.844.7
LUFFY, multimodal adaptation83.751.468.785.258.343.5
DEEPO85.857.669.786.460.853.3
DEEPO − LUFFY+2.1+6.2+1.0+1.2+2.5+9.8

DEEPO outperforms the adapted LUFFY baseline by 2.1, 6.2, and 1.0 percentage points on the three hallucination benchmarks, and by 1.2, 2.5, and 9.8 percentage points on the three general multimodal benchmarks.

These results provide direct comparative evidence of DEEPO’s effectiveness relative to the evaluated LUFFY adaptation in the studied multimodal setting. Since this additional LUFFY experiment uses a single training seed, it does not assess variability across LUFFY training seeds.

AuthorsOpenReview ↗

Response to Reviewer mj35(2/4)

W2: Relationship to SEED-GRPO and when to use conservative updates or expert guidance

Thank you for pointing out this relevant work. We have added the comparison with SEED-GRPO in Section 2 (p. 3), highlighted in blue.

SEED-GRPO and DEEPO both address semantic uncertainty near the model’s capability boundary, but intervene differently. SEED-GRPO adjusts the learning weights of existing trajectories; DEEPO introduces expert guidance to improve the trajectories generated and separately allocates update strength according to token-level confidence and advantage sign.

W2.1: When does high uncertainty call for conservative updates, and when can expert guidance help?

When learning from existing uncertain trajectories, downweighting can limit the influence of unreliable learning signals. SEED-GRPO follows this approach by using prompt-level semantic entropy to take more conservative steps on highly uncertain questions while retaining stronger learning signals on confident ones.

When suitable expert guidance is available, a complementary intervention is to improve the trajectories available for learning. Downweighting limits the influence of unreliable trajectories, but the weighting operation itself does not supply missing problem-solving information. DEEPO’s expert prefixes introduce partial reasoning guidance for questions the model struggles to solve independently.

Specifically, DEEPO changes the generation conditions for questions with high semantic uncertainty. A truncated expert prefix provides partial guidance, while the current policy generates the continuation, which is then evaluated by the reward function. This allows the model to explore solution paths that would otherwise be difficult to generate independently and learn from them through reward feedback.

The distinction is therefore between conservatively learning from existing uncertain trajectories and using available expert information to improve trajectory generation. These interventions are not mutually exclusive.

W2.2: Does expert guidance imply stronger updates on all uncertain questions?

No. DEEPO does not directly amplify a question’s updates because its semantic entropy is high. Answer-level semantic entropy determines when to introduce guidance, whereas token-level Rényi entropy and advantage sign determine how to allocate update strength.

For negative advantages, the scaling rule assigns greater relative corrective weight to low-entropy tokens, motivated by the attenuation of score gradients on highly confident predictions. For non-negative advantages, it assigns lower relative reinforcement weights to low-entropy tokens. Batch-mean normalization redistributes the weights, while KL regularization and PPO clipping continue to constrain policy updates.

Thus, any increased relative corrective weight is tied to token-level confidence and negative advantage—not to high prompt-level semantic entropy alone. Expert conditioning does not automatically justify stronger updates, nor does DEEPO require all updates to become stronger after guidance.

W2.3: What evidence supports this design motivation?

Figure 4 in Appendix A.6 examines HallusionBench and MMStar after converting multiple-choice questions into cloze-style queries, removing predefined options to assess semantic stability without answer-choice cues. DEEPO exhibits both lower answer-level semantic entropy and smaller dispersion across questions on these two tasks.

Together with the accuracy improvements in the main results, this finding supports the motivation of mitigating uncertainty-driven guessing. The component ablations in Table 2 and weighting-rule controls in Table 3 further examine the roles of expert guidance and update allocation.

These results support DEEPO’s design, while the distinction from SEED-GRPO is primarily mechanistic: semantic entropy selects the generation-time intervention, and token entropy together with advantage sign governs update allocation. We do not interpret these experiments as a direct empirical comparison with SEED-GRPO.

AuthorsOpenReview ↗

Response to Reviewer mj35(3/4)

Thank you for highlighting these gaps in our argument. We agree that high semantic entropy is not equivalent to hallucination, low token entropy is not an error label, and amplifying small gradients does not itself guarantee more accurate correction. We therefore support the design through mechanism analysis, controlled ablations, and direct output evaluation. The new hallucination evaluation appears in Section 5.4, Table 5 (p. 10), highlighted in blue.

W3.1: Why use semantic entropy when it neither identifies hallucination nor implies an all-wrong group?

DEEPO uses semantic entropy to identify semantic uncertainty warranting intervention, rather than to detect every hallucination. Prior work explains how reward schemes can incentivize guessing under uncertainty [1] and how semantic entropy detects a class of semantically unstable incorrect generations [2].

All-wrong groups are only one situation where guidance can help. Even when a group contains correct answers, substantial semantic disagreement may remain, and expert prefixes can help the model explore more reliable reasoning paths. Conversely, consistently incorrect, low-entropy groups may escape the trigger; we acknowledge this coverage limitation. Entropy estimation reuses GRPO’s existing grouped responses without additional sampling for estimation.

W3.2: If low entropy does not indicate error, why give low-entropy tokens greater corrective weight?

The mechanism allocates corrective signals rather than localizing erroneous tokens. Within negative-advantage trajectories, highly confident tokens can have attenuated score gradients. We therefore combine entropy with advantage sign instead of treating low entropy alone as evidence of error.

Annotations on 600 failed responses in Appendix A.9, Table 15 show that hallucinated spans receive 1.75×/1.80× the mean weight of non-error spans on HallusionBench/VideoMMMU, with effective logit-step intensity ratios of approximately 1.15×/1.25×. Function words are explicitly labeled as non-error content.

These results support improved relative signal allocation.

W3.3: What evidence shows actual hallucination reduction beyond larger gradients?

Tables 2–3 test the optimization design through component ablations and weighting-rule controls. Additionally, we directly annotated responses to the same 200 questions per benchmark, assessing hallucination separately from final-answer correctness. A response is counted as hallucinated if it contains at least one visual factual claim contradicting or unsupported by the visual evidence; all responses enter the denominator.

BenchmarkMethodHallucination rate ↓Answer accuracy ↑
HallusionBenchGRPO30.5%68.5%
HallusionBenchSFT+GRPO, same traces31.0%69.5%
HallusionBenchDEEPO24.5%70.0%
VideoMMMUGRPO42.5%44.5%
VideoMMMUSFT+GRPO, same traces35.5%51.0%
VideoMMMUDEEPO31.5%53.5%

Compared with same-trace SFT+GRPO, DEEPO reduces observed hallucination rates by 6.5 and 4.0 percentage points. On HallusionBench, accuracy improves by only 0.5 points, showing that the measured benefit extends beyond final-answer scores. DEEPO’s responses are also slightly longer than GRPO’s, so the lower hallucination rates relative to GRPO are not accompanied by shorter average outputs.

These results directly support lower hallucination rates for the full method on the evaluated samples.

References

[1] Kalai et al. Why Language Models Hallucinate. OpenAI and Georgia Tech, 2025. Link

[2] Farquhar et al. Detecting hallucinations in large language models using semantic entropy. Nature, 630, 625–630, 2024. Link

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion Status and Revisions

We thank the PC, SAC, AC and reviewers for the constructive discussion. We briefly summarize the current status of our submission.

Reviewer responses.

  • Reviewer mj35: confirmed that all their questions were resolved and raised their score.
  • Reviewer ZRuJ: indicated that the rebuttal addressed several concerns and raised their rating. We will state the remaining scope limitation (relatively older backbones) clearly in the revision.
  • Reviewer Jq37: noted that most concerns were addressed and maintained their weak-acceptance recommendation.

Main additions in the revised manuscript. Direct hallucination-rate annotation (Sec. 5.4, Table 5); paired VideoHallucer comparison against same-trace SFT+GRPO (App. A.3, Table 8); multimodal LUFFY comparison (App. A.4, Table 9); trigger comparison with all-wrong triggering (App. A.2, Table 7); Adam-update diagnostics (Sec. 5.4, Table 6); and discussions of SEED-GRPO, ACPO and QwenLong-L1.5 (Sec. 2).

Thank you very much to the AC and all reviewers for the time and care devoted to our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.