- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DEEPO (Dual-Entropy Enhanced Policy Optimization), a reinforcement learning framework designed to reduce hallucination in multimodal large language models (MLLMs). The authors identify two limitations in existing RL-based optimization approaches: first, high semantic entropy queries may produce unanimously incorrect rollout groups, causing the group-relative advantage signal in GRPO to collapse; second, confident but incorrect predictions receive weak corrective updates due to vanishing policy gradients in low-entropy regimes.
To address these issues, the paper introduces a dual-stage optimization framework. The first stage performs signal repair by activating entropy-triggered expert prefixes on difficult queries, providing grounded continuation signals and restoring advantage variance. The second stage performs delivery repair through an advantage-sign-aware Rényi entropy gradient scaling strategy, aiming to increase corrective updates for confident errors.
The authors evaluate DEEPO on hallucination benchmarks including POPE, VideoHallucer, and HallusionBench, as well as general multimodal reasoning benchmarks such as MMBench, MMSTAR, and VideoMMMU. Results show improvements over GRPO and other RL training strategies on the Qwen2.5-VL-7B backbone.
Critical strengths
Strength 1:
The paper provides a meaningful diagnosis of why RL optimization may fail to correct hallucinations. The observation that difficult queries can produce unanimously wrong rollout groups, causing GRPO-style relative advantages to vanish, is practically relevant and provides a useful explanation for why RL does not always improve hallucination. This diagnosis motivates the proposed intervention naturally and gives the method a clear conceptual foundation.
Strength 2:
The empirical evaluation is comprehensive and carefully designed. The authors compare DEEPO with several RL baselines under the same backbone and report improvements on both hallucination reduction and general reasoning tasks. The ablation study further demonstrates that the two proposed branches provide complementary benefits rather than merely duplicating each other's effects.
Critical weaknesses
Weakness 1:
The expert prefix mechanism introduces additional supervision requirements. Although the authors discuss self-generated hints, the main experiments rely on prefixes obtained from Video-R1 trajectories. This raises questions about scalability and whether the improvement comes primarily from better RL optimization or from additional supervised signals. More experiments with different levels of expert supervision quality and quantity would help clarify this issue.
Weakness 2:
The generalization ability of DEEPO is not sufficiently demonstrated. Most experiments focus on the Qwen2.5-VL-7B backbone and Video-R1 training data. Although the evaluation covers multiple benchmarks, it remains unclear whether the method generalizes to other architectures, larger models, or different RL optimization algorithms.
Questions for authors
Q1:
Can the authors provide empirical evidence that the proposed Rényi gradient scaling actually changes gradient allocation during PPO training, rather than being normalized away by Adam or other optimizer dynamics?
Q2: How sensitive is DEEPO to the quality and amount of expert prefixes? Does the method remain effective when the available prefixes contain incorrect reasoning traces?
Q3: What is the additional computational cost introduced by the entropy-triggered prefix generation compared with standard GRPO training?
6 comments
Response to Reviewer Jq37(2/4)
W2: Generalization across model architectures, training data, and RL algorithms
Thank you for raising the question of generalization. Our experiments provide evidence across two MLLM families and demonstrate flexibility in the source of expert guidance.
Cross-family validation
Beyond Qwen2.5-VL-7B, Appendix A.9, Table 16 reports the following four-seed means on InternVL2.5-8B:
| Method | POPE ↑ | HallusionBench ↑ | VideoMMMU ↑ |
|---|---|---|---|
| InternVL2.5-8B | 82.1 | 55.6 | 44.2 |
| + GRPO | 83.4 | 57.0 | 45.3 |
| + DEEPO | 85.0 | 58.2 | 51.0 |
| DEEPO − GRPO | +1.6 | +1.2 | +5.7 |
DEEPO improves over GRPO on all three benchmarks, spanning hallucination evaluation and multimodal reasoning. These results demonstrate effectiveness beyond the Qwen model family.
Flexibility of training guidance
DEEPO’s prefix mechanism accepts reasoning trajectories associated with the training questions. As discussed in our previous reply, suitable teacher models can generate trajectories for target-domain examples, which can then enter the same prefix truncation and filtering pipeline.
The existing DEEPO-self experiment in Appendix A.8, Table 14 provides complementary evidence: it uses verified-correct rollouts from the current policy instead of dataset-provided expert trajectories and retains approximately 71% of the full method’s VideoMMMU gain. Thus, the guidance mechanism can operate with both dataset-derived and self-generated prefixes.
Relationship to the RL framework
The Rényi entropy–score-gradient relationship underlying the scaling rule applies to categorical softmax policies and does not depend on a particular model family or GRPO-specific architecture. The scaling factor is computed from token probabilities and advantage signs before gradient aggregation.
The complete DEEPO implementation combines this rule with GRPO’s grouped rollouts and advantage estimation. Extending the design to another RL framework involves integrating expert-conditioned rollouts and token-level weighting with that framework’s sampling and advantage computation.
Together, the cross-family results and alternative guidance-source experiment support the portability of DEEPO’s design and its effectiveness across the evaluated settings.
Response to Reviewer Jq37(3/4)
Q1: Does Rényi gradient scaling change actual updates, or is its effect normalized away by Adam?
Thank you for raising this question. We have revised the optimizer discussion in Section 4.2 (pp. 6–7) and added paired Adam-update diagnostics in Section 5.4, Table 6 (p. 10), with the protocol and metric defined in Appendix A.1 (p. 16). These revisions are highlighted in purple.
Token-level reweighting changes gradient aggregation
DEEPO assigns different weights according to token entropy and advantage sign before gradient aggregation. Let denote the policy-gradient contribution of token . Without and with scaling, the aggregated policy gradients are:
Because the weights vary across tokens, is generally not a uniform multiple of : both its direction and the relative contributions across parameter coordinates can change. Adam’s normalization therefore does not imply cancellation of this nonuniform reweighting.
Paired diagnostics measure actual Adam parameter updates
We compare three fixed batches at the same checkpoint. Both branches start from identical model parameters and Adam states, reuse the same trajectories, advantages, and other training settings, and differ only in whether Rényi scaling is enabled.
| Batch | Unscaled update norm | DEEPO update norm | Update cosine similarity | Relative update difference |
|---|---|---|---|---|
| 1 | 0.01842 | 0.01876 | 0.9874 | 16.13% |
| 2 | 0.02067 | 0.02105 | 0.9817 | 19.39% |
| 3 | 0.01793 | 0.01814 | 0.9908 | 13.69% |
Here, and are the actual parameter increments after one optimizer step. The relative update difference is:
Update norms differ by only 1.17%–1.85%, while the relative differences between update vectors reach 13.69%–19.39%. The cosine similarities also show a change in update direction. Thus, the modulation persists in actual Adam parameter updates and changes their allocation beyond a uniform increase in magnitude.
These parameter-level measurements complement the weighting-rule comparisons in Table 3 and the hallucinated-span analysis in Table 15, connecting the proposed weighting to optimizer behavior, downstream performance, and relative corrective-signal allocation.
Response to Reviewer Jq37(1/4)
Thank you for your constructive comments and for recognizing the experimental design and component complementarity. We address your concerns in four separate replies. Revisions addressing your comments are highlighted in purple, including the optimizer discussion in Section 4.2, the actual Adam-update diagnostic in Section 5.4 (Table 6) and Appendix A.1, and the discussion of hint controls in Section 5.4 and Appendix A.8. Shared additions on direct hallucination evaluation and the paired VideoHallucer comparison are highlighted in blue. All references below follow the revised numbering.
W1/Q2: Do the gains come from additional supervision, and how does DEEPO depend on expert-prefix quality and availability?
Training design beyond access to expert traces
We compare DEEPO with SFT+GRPO using the same expert traces. SFT+GRPO learns complete trajectories through SFT before RL, whereas DEEPO incorporates selective prefix guidance, auxiliary prefix supervision, and sign-aware gradient scaling within RL.
DEEPO achieves higher mean scores on all six benchmarks. The additional paired analysis on VideoHallucer shows improvements in all four seed pairs, with a mean gain of 1.20 percentage points (Appendix A.3, Table 8; Section 5.4).
The component ablations in Table 2 and weighting-rule controls in Table 3 further examine the training design. Together, these results support the value of how expert information is incorporated and how updates are allocated.
Prefix source and quality
Our main experiments reuse existing Video-R1 trajectories without online teacher calls during RL. Video-R1’s CoT annotations are generated by Qwen2.5-VL-72B-Instruct and filtered for quality. This provides a reusable approach to obtaining guidance for other domains: a suitable teacher model can generate CoT trajectories for target-domain training examples, which can then be processed through DEEPO’s prefix truncation and filtering pipeline. The guidance source is therefore not restricted to the existing Video-R1 traces.
Appendix A.8, Tables 13–14 further examine shuffled and self-generated hints:
Shuffling prefixes across questions removes the gains, supporting the importance of query-relevant guidance. DEEPO-self instead uses prefixes from verified-correct rollouts within the same group, requiring no external expert traces. It retains approximately 71% of the full method’s VideoMMMU gain: (50.8 − 44.7)/(53.3 − 44.7).
Prefix availability
Guidance is selective rather than required for every training instance. Table 18 and Algorithm 1 specify entropy-based triggering and 20% hint dropout. DEEPO-self additionally operates with prefixes available only when the original rollout group contains a verified-correct answer.
These experiments show that useful gains can be obtained with self-generated guidance and selective hint availability, while dataset-derived, question-matched prefixes provide further improvements. The revised discussion appears in Appendix A.8 (p. 19) and Section 5.4 (p. 10).