acceptodds
Under review as a conference paper at ICLR 2027

DSE-GRPO: Differential Semantic Entropy for Debiased Noisy-Rollout Policy Optimization

Abstract

Reinforcement learning (RL) post-training for vision–language models (VLMs) increasingly relies on noisy rollouts to encourage exploration and improve robustness. However, mixing rollouts generated from clean and perturbed images introduces a hidden off-policy mismatch: noise can change the semantic interpretation of the input, yielding biased and high-variance policy gradients when these trajectories are optimized together under GRPO. We propose DSE-GRPO, an uncertainty-aware variant of GRPO that explicitly measures and suppresses noise-induced semantic drift. Our key idea is Differential Semantic Entropy (DSE): we estimate semantic entropy over response clusters for clean and noisy rollout groups separately, and use their non-negative gap as a differential uncertainty signal. DSE-GRPO then adaptively down-weights the advantages of noisy rollouts with large DSE, filtering harmful off-policy updates while retaining beneficial exploratory trajectories. We provide theoretical analysis showing that DSE-based reweighting attenuates off-policy bias and stabilizes optimization dynamics, while slowing policy-entropy collapse to preserve exploration. Empirically, RL-tuning with Geometry3K and MMK12 shows that DSE-GRPO consistently outperforms vanilla GRPO and NoisyRollout baselines, improving the average accuracy of Qwen2.5-VL-7B and Qwen3-VL-8B by 8.0 and 1.9 percentage points, respectively, across five out-of-domain benchmarks, with markedly smoother training dynamics.

Primary area:
reinforcement learning
Keywords:
Semantic Entropy, MLLM, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LuEoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Differential Semantic Entropy GRPO (DSE-GRPO) to address the potential off-policy mismatch introduced by noisy rollouts in reinforcement learning post-training for vision-language models. The proposed method measures the difference in semantic entropy between clean and noisy rollout groups and uses this signal to adaptively down-weight the advantages of noisy responses. The authors provide theoretical analyses of the effects of advantage reweighting on optimization bias, update magnitude, and policy entropy. Experimental results on several vision-language benchmarks show improvements over vanilla GRPO and NoisyRollout baselines.

Critical strengths
  1. This paper identifies a meaningful problem in noisy-rollout RL training for VLMs, where image perturbations may introduce unreliable learning signals and affect optimization.

  2. The proposed method is simple to implement and introduces negligible computational overhead compared with standard noisy-rollout training.

  3. The experiments cover multiple model scales and benchmarks, with ablation studies demonstrating the empirical benefits of the proposed differential entropy term.

Critical weaknesses
  1. Insufficient motivation and methodological justification. The paper lacks a systematic analysis of how noisy rollouts lead to harmful policy updates. More importantly, the connection between the identified off-policy mismatch and the proposed differential semantic entropy is not sufficiently established.

  2. Questionable effectiveness of DSE as a measure of semantic drift. DSE only measures increases in answer entropy rather than the actual discrepancy between clean and noisy answer distributions. Substantial semantic shifts can occur with zero DSE, while increased entropy may also reflect beneficial exploration.

  3. Limited theoretical contribution. The theoretical analyses mainly characterize general properties of bounded advantage reweighting. They do not establish that DSE-based weighting reduces gradient bias or variance, nor do they explain why this particular uncertainty signal should improve optimization.

  4. Insufficient justification for the algorithm design. The paper directly introduces a multiplicative linear weighting function without adequately explaining its formulation or why it is preferable to alternative weighting strategies. The methodological novelty over existing semantic-entropy-based GRPO methods appears relatively limited.

  5. Insufficient empirical validation of the proposed mechanism. The experiments primarily demonstrate improvements in downstream accuracy without directly verifying the relationship between DSE and gradient reliability. Additionally, the reported gains lack multi-seed statistical analysis, making their robustness difficult to assess.

Questions for authors
  1. Can the authors provide a more detailed analysis of how noisy rollouts introduce gradient bias and explain why differential semantic entropy is an appropriate proxy for identifying harmful updates?

  2. Since DSE cannot capture semantic shifts that preserve answer entropy, why not directly measure the discrepancy between clean and noisy answer distributions, such as through Jensen-Shannon divergence?

  3. Can the authors provide empirical evidence showing that larger DSE values correspond to less reliable policy gradients, for example through gradient alignment or bias measurements?

  4. What motivates the specific multiplicative linear advantage-weighting function? Can the authors compare it with alternative weighting functions or uncertainty-based selection strategies?

  5. Can the authors provide multi-seed experiments with variance estimates or confidence intervals to demonstrate the statistical robustness of the reported improvements?

Reviewer_HGbHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DSE-GRPO, an uncertainty-aware GRPO, which combines clean/noisy rollout sampling with semantic-entropy-based advantage weighting. Its main addition is a positive clean–noisy entropy gap that further downweights noisy rollouts. The motivation is reasonable, and the performance is good.

Critical strengths
  1. Practical motivation. Distinguishing intrinsic uncertainty from uncertainty introduced by image perturbations is a reasonable direction for improving noisy-rollout training.

  2. The proposed method is simple and inexpensive. The method reuses sampled responses and requires only answer clustering and lightweight advantage reweighting.

  3. Promising empirical results. Improvements are reported across several benchmarks and model sizes. Table 3 includes a relevant absolute-entropy-only baseline, providing initial evidence for the differential term.

Critical weaknesses
  1. DSE is not validated as a signal of harmful semantic drift. DSE measures an increase in answer dispersion rather than a discrepancy between the clean and noisy answer distributions. For example, clean rollouts may consistently produce the correct answer A, while noisy rollouts consistently produce an incorrect answer B. Both groups have zero semantic entropy, yielding zero DSE despite a complete shift in answers. Conversely, finite-sample fluctuations can produce positive DSE even when the underlying answer distributions are identical. Although the paper acknowledges relevant failure cases, it does not quantify their prevalence or establish a reliable relationship between DSE and harmful updates. How frequently do these cases occur during training, and how much do they affect the method’s effectiveness?

  2. The optimization conditioning differs from the original NoisyRollout formulation, leaving the source of improvement unclear. The original NoisyRollout generates responses from both clean and noisy images but conditions all policy updates on the clean image. In contrast, Eq. (5) conditions each response on its generating observation, so noisy responses are optimized under the noisy image. Consequently, removing the entropy weights from the proposed objective does not automatically recover the original NoisyRollout formulation. The authors should clarify which conditioning is used in the reported NoisyRollout baseline and the zero-weight ablation, and disentangle this design choice from the effect of DSE. Otherwise, the reported improvements could reflect both a change in the training objective and differential entropy weighting.

  3. The theoretical analysis does not establish the claimed benefits. The bias identity, attenuation bounds, and local entropy-change bound do not demonstrate that DSE reduces gradient bias or variance, or preserves exploration. These results primarily characterize general properties of bounded reweighting without establishing the crucial connection between DSE and the advertised benefits. The manuscript itself explicitly acknowledges these limitations.

  4. Evidence that the differential term generalizes across settings is limited. The ablation isolating the differential component is conducted primarily on Qwen2.5-VL-7B trained on Geometry3K. The additional experiments broaden the evaluation, but do not establish that this component consistently accounts for the gains across models and training datasets. In particular, the Qwen3-VL experiment compares the proposed method only against the base model. This cannot separate the contribution of the differential term from that of RL fine-tuning or the existing noisy-rollout and absolute-entropy components, nor establish an advantage over matched GRPO, NoisyRollout, or absolute-only training.

Questions for authors

See Weaknesses.

Reviewer_TJFiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces DSE-GRPO, an improved reinforcement learning method for Vision-Language Models that addresses the issue of "semantic drift" caused by noisy training images. By measuring Differential Semantic Entropy (DSE) between clean and noisy rollouts, the framework dynamically down-weights harmful, high-variance noisy trajectories while preserving beneficial exploration. This approach effectively reduces off-policy bias, smooths optimization dynamics, and significantly improves out-of-domain accuracy over standard GRPO baselines.

Critical strengths

Targeted Noise Filtering: It elegantly solves the off-policy mismatch problem by identifying exactly when image noise changes the underlying semantics, filtering out harmful gradients while retaining beneficial, exploratory noise.

Strong Empirical Gains: It demonstrates significant real-world improvements on out-of-domain benchmarks (boosting accuracy by up to 8.0 percentage points on state-of-the-art VLMs like Qwen2.5-VL-7B).

Critical weaknesses

Complexity in Open-Ended Tasks: While highly effective for tasks with clear semantic boundaries (like VQA or geometry), calculating precise semantic entropy may become highly complex or unreliable in long-form, open-ended VLM generation tasks where responses are harder to cluster.

Questions for authors

na

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.