- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Differential Semantic Entropy GRPO (DSE-GRPO) to address the potential off-policy mismatch introduced by noisy rollouts in reinforcement learning post-training for vision-language models. The proposed method measures the difference in semantic entropy between clean and noisy rollout groups and uses this signal to adaptively down-weight the advantages of noisy responses. The authors provide theoretical analyses of the effects of advantage reweighting on optimization bias, update magnitude, and policy entropy. Experimental results on several vision-language benchmarks show improvements over vanilla GRPO and NoisyRollout baselines.
Critical strengths
-
This paper identifies a meaningful problem in noisy-rollout RL training for VLMs, where image perturbations may introduce unreliable learning signals and affect optimization.
-
The proposed method is simple to implement and introduces negligible computational overhead compared with standard noisy-rollout training.
-
The experiments cover multiple model scales and benchmarks, with ablation studies demonstrating the empirical benefits of the proposed differential entropy term.
Critical weaknesses
-
Insufficient motivation and methodological justification. The paper lacks a systematic analysis of how noisy rollouts lead to harmful policy updates. More importantly, the connection between the identified off-policy mismatch and the proposed differential semantic entropy is not sufficiently established.
-
Questionable effectiveness of DSE as a measure of semantic drift. DSE only measures increases in answer entropy rather than the actual discrepancy between clean and noisy answer distributions. Substantial semantic shifts can occur with zero DSE, while increased entropy may also reflect beneficial exploration.
-
Limited theoretical contribution. The theoretical analyses mainly characterize general properties of bounded advantage reweighting. They do not establish that DSE-based weighting reduces gradient bias or variance, nor do they explain why this particular uncertainty signal should improve optimization.
-
Insufficient justification for the algorithm design. The paper directly introduces a multiplicative linear weighting function without adequately explaining its formulation or why it is preferable to alternative weighting strategies. The methodological novelty over existing semantic-entropy-based GRPO methods appears relatively limited.
-
Insufficient empirical validation of the proposed mechanism. The experiments primarily demonstrate improvements in downstream accuracy without directly verifying the relationship between DSE and gradient reliability. Additionally, the reported gains lack multi-seed statistical analysis, making their robustness difficult to assess.
Questions for authors
-
Can the authors provide a more detailed analysis of how noisy rollouts introduce gradient bias and explain why differential semantic entropy is an appropriate proxy for identifying harmful updates?
-
Since DSE cannot capture semantic shifts that preserve answer entropy, why not directly measure the discrepancy between clean and noisy answer distributions, such as through Jensen-Shannon divergence?
-
Can the authors provide empirical evidence showing that larger DSE values correspond to less reliable policy gradients, for example through gradient alignment or bias measurements?
-
What motivates the specific multiplicative linear advantage-weighting function? Can the authors compare it with alternative weighting functions or uncertainty-based selection strategies?
-
Can the authors provide multi-seed experiments with variance estimates or confidence intervals to demonstrate the statistical robustness of the reported improvements?