- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies entropy control for reinforcement learning on heterogeneous task mixtures and proposes Group Entropy-Controlled Policy Optimization (GEPO), an extension of GRPO. It motivates the method through observations of different entropy regimes across prompt groups and an analysis of discrepancies between policy-weighted and reference-normalized advantages. GEPO estimates each prompt group's entropy from existing rollout responses and uses it to condition advantage shaping. Specifically, it attenuates positive advantages in low-entropy groups and negative advantages in high-entropy groups, with different scaling strengths. The entropy thresholds are derived from batch statistics and smoothed using exponential moving averages. Experiments on Intern-S1-mini and Qwen3.5-9B across thirteen benchmarks report higher average scores than the evaluated baselines, together with more stable training and differentiated entropy regimes across tasks.
Critical strengths
-
GEPO offers a simple and practical way to address heterogeneous exploration needs at the prompt-group level. It reuses existing rollout responses and requires neither explicit task labels nor additional sampling.
-
The evaluation covers two base models and thirteen benchmarks across diverse domains. The average improvement on Intern-S1-mini is encouraging (54.2 versus 51.8 for the strongest baselines), and the reported training curves provide additional evidence of improved stability.
Critical weaknesses
-
The connection between the theoretical analysis and the proposed intervention needs further support. Propositions 3.1–3.2 provide upper bounds on discrepancies relative to a reference distribution, but a larger upper bound does not establish greater realized bias or stronger gradient contributions from low-entropy groups. These results also do not show that the sign-dependent rescaling in Equation (7) reduces the identified discrepancy. A more direct analysis or controlled empirical measurement of gradient contributions and bias before and after shaping would strengthen this central claim.
-
The ablations do not sufficiently isolate the contribution of group-level entropy control. Table 2 removes low-entropy control, high-entropy control, or asymmetric shaping, but does not compare group-level and global entropy signals under the same shaping rule and threshold update procedure. Moreover, the average gain over Clip-Cov on Qwen3.5-9B is relatively small (71.2 versus 70.9), with no reported variability across independent training runs. A matched group-versus-global comparison and repeated runs would help clarify which component drives the improvements and how reliable the gains are.
Questions for authors
-
Please check and correct the inconsistent GEPO entries in Tables 1 and 2: MathVista is reported as 71.0 versus 71.2, and MMMU-Pro as 59.8 versus 55.8. Please also verify the reported average of 54.2, as the entries in Table 2 average approximately 53.9.
-
Please proofread the manuscript for minor grammatical, punctuation, and formatting errors, such as “Qwen3,5-9B” (p. 7, line 377; should be “Qwen3.5-9B”).
-
To complement Figure 1, please consider adding an end-to-end schematic linking grouped rollouts, group entropy estimation, adaptive threshold updates, asymmetric advantage shaping, and the policy update.