acceptodds
Withdrawn from ICLR 2027

Group Entropy-Controlled Policy Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Entropy control has become an effective tool in reinforcement learning (RL) of large language models (LLMs), helping balance exploration-exploitation trade-off during alignment process. Such RL paradigm is often conducted on mixtures of heterogeneous tasks, which induce distinct entropy regimes under the same policy, making global or token-level entropy regulation insufficient to corresponding heterogeneous needs of exploration. This heterogeneity further makes GRPO-style normalized advantages induce an entropy-dependent bias, making advantage signals across prompt groups statistically non-comparable. To address this issue, we propose Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy, estimated from existing grouped samples to perform entropy-conditioned asymmetric advantage shaping. GEPO attenuates positive advantages in low-entropy groups to reduce over-exploitation, and negative advantages in high-entropy groups to preserve exploration, with adaptive thresholds derived from historical entropy statistics. Extensive experiments on two base models across thirteen benchmarks spanning mathematics, physics, science, code generation, and instruction following show that GEPO consistently outperforms GRPO and recent entropy-controlled methods, delivering balanced cross-task improvements while preserving task-specific exploration levels throughout training.

Primary area:
reinforcement learning
Keywords:
large language model, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vUpwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies entropy control for reinforcement learning on heterogeneous task mixtures and proposes Group Entropy-Controlled Policy Optimization (GEPO), an extension of GRPO. It motivates the method through observations of different entropy regimes across prompt groups and an analysis of discrepancies between policy-weighted and reference-normalized advantages. GEPO estimates each prompt group's entropy from existing rollout responses and uses it to condition advantage shaping. Specifically, it attenuates positive advantages in low-entropy groups and negative advantages in high-entropy groups, with different scaling strengths. The entropy thresholds are derived from batch statistics and smoothed using exponential moving averages. Experiments on Intern-S1-mini and Qwen3.5-9B across thirteen benchmarks report higher average scores than the evaluated baselines, together with more stable training and differentiated entropy regimes across tasks.

Critical strengths
  1. GEPO offers a simple and practical way to address heterogeneous exploration needs at the prompt-group level. It reuses existing rollout responses and requires neither explicit task labels nor additional sampling.

  2. The evaluation covers two base models and thirteen benchmarks across diverse domains. The average improvement on Intern-S1-mini is encouraging (54.2 versus 51.8 for the strongest baselines), and the reported training curves provide additional evidence of improved stability.

Critical weaknesses
  1. The connection between the theoretical analysis and the proposed intervention needs further support. Propositions 3.1–3.2 provide upper bounds on discrepancies relative to a reference distribution, but a larger upper bound does not establish greater realized bias or stronger gradient contributions from low-entropy groups. These results also do not show that the sign-dependent rescaling in Equation (7) reduces the identified discrepancy. A more direct analysis or controlled empirical measurement of gradient contributions and bias before and after shaping would strengthen this central claim.

  2. The ablations do not sufficiently isolate the contribution of group-level entropy control. Table 2 removes low-entropy control, high-entropy control, or asymmetric shaping, but does not compare group-level and global entropy signals under the same shaping rule and threshold update procedure. Moreover, the average gain over Clip-Cov on Qwen3.5-9B is relatively small (71.2 versus 70.9), with no reported variability across independent training runs. A matched group-versus-global comparison and repeated runs would help clarify which component drives the improvements and how reliable the gains are.

Questions for authors
  1. Please check and correct the inconsistent GEPO entries in Tables 1 and 2: MathVista is reported as 71.0 versus 71.2, and MMMU-Pro as 59.8 versus 55.8. Please also verify the reported average of 54.2, as the entries in Table 2 average approximately 53.9.

  2. Please proofread the manuscript for minor grammatical, punctuation, and formatting errors, such as “Qwen3,5-9B” (p. 7, line 377; should be “Qwen3.5-9B”).

  3. To complement Figure 1, please consider adding an end-to-end schematic linking grouped rollouts, group entropy estimation, adaptive threshold updates, asymmetric advantage shaping, and the policy update.

Reviewer_bZszOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GEPO, an extension of GRPO for training on heterogeneous task mixtures. It estimates prompt-group entropy from sampled response log-probabilities and attenuates positive advantages in low-entropy groups and negative advantages in high-entropy groups. Entropy thresholds track batch statistics through exponential moving averages.

Critical strengths

A simple intervention with low integration cost. Equation (7) uses information already available from grouped rollouts and requires no additional response sampling or critic. A reliable improvement from such a small change would be practically consequential for existing GRPO training systems.

Evaluation across heterogeneous tasks and two backbones. The benchmark coverage is appropriate for the stated multi-task motivation. In particular, the reported Intern-S1-mini average improves from 51.5 with GRPO to 54.2 with GEPO, and the component ablations attempt to separate the two entropy regimes. These results provide a useful initial empirical signal.

Critical weaknesses

The theoretical analysis does not establish the optimization pathology or justify the proposed correction. Propositions 3.1–3.2 bound discrepancies relative to a reference distribution, with the entropy-dependent form relying on a uniform distribution over the response space. The paper does not establish why advantages standardized under that reference are the appropriate target for on-policy reward optimization. Moreover, an upper bound that becomes looser at lower entropy does not establish that low-entropy groups actually have greater distortion or dominate the batch gradient, as asserted in the “Optimization consequences” paragraph of Section 3.2. Observation 2 explicitly acknowledges that its advantage asymmetry arises mechanically from group accuracy. The experiments do not isolate an additional effect of entropy after accounting for accuracy. Finally, the propositions concern sequence entropy, whereas the intervention uses a length-normalized estimate, and no result shows that the asymmetric scaling in Equation (7) reduces the stated discrepancy or produces the claimed entropy change. Consequently, the main mechanistic explanation remains a hypothesis rather than a demonstrated basis for the method.

The experimental evidence is insufficiently specified and internally inconsistent for the claimed robustness. Section 4.1 uses an in-house training dataset without reporting its size, task proportions, or task-specific reward construction, and omits important details such as the learning rate and checkpoint-selection rule. No variation across independent training seeds is reported. Clip-Cov and KL-Cov inherit parameters from Qwen2.5-7B, leaving their tuning suitability for the evaluated backbones unclear. These omissions matter when the Qwen3.5-9B advantage over Clip-Cov is only 71.2 versus 70.9 and baseline collapse is a central argument. There is also a concrete inconsistency: the full GEPO row reports Intern-S1-mini MMMU-Pro as 59.8 in Table 1 and 55.8 in Table 2, while both report an average of 54.2. The thirteen entries in the Table 2 row average approximately 53.86. Corrected results and a reproducible comparison protocol are necessary to determine the size and reliability of the gain.

Questions for authors

What optimization objective makes the uniform-reference normalized advantage the desired target?

Does the benefit persist against accuracy-conditioned scaling and entropy-independent asymmetric scaling with comparable intervention frequency?

Please reconcile the full GEPO rows in Tables 1 and 2 and their averages, specify the training-data/reward and checkpoint-selection protocols, and state the number of independent training runs and baseline-tuning budgets. What uncertainty accompanies the 0.3-point Qwen3.5-9B improvement over Clip-Cov?

Reviewer_KYMgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GEPO, a lightweight extension to GRPO for multi-task LLM reinforcement learning. It estimates entropy from responses sampled for each prompt and uses adaptive thresholds to attenuate positive advantages in low-entropy groups and negative advantages in high-entropy groups. Experiments on two base models across 13 benchmarks show improvements in average performance and training stability.

Critical strengths
  1. The method is simple and readily applicable to existing GRPO pipelines, reusing rollout samples and log probabilities without additional sampling.
  2. The evaluation covers diverse task domains, and training curves and component ablations provide evidence of its potential benefits.
Critical weaknesses
  1. Prompt-specific entropy-conditioned updates have already been explored in SEED-GRPO, while entropy-based differential treatment of positive and negative samples also appears in related work such as GTPO / GRPO-S. These ideas alone should therefore not be presented as the main novelty. GEPO primarily differs in its entropy estimator, advantage-scaling rules, and threshold design. These closely related methods should serve as primary baselines to establish the benefits of these design choices. Clip-Cov, AEPO, and other existing baselines provide useful performance references but are less informative for assessing the incremental contribution over the closest prior work.

  2. The theoretical analysis concerns sequence-level Shannon entropy, whereas the algorithm uses length-normalized sampled negative log-likelihood. These are different quantities and need not induce the same ordering of prompts. Moreover, length normalization does not automatically preserve the unbiasedness of the unnormalized estimator. The paper should clarify how the theoretical bounds support the entropy measure actually used for control.

  3. Clip-Cov and KL-Cov reuse hyperparameters originally selected for Qwen2.5-7B, despite being evaluated on different base models here. The curves show substantial entropy growth for these baselines and late-stage performance collapse in some cases. Have the authors tuned them for the current models and training setup? Without such validation, it is difficult to distinguish methodological limitations from poorly transferred hyperparameters.

  4. Some gains are small—for example, the average improvement over Clip-Cov on Qwen3.5-9B is only 0.3 points—but no multi-seed results or confidence intervals are reported. Additionally, the text describes late-stage collapse for some baselines, while the main table still reports strong scores. The authors should clarify whether these results use final or best checkpoints, whether all benchmarks are evaluated using the same checkpoint, and whether the table and training curves follow identical evaluation protocols.

Questions for authors

Why do the full GEPO results differ between Tables 1 and 2? MathVista is reported as 71.0 versus 71.2, and MMMU-Pro as 59.8 versus 55.8, while both tables report an average of 54.2. The values listed in Table 2 instead average to approximately 53.9. Are these discrepancies typographical errors or results from different runs?

The experiments use a batch size of 256. Larger batches may contain more heterogeneous prompts, and the entropy thresholds explicitly depend on within-batch statistics. Have the authors evaluated different batch sizes to determine whether the performance and stability gains remain consistent?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.