- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Entropic Advantage Policy Optimization (EAPO), a token-level advantage reweighting method for reinforcement learning with verifiable rewards. EAPO combines normalized policy entropy with the sign of the response-level advantage, emphasizing high-entropy positions for positive advantages and low-entropy positions for negative advantages. The positive weights preserve the advantage sign and its within-response mean. The method is motivated by continuation-resampling observations and supported by theoretical interpretations under simplified assumptions. Experiments report improvements across four base and reasoning backbones on mathematical benchmarks, alongside transfer to non-mathematical reasoning tasks.
Critical strengths
EAPO uses policy entropy and existing response-level advantages to derive token-level learning signals without requiring auxiliary reward models, privileged teacher information, or additional training-time branching rollouts. Its weighting mechanism preserves advantage signs and within-response means, while the concentration parameter explicitly bounds relative token weights. These properties provide a controlled and interpretable way to redistribute feedback, making the method straightforward to integrate into existing RLVR pipelines.
EAPO achieves the highest macro-averaged avg@32 and pass@32 across all four evaluated base and reasoning backbones. On Qwen3-4B-Base, its macro-averaged accuracy exceeds the strongest baseline, RLRT, by approximately 3.8 percentage points. The consistency of the overall improvements across different starting models supports the empirical usefulness of the method.
The improvements extend beyond the mathematical tasks used for training. On eight Reasoning Gym tasks, EAPO improves macro-averaged accuracy over the strongest respective baselines by 1.46 and 3.02 percentage points for the two base backbones. This provides evidence that the benefits transfer to other reasoning tasks rather than being confined to the mathematical evaluation setting.
The motivating resampling experiment provides a useful empirical basis for the proposed asymmetry. The authors compare high- and low-entropy windows within individual responses, constrain their relative positions, and observe consistent trends across two models. These controls make the motivation more informative than an argument based solely on intuition and offer an interesting observation about the relationship between uncertainty, reproducible success, and recovery from failure.
Critical weaknesses
The empirical effectiveness of the allocation rule is better established than its proposed mechanism. Low recovery accuracy from a low-entropy window may reflect an error already present in the prefix rather than a failure attributable to the decisions being reweighted. Although the paper explicitly avoids equating entropy with token correctness, the resampling results do not fully explain why concentrating penalties at these positions corrects repeated failure. The theoretical analyses provide interpretations under restricted assumptions: Proposition 1 adopts the signed-entropy utility, Proposition 2 considers a single direct logit update, and Appendix E.3 assumes feedback identifying the correctness of a local binary decision. Furthermore, higher final-answer diversity and epistemic-marker frequency can reflect diverse incorrect outputs or stylistic changes rather than productive exploration. Together, these limitations leave the specific explanation of correcting recurrent failures while preserving useful alternatives insufficiently validated, without negating the reported performance gains.
The stability of some improvements over the strongest baselines remains unclear. On three of the four backbones, the macro-averaged accuracy gains are approximately 0.93, 1.08, and 0.58 percentage points. Appendix D.6 includes three-seed training-reward curves, but does not report corresponding test-set variability or include RLRT and 80/20 in that comparison. Consequently, it remains difficult to assess how reliably these smaller gains persist across training runs.
Appendix C.1 does not explicitly establish the decoding settings, random-seed handling, and continuation-length or stopping rules used for the initial generations and resampled continuations, nor clearly state whether Table 8 applies. These settings directly influence both common-prefix overlap and recovery accuracy. Clarifying their relationship to the reported training configuration is therefore important for reproducing and interpreting the motivating experiment.
Questions for authors
What evidence distinguishes low-entropy decisions that contribute to failure from decisions that merely continue an already incorrect prefix? Is there existing local-intervention or step-level analysis supporting the proposed mechanism? If not, could the discussion more explicitly distinguish the demonstrated effectiveness of the reweighting rule from the hypothesized explanation involving recurrent failure and useful alternatives?
Are test-set results available for the multiple training seeds reported in Appendix D.6? Could you provide their means and variability, together with comparisons against the strongest baselines where available, particularly RLRT and 80/20, to assess the stability of the smaller reported gains?
Appendix B discusses distinctions from GEPO, LESS, and A3PO, but these methods are not directly evaluated in the main experiments. Could you explain their omission and indicate whether matched comparisons are available to assess the empirical value of EAPO’s specific formulation? In particular, the surprisal substitution in Appendix D.1 tests a variant of EAPO rather than the complete A3PO method.
What decoding configurations were used for the initial 32 generations and the 64 continuations per selected prefix in Appendix C.1? Please specify the sampling mode, temperature, top-p/top-k, random-seed handling, and continuation-length and stopping rules, and clarify whether Table 8 applies to both stages. Were identical settings used for high- and low-entropy windows?
6 comments
Response to Reviewer cd2D (2/3)
W1-3. Final-answer diversity and epistemic markers as evidence of exploration
We appreciate the comment on final-answer diversity and epistemic markers as evidence of exploration. Beyond these proxies, we directly measure productive exploration with strategy coverage, where EAPO discovers and preserves more distinct successful strategies than all baselines. Specifically, we utilize LLM judges to summarize the solution strategy of each response and cluster them by strategy (using GPT-6-Luna and GPT-6.1-Sol, respectively). Strategy coverage@k is the fraction of all correct strategies found by any method that a method recovers with correct responses within samples, so it cannot be inflated by diverse incorrect outputs.
Table C.2. Strategy coverage@k (%) on Qwen3-4B-Base (full curve in Figure 4)
| k | Initial | GRPO | EntropyAdv | HAPO | 80/20 | RLRT | EAPO |
|---|---|---|---|---|---|---|---|
| 1 | 8.6 | 12.8 | 12.7 | 12.3 | 11.1 | 16.3 | 21.6 |
| 8 | 24.1 | 35.0 | 33.4 | 30.0 | 26.6 | 39.3 | 45.5 |
| 32 | 39.6 | 51.6 | 53.3 | 45.4 | 44.5 | 58.9 | 62.2 |
As shown in Table C.2, EAPO recovers the most successful strategies at every budget, indicating that its exploration gains come from a wider range of successful strategies rather than diverse incorrect outputs or stylistic changes. We have incorporated this analysis into the revision, with full results and experimental details provided in Sections 5.4 and C.4.
Final-answer diversity and epistemic markers serve as complementary proxies to these direct measures of pass@k and strategy coverage. Final-answer diversity captures whether the policy avoids converging to the same incorrect answer on challenging problems, even within the same strategy, while epistemic markers have been shown to be an effective measure of exploratory reasoning in [1].
[1] Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty, Kim et al., arXiv 2026
W2 & Q2. Held-out variability across training runs
We thank the reviewer for the suggestion to assess how reliably the gains persist. The gains of EAPO remain robust on held-out benchmarks across multiple training runs, and we have extended the training curves across seeds to include all baselines. Specifically, we evaluated the final checkpoints of three training runs per method on Qwen3-4B-Base on the held-out benchmarks.
Table C.3. Held-out results across three training runs on Qwen3-4B-Base (avg@32, mean ± std)
| Method | AIME24 | AIME25 | AIME26 | HMMT26 |
|---|---|---|---|---|
| GRPO | 12.8 ± 0.7 | 12.7 ± 0.7 | 9.2 ± 1.7 | 5.9 ± 0.9 |
| EntropyAdv | 12.3 ± 0.8 | 10.5 ± 1.2 | 8.2 ± 0.4 | 5.1 ± 1.1 |
| HAPO | 13.1 ± 0.8 | 11.6 ± 0.8 | 8.6 ± 0.7 | 6.5 ± 0.2 |
| 80/20 | 11.4 ± 1.1 | 12.9 ± 0.5 | 6.3 ± 0.3 | 5.9 ± 0.9 |
| RLRT | 13.2 ± 0.7 | 14.0 ± 1.0 | 13.9 ± 0.7 | 7.7 ± 0.7 |
| EAPO | 20.2 ± 0.7 | 17.4 ± 0.8 | 16.2 ± 1.4 | 11.4 ± 0.6 |
As shown in Table C.3, the avg@32 gaps between EAPO and the strongest baseline, RLRT, exceed the standard deviations of both methods on all four benchmarks, confirming that its gains are consistent across training runs. We further re-evaluated all checkpoints in Table 2 under three evaluation seeds, where EAPO achieves the best mean avg@32 on most benchmarks across all four backbones (Table 12). Both analyses are included in Section D.6.
Response to Reviewer cd2D (3/3)
Q3. Comparison with GEPO, LESS, and A3PO
We appreciate the suggestion to evaluate these related methods directly. A3PO could not be reproduced under our setting, and its surprisal-based signal consistently led to reward collapse, whereas our entropy-based allocation remains stable. Using the released code and the original settings, including its decay schedule, while matching only the hyperparameters shared by all methods (e.g., LoRA and batch size), we could not reproduce the reported results, and thus did not include A3PO in the main paper. Replacing the entropy term of EAPO with the sampled-token surprisal used by A3PO also led to a collapse of the training reward (Section D.1), as did our runs reproducing A3PO.
GEPO and LESS, on the other hand, differ from EAPO in the granularity of credit assignment, as GEPO adjusts advantages per response group and LESS per low-entropy segment rather than redistributing credit across individual tokens, and thus were not included in the main experiments. We additionally provide their results in Table C.4, trained under the same configuration as all other methods.
Table C.4. Comparison with GEPO and LESS on Qwen3-4B-Base (avg@32 / pass@32)
| Method | AIME24 | AIME25 | AIME26 | HMMT26 |
|---|---|---|---|---|
| GEPO | 12.8 / 36.7 | 11.3 / 36.7 | 8.4 / 30.0 | 6.9 / 27.3 |
| LESS | 14.6 / 43.3 | 12.9 / 40.0 | 10.9 / 36.7 | 8.1 / 30.3 |
| EAPO | 20.2 / 53.3 | 17.1 / 46.7 | 15.7 / 50.0 | 13.4 / 33.3 |
As shown in Table C.4, EAPO surpasses GEPO and LESS on all four benchmarks with large margins in avg@32, suggesting that redistributing credit across individual tokens, toward uncertain decisions under success and confident decisions under failure, is central to its gains.
W3 & Q4. Decoding configuration of the resampling experiment
Thank you for the careful attention to the details of the resampling protocol. Both the initial generations and the resampled continuations in Section C.1 use the sampling settings of training rollouts (Table 8), with identical settings for high- and low-entropy windows. We have reflected these settings in Section C.1 of the revision, as summarized in Table C.5.
Table C.5. Decoding configuration of the resampling experiment
| Stage | Samples | Temperature / top-p | Top-k | Maximum length | Stopping rule | Random seed |
|---|---|---|---|---|---|---|
| Initial generations | 32 per problem | 1.0 / 1.0 | Disabled | 10,240 tokens | EOS or maximum length | 42 |
| Continuations | 64 per prefix | 1.0 / 1.0 | Disabled | 10,240 tokens including the prefix | EOS or maximum length | 42 |
Response to Reviewer cd2D (1/3)
We sincerely thank you for your detailed and constructive review. We appreciate your recognition that EAPO derives token-level learning signals from policy entropy and existing response-level advantages without auxiliary reward models, privileged teacher information, or additional branching rollouts. We are also grateful for your acknowledgement that EAPO achieves the highest avg@32 and pass@32 across all backbones, that its benefits transfer to other reasoning tasks, and that the resampling experiment provides a useful empirical basis for the proposed asymmetry. We have carefully reviewed your comments and addressed them in detail below.
We appreciate this insightful question. EAPO is not intended to penalize only the decision at which an error first arises, but confident continuations from failing states, which our local intervention shows remain recoverable. Whether the error arises at these decisions or earlier in the prefix, continuing confidently along the original path is worse than the alternatives still available, which is what matters for credit assignment. As resampling alone (
Table 1) cannot tell whether such a prefix is already irrecoverable, we insert a short reflection cue (e.g., "Wait, let me reconsider this step.") at low-entropy windows of newly drawn failed responses followingSection C.1, and compare 64 continuations from the same prefix with and without the cue.Table C.1. Accuracy of continuations from low-entropy windows of failed responses, with and without a reflection cue
As shown in
Table C.1, the cue significantly improves accuracy on both models, showing that a correct continuation remains reachable from these prefixes once the policy departs from its original confident behavior. Without the cue, however, the policy largely regenerates the same path and rarely recovers (Table 1), which is precisely the confident continuation on which EAPO concentrates penalties in failed responses.Thank you for the careful reading of our theoretical analyses. The theoretical analyses are intended to provide a rationale for each component of EAPO, while its mechanism is examined empirically.
Proposition 1shows that the objective of EAPO comes from transferring the KL anchoring of RLVR from policy space to token space, favoring signed uncertainty while anchoring credit to uniform allocation, with κ controlling the strength of this anchor.Proposition 2provides the rationale for attenuating penalties at positions where alternatives remain, as stronger penalties sharpen the distribution over these alternatives, andSection E.3analyzes the effectiveness of our sign–entropy coupling from an information-theoretic perspective, showing that success conveys more information at uncertain decisions and failure at confident ones. To this end, EAPO integrates these components into a single allocation rule, which achieves the highest macro-averaged avg@32 and pass@32 across all four backbones (Table 2), while its sign–entropy coupling is empirically proven to be effective through the ablations inTables 5and9.