acceptodds
Under review as a conference paper at ICLR 2027

Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning

Abstract

Reinforcement learning with verifiable rewards (RLVR) enhances reasoning in large language models (LLMs) through outcome-level feedback, yet recent approaches to finer-grained credit assignment often require auxiliary models, additional sampling, or privileged information. Although policy entropy provides a readily available signal, prioritizing uncertain positions under both reinforcement and penalization concentrates penalties where failed responses still retain alternatives for recovery, which can suppress opportunities for exploration. To address this, we introduce Entropic Advantage Policy Optimization (EAPO), an entropy-guided credit assignment method that treats success and failure asymmetrically. Specifically, motivated by the observation that success under uncertainty is less repeatable while confident failures tend to recur, EAPO couples normalized policy entropy with the sign of the response advantage to reinforce surprising success and correct repeated failure. It assigns stronger reinforcement to high-entropy decisions in successful responses and stronger penalties to low-entropy decisions in failed responses, while attenuating penalties at uncertain positions to preserve opportunities for recovery. By redistributing the response advantage across tokens, EAPO derives token-level credit directly from existing rollout signals without additional supervision. We validate EAPO on a range of reasoning tasks across both base and reasoning backbones, demonstrating that it achieves the best overall performance. We further show that EAPO promotes more effective exploration, broadening problem coverage and generating more diverse candidate answers.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning with Verifiable Rewards, Token-Level Credit Assignment, Exploration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cd2DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Entropic Advantage Policy Optimization (EAPO), a token-level advantage reweighting method for reinforcement learning with verifiable rewards. EAPO combines normalized policy entropy with the sign of the response-level advantage, emphasizing high-entropy positions for positive advantages and low-entropy positions for negative advantages. The positive weights preserve the advantage sign and its within-response mean. The method is motivated by continuation-resampling observations and supported by theoretical interpretations under simplified assumptions. Experiments report improvements across four base and reasoning backbones on mathematical benchmarks, alongside transfer to non-mathematical reasoning tasks.

Critical strengths

EAPO uses policy entropy and existing response-level advantages to derive token-level learning signals without requiring auxiliary reward models, privileged teacher information, or additional training-time branching rollouts. Its weighting mechanism preserves advantage signs and within-response means, while the concentration parameter explicitly bounds relative token weights. These properties provide a controlled and interpretable way to redistribute feedback, making the method straightforward to integrate into existing RLVR pipelines.

EAPO achieves the highest macro-averaged avg@32 and pass@32 across all four evaluated base and reasoning backbones. On Qwen3-4B-Base, its macro-averaged accuracy exceeds the strongest baseline, RLRT, by approximately 3.8 percentage points. The consistency of the overall improvements across different starting models supports the empirical usefulness of the method.

The improvements extend beyond the mathematical tasks used for training. On eight Reasoning Gym tasks, EAPO improves macro-averaged accuracy over the strongest respective baselines by 1.46 and 3.02 percentage points for the two base backbones. This provides evidence that the benefits transfer to other reasoning tasks rather than being confined to the mathematical evaluation setting.

The motivating resampling experiment provides a useful empirical basis for the proposed asymmetry. The authors compare high- and low-entropy windows within individual responses, constrain their relative positions, and observe consistent trends across two models. These controls make the motivation more informative than an argument based solely on intuition and offer an interesting observation about the relationship between uncertainty, reproducible success, and recovery from failure.

Critical weaknesses

The empirical effectiveness of the allocation rule is better established than its proposed mechanism. Low recovery accuracy from a low-entropy window may reflect an error already present in the prefix rather than a failure attributable to the decisions being reweighted. Although the paper explicitly avoids equating entropy with token correctness, the resampling results do not fully explain why concentrating penalties at these positions corrects repeated failure. The theoretical analyses provide interpretations under restricted assumptions: Proposition 1 adopts the signed-entropy utility, Proposition 2 considers a single direct logit update, and Appendix E.3 assumes feedback identifying the correctness of a local binary decision. Furthermore, higher final-answer diversity and epistemic-marker frequency can reflect diverse incorrect outputs or stylistic changes rather than productive exploration. Together, these limitations leave the specific explanation of correcting recurrent failures while preserving useful alternatives insufficiently validated, without negating the reported performance gains.

The stability of some improvements over the strongest baselines remains unclear. On three of the four backbones, the macro-averaged accuracy gains are approximately 0.93, 1.08, and 0.58 percentage points. Appendix D.6 includes three-seed training-reward curves, but does not report corresponding test-set variability or include RLRT and 80/20 in that comparison. Consequently, it remains difficult to assess how reliably these smaller gains persist across training runs.

Appendix C.1 does not explicitly establish the decoding settings, random-seed handling, and continuation-length or stopping rules used for the initial generations and resampled continuations, nor clearly state whether Table 8 applies. These settings directly influence both common-prefix overlap and recovery accuracy. Clarifying their relationship to the reported training configuration is therefore important for reproducing and interpreting the motivating experiment.

Questions for authors

What evidence distinguishes low-entropy decisions that contribute to failure from decisions that merely continue an already incorrect prefix? Is there existing local-intervention or step-level analysis supporting the proposed mechanism? If not, could the discussion more explicitly distinguish the demonstrated effectiveness of the reweighting rule from the hypothesized explanation involving recurrent failure and useful alternatives?

Are test-set results available for the multiple training seeds reported in Appendix D.6? Could you provide their means and variability, together with comparisons against the strongest baselines where available, particularly RLRT and 80/20, to assess the stability of the smaller reported gains?

Appendix B discusses distinctions from GEPO, LESS, and A3PO, but these methods are not directly evaluated in the main experiments. Could you explain their omission and indicate whether matched comparisons are available to assess the empirical value of EAPO’s specific formulation? In particular, the surprisal substitution in Appendix D.1 tests a variant of EAPO rather than the complete A3PO method.

What decoding configurations were used for the initial 32 generations and the 64 continuations per selected prefix in Appendix C.1? Please specify the sampling mode, temperature, top-p/top-k, random-seed handling, and continuation-length and stopping rules, and clarify whether Table 8 applies to both stages. Were identical settings used for high- and low-entropy windows?

6 comments
AuthorsOpenReview ↗

Response to Reviewer cd2D (1/3)

We sincerely thank you for your detailed and constructive review. We appreciate your recognition that EAPO derives token-level learning signals from policy entropy and existing response-level advantages without auxiliary reward models, privileged teacher information, or additional branching rollouts. We are also grateful for your acknowledgement that EAPO achieves the highest avg@32 and pass@32 across all backbones, that its benefits transfer to other reasoning tasks, and that the resampling experiment provides a useful empirical basis for the proposed asymmetry. We have carefully reviewed your comments and addressed them in detail below.


W1-1 & Q1. Low-entropy decisions that contribute to failure vs. continuation of an incorrect prefix

We appreciate this insightful question. EAPO is not intended to penalize only the decision at which an error first arises, but confident continuations from failing states, which our local intervention shows remain recoverable. Whether the error arises at these decisions or earlier in the prefix, continuing confidently along the original path is worse than the alternatives still available, which is what matters for credit assignment. As resampling alone (Table 1) cannot tell whether such a prefix is already irrecoverable, we insert a short reflection cue (e.g., "Wait, let me reconsider this step.") at low-entropy windows of newly drawn failed responses following Section C.1, and compare 64 continuations from the same prefix with and without the cue.

Table C.1. Accuracy of continuations from low-entropy windows of failed responses, with and without a reflection cue

ModelResampled (%)Intervened (%)Δ\Delta (pp)
Qwen3-4B-Base15.925.8+10.0
Qwen3-8B-Base10.517.4+6.9

As shown in Table C.1, the cue significantly improves accuracy on both models, showing that a correct continuation remains reachable from these prefixes once the policy departs from its original confident behavior. Without the cue, however, the policy largely regenerates the same path and rarely recovers (Table 1), which is precisely the confident continuation on which EAPO concentrates penalties in failed responses.


W1-2. Role of the theoretical analyses

Thank you for the careful reading of our theoretical analyses. The theoretical analyses are intended to provide a rationale for each component of EAPO, while its mechanism is examined empirically. Proposition 1 shows that the objective of EAPO comes from transferring the KL anchoring of RLVR from policy space to token space, favoring signed uncertainty while anchoring credit to uniform allocation, with κ\kappa controlling the strength of this anchor. Proposition 2 provides the rationale for attenuating penalties at positions where alternatives remain, as stronger penalties sharpen the distribution over these alternatives, and Section E.3 analyzes the effectiveness of our sign–entropy coupling from an information-theoretic perspective, showing that success conveys more information at uncertain decisions and failure at confident ones. To this end, EAPO integrates these components into a single allocation rule, which achieves the highest macro-averaged avg@32 and pass@32 across all four backbones (Table 2), while its sign–entropy coupling is empirically proven to be effective through the ablations in Tables 5 and 9.

AuthorsOpenReview ↗

Response to Reviewer cd2D (2/3)

W1-3. Final-answer diversity and epistemic markers as evidence of exploration

We appreciate the comment on final-answer diversity and epistemic markers as evidence of exploration. Beyond these proxies, we directly measure productive exploration with strategy coverage, where EAPO discovers and preserves more distinct successful strategies than all baselines. Specifically, we utilize LLM judges to summarize the solution strategy of each response and cluster them by strategy (using GPT-6-Luna and GPT-6.1-Sol, respectively). Strategy coverage@k is the fraction of all correct strategies found by any method that a method recovers with correct responses within kk samples, so it cannot be inflated by diverse incorrect outputs.

Table C.2. Strategy coverage@k (%) on Qwen3-4B-Base (full curve in Figure 4)

kInitialGRPOEntropyAdvHAPO80/20RLRTEAPO
18.612.812.712.311.116.321.6
824.135.033.430.026.639.345.5
3239.651.653.345.444.558.962.2

As shown in Table C.2, EAPO recovers the most successful strategies at every budget, indicating that its exploration gains come from a wider range of successful strategies rather than diverse incorrect outputs or stylistic changes. We have incorporated this analysis into the revision, with full results and experimental details provided in Sections 5.4 and C.4.

Final-answer diversity and epistemic markers serve as complementary proxies to these direct measures of pass@k and strategy coverage. Final-answer diversity captures whether the policy avoids converging to the same incorrect answer on challenging problems, even within the same strategy, while epistemic markers have been shown to be an effective measure of exploratory reasoning in [1].

[1] Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty, Kim et al., arXiv 2026


W2 & Q2. Held-out variability across training runs

We thank the reviewer for the suggestion to assess how reliably the gains persist. The gains of EAPO remain robust on held-out benchmarks across multiple training runs, and we have extended the training curves across seeds to include all baselines. Specifically, we evaluated the final checkpoints of three training runs per method on Qwen3-4B-Base on the held-out benchmarks.

Table C.3. Held-out results across three training runs on Qwen3-4B-Base (avg@32, mean ± std)

MethodAIME24AIME25AIME26HMMT26
GRPO12.8 ± 0.712.7 ± 0.79.2 ± 1.75.9 ± 0.9
EntropyAdv12.3 ± 0.810.5 ± 1.28.2 ± 0.45.1 ± 1.1
HAPO13.1 ± 0.811.6 ± 0.88.6 ± 0.76.5 ± 0.2
80/2011.4 ± 1.112.9 ± 0.56.3 ± 0.35.9 ± 0.9
RLRT13.2 ± 0.714.0 ± 1.013.9 ± 0.77.7 ± 0.7
EAPO20.2 ± 0.717.4 ± 0.816.2 ± 1.411.4 ± 0.6

As shown in Table C.3, the avg@32 gaps between EAPO and the strongest baseline, RLRT, exceed the standard deviations of both methods on all four benchmarks, confirming that its gains are consistent across training runs. We further re-evaluated all checkpoints in Table 2 under three evaluation seeds, where EAPO achieves the best mean avg@32 on most benchmarks across all four backbones (Table 12). Both analyses are included in Section D.6.

AuthorsOpenReview ↗

Response to Reviewer cd2D (3/3)

Q3. Comparison with GEPO, LESS, and A3PO

We appreciate the suggestion to evaluate these related methods directly. A3PO could not be reproduced under our setting, and its surprisal-based signal consistently led to reward collapse, whereas our entropy-based allocation remains stable. Using the released code and the original settings, including its decay schedule, while matching only the hyperparameters shared by all methods (e.g., LoRA and batch size), we could not reproduce the reported results, and thus did not include A3PO in the main paper. Replacing the entropy term of EAPO with the sampled-token surprisal used by A3PO also led to a collapse of the training reward (Section D.1), as did our runs reproducing A3PO.

GEPO and LESS, on the other hand, differ from EAPO in the granularity of credit assignment, as GEPO adjusts advantages per response group and LESS per low-entropy segment rather than redistributing credit across individual tokens, and thus were not included in the main experiments. We additionally provide their results in Table C.4, trained under the same configuration as all other methods.

Table C.4. Comparison with GEPO and LESS on Qwen3-4B-Base (avg@32 / pass@32)

MethodAIME24AIME25AIME26HMMT26
GEPO12.8 / 36.711.3 / 36.78.4 / 30.06.9 / 27.3
LESS14.6 / 43.312.9 / 40.010.9 / 36.78.1 / 30.3
EAPO20.2 / 53.317.1 / 46.715.7 / 50.013.4 / 33.3

As shown in Table C.4, EAPO surpasses GEPO and LESS on all four benchmarks with large margins in avg@32, suggesting that redistributing credit across individual tokens, toward uncertain decisions under success and confident decisions under failure, is central to its gains.


W3 & Q4. Decoding configuration of the resampling experiment

Thank you for the careful attention to the details of the resampling protocol. Both the initial generations and the resampled continuations in Section C.1 use the sampling settings of training rollouts (Table 8), with identical settings for high- and low-entropy windows. We have reflected these settings in Section C.1 of the revision, as summarized in Table C.5.

Table C.5. Decoding configuration of the resampling experiment

StageSamplesTemperature / top-pTop-kMaximum lengthStopping ruleRandom seed
Initial generations32 per problem1.0 / 1.0Disabled10,240 tokensEOS or maximum length42
Continuations64 per prefix1.0 / 1.0Disabled10,240 tokens including the prefixEOS or maximum length42
Reviewer_HavsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Standard GRPO assigns the same response-level advantage to every generated token. EAPO redistributes that advantage according to the model’s token entropy: uncertain positions receive stronger reinforcement in positive-advantage responses, while confident positions receive stronger penalties in negative-advantage responses. Its exponential weights are normalized within each response, preserving the mean advantage.

Training takes prompts, sampled responses, verifiable outcome rewards, and the rollout policy’s token distributions. It produces an updated language model without requiring token labels or a teacher. Inference remains ordinary response generation. The distinctive contribution is this particular bounded, normalized redistribution rule.

Critical strengths
  1. The method is practical. It can fit an existing RLVR pipeline using signals already available during training.
  2. The direction ablations are persuasive. Tables 5 and 9 test all nine combinations of entropy preference for positive and negative advantages. EAPO’s combination performs best on both base models, improving macro accuracy over uniform allocation by 6.52 and 7.24 percentage points.
  3. The improvement extends beyond one configuration. The paper reports gains on base and reasoning models and on Reasoning Gym tasks. The supplement also examines full fine-tuning, model scaling, and a response-length control.
Critical weaknesses
  1. The novelty argument mischaracterizes an important baseline. The introduction suggests that existing methods concentrate penalties on uncertain positions in failed responses. However, EntropyAdv adds a nonnegative, clipped entropy bonus to the signed advantage. Higher entropy therefore strengthens positive updates and reduces the magnitude of negative updates. This already produces the qualitative asymmetry motivating EAPO. EAPO’s normalization and relative redistribution are distinct, but the direction itself is not new.

    A3PO also already treats positive and negative tokens differently, while UCAS uses outcome-dependent confidence weighting. Both are discussed in Appendix B but lack empirical comparisons. The failed surprisal substitution in Appendix D.1 does not reproduce A3PO’s thresholding and adaptive weighting. Consequently, the paper establishes that EAPO’s chosen directions work within its formulation, but less clearly establishes the added value of its remaining differences from these close methods.

  2. The proposed mechanism remains only partially supported:

  • The resampling analysis does not establish that the selected low-entropy decisions actually caused failure.
  • Final-answer diversity and words such as “wait” can increase without more useful exploration.
  • Figure 4 provides meaningful coverage evidence, including the initial model and K up to 256, but only for one backbone on 30 AIME26 problems.
  • ReasoningGym results demonstrate transfer, rather than directly measuring exploration as branching or search difficulty increases. Thus, the unresolved question is whether EAPO preserves and discovers useful alternatives, rather than simply improving optimization or changing response style.
  1. On three of the four main backbones, the mean accuracy advantage over the strongest tested baseline is approximately 0.6–1.1%. Appendix D.6 reports three-seed training rewards, not held-out benchmark variability for these comparisons, and excludes RLRT. This does not invalidate the gains, especially the larger improvement on Qwen3-4B-Base, but makes the smaller advantages harder to assess.
Questions for authors
  1. The introduction suggests that existing entropy-based methods concentrate penalties on uncertain decisions. However, EntropyAdv’s nonnegative entropy bonus can attenuate negative advantages at high-entropy positions. How does this affect the distinction between EAPO’s contribution and existing asymmetric treatment of positive and negative updates?

  2. Should “improved exploration” be interpreted primarily as improved problem coverage within a sampling budget, or as discovering and preserving distinct successful reasoning strategies? Beyond the coverage gains in Figure 4, which existing results support the latter interpretation?

  3. Were the three training runs in Appendix D.6 also evaluated on the held-out benchmarks? If those results are available, how consistent are EAPO’s improvements across seeds, particularly where its advantage over competing methods is small?

9 comments
AuthorsOpenReview ↗

Response to Reviewer Havs (1/3)

We sincerely appreciate your thoughtful and detailed review. We are grateful for your acknowledgement that EAPO is practical and can be easily integrated into existing RLVR pipelines using only signals already available during training, that the sign–entropy ablations demonstrate the effectiveness of each direction in our design, and that its effectiveness holds across backbones, tasks, and training configurations, supported by several additional analyses. We have carefully considered all comments and concerns, and have made every effort to address them thoroughly.


W1-1 & Q1. Asymmetric treatment in EntropyAdv

Thank you for this careful comparison. EntropyAdv implements only a partial asymmetry, as it adds a nonnegative entropy bonus in an outcome-agnostic manner and leaves confident failure uncorrected. As the bonus is assigned regardless of the advantage sign, this asymmetry is a by-product of a global entropy incentive rather than an intended design. Moreover, the bonus in EntropyAdv can only attenuate penalties, never strengthen them. In contrast, EAPO assigns larger penalties to low-entropy tokens in failed responses, directly correcting confident failures that tend to recur.

To empirically verify this, we measure how often the strategies behind the initial policy's low-entropy failures recur after training. Specifically, on AIME24/25/26 and HMMT26, we identify failed responses of the initial Qwen3-4B-Base with low entropy at their critical decision points, and count how often each trained policy reproduces the same strategy (labeled by the LLM judges in Section 5.4) with an incorrect answer. As shown in Table B.1, EntropyAdv shows a recurrence rate similar to GRPO, whereas EAPO substantially reduces it by concentrating penalties on low-entropy tokens. Consistent with this, in the sign–entropy ablation (Tables 5 and 9), shifting only the penalization toward low-entropy positions, (0,0)→(0,−1)(0,0)\rightarrow(0,-1), improves accuracy on both base models.

Table B.1. Recurrence of low-entropy failure strategies of the initial Qwen3-4B-Base

MethodRecurrence (%, ↓)
GRPO34.0
EntropyAdv33.9
HAPO34.2
80/2036.1
RLRT29.3
EAPO23.6

Furthermore, the additive bonus alters the outcome signal itself, inflating the reinforcement of successful responses and shrinking the penalty of failed ones, which biases the update toward higher entropy irrespective of the outcome. EAPO instead preserves the mean advantage within each response and changes only its distribution across tokens. Consistent with this, EntropyAdv reaches the highest entropy among all methods (Figure 3) without a corresponding gain in accuracy, whereas EAPO achieves higher accuracy with lower entropy.


W1-2. Empirical comparison with A3PO and UCAS

We appreciate the suggestion to compare with these closely related methods. A3PO could not be reproduced under our setting, and its surprisal-based signal consistently led to reward collapse, whereas our entropy-based allocation remains stable. Using the released code and the original settings, including its decay schedule, while matching only the hyperparameters shared by all methods (e.g., LoRA and batch size), we could not reproduce the reported results, and thus did not include A3PO in the main paper. Replacing the entropy term of EAPO with the sampled-token surprisal used by A3PO also led to a collapse of the training reward (Section D.1), as did our runs reproducing A3PO.

UCAS, on the other hand, implements only a partial asymmetry, mainly at different granularity (i.e., response level). It scales all tokens of a response by its sequence confidence (not token-level), while its token-level term is a small confidence penalty applied regardless of the advantage sign, which strengthens penalties on confident tokens but never relieves those on uncertain ones (i.e., the opposite direction of EntropyAdv). While we did not include UCAS in the main experiments due to its focus on response-level weighting, we provide its results in Table B.2, together with other related methods discussed in Section B.

Table B.2. Comparison with UCAS and other related methods on Qwen3-4B-Base (avg@32 / pass@32)

MethodAIME24AIME25AIME26HMMT26
UCAS13.4 / 40.011.8 / 36.79.3 / 33.37.4 / 30.3
GEPO12.8 / 36.711.3 / 36.78.4 / 30.06.9 / 27.3
LESS14.6 / 43.312.9 / 40.010.9 / 36.78.1 / 30.3
EAPO20.2 / 53.317.1 / 46.715.7 / 50.013.4 / 33.3

As shown in Table B.2, EAPO outperforms all three methods on every benchmark by a large margin. GEPO adjusts advantages per response group while keeping token-level credit uniform, and LESS focuses its updates on low-entropy segments under both outcomes, whereas EAPO redistributes credit within each response toward uncertain decisions under success and confident decisions under failure, suggesting that this token-level asymmetry is central to its gains.

AuthorsOpenReview ↗

Response to Reviewer Havs (2/3)

W2-1. Resampling does not establish that low-entropy decisions caused failure

We appreciate this insightful question on the causal role of low-entropy decisions. EAPO does not require low-entropy decisions to be the cause of failure, but only that continuing confidently from them is suboptimal, which our intervention confirms. Whether the error arises at these decisions or earlier in the response, this is what matters for credit assignment. As resampling alone (Table 1) cannot tell whether such a prefix is already irrecoverable, we insert a short reflection cue (e.g., "Wait, let me reconsider this step.") at low-entropy windows of newly drawn failed responses, and compare continuations from the same prefix with and without the cue, following Section C.1.

Table B.3. Accuracy of continuations from low-entropy windows of failed responses

ModelResampled (%)Intervened (%)Δ\Delta (pp)
Qwen3-4B-Base15.925.8+10.0
Qwen3-8B-Base10.517.4+6.9

As shown in Table B.3, the cue significantly improves accuracy on both models, showing that the policy can recover from these states once it departs from its original confident behavior. Without the cue, however, the policy largely regenerates the same path (Table 1), which motivates EAPO to concentrate penalties on such confident continuations.


W2-2 & Q2. Whether EAPO discovers and preserves useful alternatives

Thank you for this thoughtful question on how improved exploration should be interpreted. Beyond broader coverage, EAPO discovers and preserves more distinct successful strategies than all baselines. To measure this directly, we utilize LLM judges to summarize the solution strategy of each response and cluster them by strategy (using GPT-6-Luna and GPT-6.1-Sol, respectively). Strategy coverage@k is the fraction of all correct strategies found by any method that a method recovers with correct responses within kk samples, so it cannot be inflated by diverse incorrect outputs.

Table B.4. Strategy coverage@k (%) on Qwen3-4B-Base (full curve in Figure 4)

kInitialGRPOEntropyAdvHAPO80/20RLRTEAPO
18.612.812.712.311.116.321.6
824.135.033.430.026.639.345.5
3239.651.653.345.444.558.962.2

As shown in Table B.4, EAPO recovers the most successful strategies at every budget, indicating that its coverage gains come from a wider range of strategies rather than reaching the same answer repeatedly. We have incorporated this analysis into the revision, with full results and experimental details provided in Sections 5.4 and C.4.

Final-answer diversity and epistemic markers serve as complementary proxies to these direct measures of pass@k and strategy coverage. Final-answer diversity captures whether the policy avoids converging to the same incorrect answer on challenging problems, even within the same strategy, while epistemic markers have been shown to be an effective measure of exploratory reasoning in [1].

[1] Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty, Kim et al., arXiv 2026


W2-3. Coverage evidence limited to one backbone on AIME26

We thank the reviewer for acknowledging the coverage experiment as a measure of exploration. We further measure pass@k on additional backbones and benchmarks, where EAPO consistently achieves the highest coverage. Specifically, following the same sampling procedure as Figure 4 with up to 256 samples per problem, we evaluate HMMT26 with Qwen3-4B-Base and AIME26 and HMMT26 with Qwen3-8B-Base.

Table B.5. Pass@k (%) on additional backbones and benchmarks (full curves in Figure 11)

SettingkInitialGRPOEntropyAdvHAPO80/20RLRTEAPO
HMMT26 (Qwen3-4B-Base)14.55.86.15.66.18.311.5
813.015.617.916.715.323.827.6
3220.822.526.225.022.830.834.8
25630.336.436.436.436.442.451.5
AIME26 (Qwen3-8B-Base)18.010.98.88.612.817.217.6
821.625.724.124.927.134.237.6
3230.632.332.032.733.947.452.7
25640.046.750.046.753.370.073.3
HMMT26 (Qwen3-8B-Base)15.46.45.65.37.012.112.1
817.118.616.616.520.527.226.7
3223.626.324.624.730.433.534.2
25630.336.436.439.442.442.442.4

As shown in Table B.5, EAPO maintains its margin over the baselines even at the largest budget on HMMT26 with Qwen3-4B-Base and AIME26 with Qwen3-8B-Base, indicating that its broader problem coverage is not specific to a particular backbone or benchmark. We have incorporated this analysis into the revision, with full results provided in Section D.7.

AuthorsOpenReview ↗

Response to Reviewer Havs (3/3)

W2-4. Role of the Reasoning Gym evaluation

Thank you for your careful comment. The Reasoning Gym evaluation in Table 3 is intended to examine whether the gains of EAPO generalize beyond mathematics, rather than to measure exploration. EAPO achieves the best results on both base backbones, showing that its gains generalize well to diverse reasoning tasks. EAPO promotes exploration by keeping alternatives open at uncertain decisions, so that the policy reaches more distinct solutions across samples, which concerns the policy itself rather than the search space of a task. We therefore measure it directly on fixed problems in Section 5.4, through pass@k, strategy coverage, and final-answer diversity.


W3 & Q3. Held-out performance across seeds

We appreciate the comment on the need for evaluation across seeds. We evaluate EAPO and the baselines across both evaluation and training seeds, where EAPO remains consistently superior with variability far smaller than its margins. Specifically, we re-evaluated all checkpoints in Table 2 under three evaluation seeds (Table B.6), and evaluated three training runs on Qwen3-4B-Base on the held-out benchmarks, including RLRT and 80/20 (Table B.7).

Table B.6. Results across three evaluation seeds (avg@32, mean ± std)

ModelMethodAIME24AIME25AIME26HMMT26
Qwen3-4B-BaseInitial10.7 ± 0.25.7 ± 0.46.6 ± 0.23.9 ± 0.0
GRPO12.0 ± 0.412.8 ± 0.69.1 ± 0.55.9 ± 0.2
EntropyAdv12.9 ± 0.411.4 ± 0.38.5 ± 0.75.9 ± 0.3
HAPO13.2 ± 1.111.9 ± 0.28.9 ± 0.36.1 ± 0.4
80/2010.9 ± 0.311.4 ± 0.37.3 ± 0.36.8 ± 0.2
RLRT14.5 ± 0.613.9 ± 0.613.6 ± 0.67.4 ± 0.2
EAPO20.5 ± 0.417.1 ± 0.516.2 ± 0.612.0 ± 0.2
Qwen3-8B-BaseInitial13.2 ± 0.513.1 ± 0.47.6 ± 0.35.7 ± 0.1
GRPO16.1 ± 0.314.1 ± 0.711.1 ± 0.86.7 ± 0.4
EntropyAdv15.9 ± 0.915.2 ± 0.38.8 ± 0.96.1 ± 0.9
HAPO14.9 ± 0.713.6 ± 1.28.6 ± 0.25.1 ± 0.7
80/2016.6 ± 0.314.8 ± 0.513.1 ± 0.27.1 ± 0.3
RLRT22.8 ± 0.318.8 ± 0.417.1 ± 0.611.8 ± 0.3
EAPO23.9 ± 0.420.7 ± 0.417.6 ± 1.712.5 ± 0.8
Qwen3-4BInitial69.3 ± 1.759.8 ± 0.961.1 ± 1.140.6 ± 0.1
GRPO70.7 ± 0.763.4 ± 1.062.5 ± 0.540.5 ± 0.6
EntropyAdv72.2 ± 0.662.7 ± 0.663.8 ± 0.442.6 ± 0.8
HAPO71.2 ± 1.362.7 ± 0.562.3 ± 0.841.6 ± 0.7
80/2069.6 ± 0.361.6 ± 0.359.6 ± 0.541.0 ± 0.1
RLRT73.8 ± 0.863.3 ± 0.463.2 ± 0.142.3 ± 0.8
EAPO72.5 ± 0.864.4 ± 1.266.1 ± 0.644.1 ± 0.5
Olmo-3-7B-Think-DPOInitial74.5 ± 0.863.8 ± 1.568.7 ± 1.544.9 ± 0.6
GRPO75.6 ± 0.264.3 ± 0.869.6 ± 1.646.1 ± 0.8
EntropyAdv75.4 ± 0.664.7 ± 0.769.5 ± 1.045.5 ± 0.2
HAPO75.8 ± 1.063.0 ± 0.566.1 ± 1.545.5 ± 0.7
80/2075.2 ± 0.565.1 ± 0.769.5 ± 0.845.5 ± 0.7
RLRT76.8 ± 0.563.4 ± 0.267.6 ± 0.444.1 ± 0.5
EAPO76.5 ± 1.065.2 ± 0.870.7 ± 0.746.4 ± 0.6

Table B.7. Held-out results across three training runs on Qwen3-4B-Base (avg@32, mean ± std)

MethodAIME24AIME25AIME26HMMT26
GRPO12.8 ± 0.712.7 ± 0.79.2 ± 1.75.9 ± 0.9
EntropyAdv12.3 ± 0.810.5 ± 1.28.2 ± 0.45.1 ± 1.1
HAPO13.1 ± 0.811.6 ± 0.88.6 ± 0.76.5 ± 0.2
80/2011.4 ± 1.112.9 ± 0.56.3 ± 0.35.9 ± 0.9
RLRT13.2 ± 0.714.0 ± 1.013.9 ± 0.77.7 ± 0.7
EAPO20.2 ± 0.717.4 ± 0.816.2 ± 1.411.4 ± 0.6

As shown in Table B.6, the variation across evaluation seeds is small relative to the differences between methods, and EAPO achieves the best mean avg@32 on most benchmarks across backbones, indicating that its gains do not stem from a particular evaluation seed. Even for reasoning backbones, where gaps between methods are narrower due to extensive prior post-training, EAPO outperforms the strongest baseline in most cases. Across training runs (Table B.7), EAPO also achieves the best mean avg@32 on all four benchmarks with gaps exceeding the standard deviations of both methods, confirming that its gains on held-out benchmarks are consistent across training runs. We have added both analyses to Section D.6.

Reviewer_qW2LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies token-level credit assignment in RLVR methods such as GRPO, where one response-level advantage is shared by all tokens. Prior entropy-based methods often favor high-entropy tokens under both success and failure, which can put strong penalties on uncertain positions in failed responses. The authors propose EAPO: they use the policy’s own entropy and weight tokens differently by the sign of the advantage. Successful responses put more weight on high-entropy tokens; failed responses put more weight on low-entropy tokens and soften the penalty on uncertain ones. No auxiliary model or privileged information is needed. Experiments cover Qwen3 and Olmo base/reasoning models on several math benchmarks and OOD tasks, with pass@k, diversity analyses, and ablations. EAPO reports strong macro-averaged gains especially on base models.

Critical strengths
  1. The asymmetric entropy idea is clear and backed by a clean ablation. Unlike prior entropy methods that largely favor high-entropy tokens in both cases, EAPO reverses the preference under failure. In Table 5, the default setting (high-entropy under success, low-entropy under failure) is best, and clearly better than uniform credit. This is the main reason I lean accept.
  2. The method is light and shows solid gains on base models. It only redistributes the existing advantage. On Qwen3-4B/8B-Base, Table 2 reports clear macro-average gains over strong entropy-style baselines, and it often remains competitive with methods that use privileged information. The writing is easy to follow.
Critical weaknesses
  1. Statistical reliability of the main numbers is not fully clear. Training uses a fixed seed by default. The appendix shows multi-seed training reward, but not mean±std for the main avg@32 / pass@32 metrics in Table 2. AIME-style sets are small, so single-run gaps can move around.
  2. Gains shrink on strong reasoning backbones, and some “exploration” evidence relies on proxies. Improvements on already-strong reasoning models are smaller than on base models, and EAPO is not best in every cell. Answer entropy and epistemic markers show diversity or style changes, but they do not by themselves equal better reasoning search. Pass@k is stronger evidence; the link to actual recovery could be clearer. These points do not outweigh the main idea and the ablation for me, but they keep the paper at weak accept rather than a stronger accept. Multi-seed main metrics and a clearer analysis on reasoning backbones would make me more confident.
Questions for authors
  1. Can you report mean±std over at least three seeds for the main Table 2 metrics (or at least AIME24/25/26)?
  2. On reasoning backbones, when does asymmetric credit still help, and when is it close to GRPO? Are there settings with little or no gain?
  3. The §3 / Table 1 study is suggestive but small. During training, do high-entropy successes become more repeatable, and low-entropy failures less frequent?
  4. Does the rise in epistemic markers correlate with actual recovery, or mainly with longer traces?
4 comments
AuthorsOpenReview ↗

Response to Reviewer qW2L (1/2)

We sincerely appreciate your thoughtful review. We are grateful for your recognition that the idea of asymmetric entropy is clear and supported by a clean ablation, which reverses the entropy preference under failure unlike prior entropy-based methods, and that EAPO is lightweight, redistributing only the existing advantage while showing solid gains over strong entropy-based baselines on base models. We have carefully considered all comments and concerns, and have made every effort to address them thoroughly.


W1 & Q1. Statistical reliability across multiple seeds

We appreciate the suggestion to strengthen the statistical reliability of the main results. EAPO remains consistently superior across both training and evaluation seeds, with variation small relative to the differences between methods. Specifically, we re-evaluated all checkpoints in Table 2 under three evaluation seeds on AIME24/25/26 and HMMT26.

Table A.1. Results across three evaluation seeds (avg@32, mean ± std)

ModelMethodAIME24AIME25AIME26HMMT26
Qwen3-4B-BaseInitial10.7 ± 0.25.7 ± 0.46.6 ± 0.23.9 ± 0.0
GRPO12.0 ± 0.412.8 ± 0.69.1 ± 0.55.9 ± 0.2
EntropyAdv12.9 ± 0.411.4 ± 0.38.5 ± 0.75.9 ± 0.3
HAPO13.2 ± 1.111.9 ± 0.28.9 ± 0.36.1 ± 0.4
80/2010.9 ± 0.311.4 ± 0.37.3 ± 0.36.8 ± 0.2
RLRT14.5 ± 0.613.9 ± 0.613.6 ± 0.67.4 ± 0.2
EAPO20.5 ± 0.417.1 ± 0.516.2 ± 0.612.0 ± 0.2
Qwen3-8B-BaseInitial13.2 ± 0.513.1 ± 0.47.6 ± 0.35.7 ± 0.1
GRPO16.1 ± 0.314.1 ± 0.711.1 ± 0.86.7 ± 0.4
EntropyAdv15.9 ± 0.915.2 ± 0.38.8 ± 0.96.1 ± 0.9
HAPO14.9 ± 0.713.6 ± 1.28.6 ± 0.25.1 ± 0.7
80/2016.6 ± 0.314.8 ± 0.513.1 ± 0.27.1 ± 0.3
RLRT22.8 ± 0.318.8 ± 0.417.1 ± 0.611.8 ± 0.3
EAPO23.9 ± 0.420.7 ± 0.417.6 ± 1.712.5 ± 0.8
Qwen3-4BInitial69.3 ± 1.759.8 ± 0.961.1 ± 1.140.6 ± 0.1
GRPO70.7 ± 0.763.4 ± 1.062.5 ± 0.540.5 ± 0.6
EntropyAdv72.2 ± 0.662.7 ± 0.663.8 ± 0.442.6 ± 0.8
HAPO71.2 ± 1.362.7 ± 0.562.3 ± 0.841.6 ± 0.7
80/2069.6 ± 0.361.6 ± 0.359.6 ± 0.541.0 ± 0.1
RLRT73.8 ± 0.863.3 ± 0.463.2 ± 0.142.3 ± 0.8
EAPO72.5 ± 0.864.4 ± 1.266.1 ± 0.644.1 ± 0.5
Olmo-3-7B-Think-DPOInitial74.5 ± 0.863.8 ± 1.568.7 ± 1.544.9 ± 0.6
GRPO75.6 ± 0.264.3 ± 0.869.6 ± 1.646.1 ± 0.8
EntropyAdv75.4 ± 0.664.7 ± 0.769.5 ± 1.045.5 ± 0.2
HAPO75.8 ± 1.063.0 ± 0.566.1 ± 1.545.5 ± 0.7
80/2075.2 ± 0.565.1 ± 0.769.5 ± 0.845.5 ± 0.7
RLRT76.8 ± 0.563.4 ± 0.267.6 ± 0.444.1 ± 0.5
EAPO76.5 ± 1.065.2 ± 0.870.7 ± 0.746.4 ± 0.6

As shown in Table A.1, EAPO achieves the best mean avg@32 on most benchmarks across backbones, indicating that its gains do not stem from a particular evaluation seed. Even for reasoning backbones, where gaps between methods are narrower due to extensive prior post-training, EAPO outperforms the strongest baseline in most cases. We further evaluated three training runs per method on Qwen3-4B-Base, where EAPO achieves the best mean avg@32 and pass@32 on all four benchmarks (Table 11), and we have incorporated both analyses to Section D.6.


W2-1 & Q2. Gains on reasoning backbones

We thank the reviewer for this point about the smaller gains on reasoning backbones. Reasoning backbones leave limited headroom for further post-training due to their extensive prior post-training, yet EAPO keeps its asymmetric credit distinct from GRPO by normalizing entropy relative to each rollout batch. Indeed, the official RL stage applied on top of Olmo-3-7B-Think-DPO, the checkpoint we use, even lowers its accuracy on several math benchmarks [1]. Although these backbones exhibit sharpened distributions with low entropy overall, EAPO maps token entropies onto shared percentiles within each rollout batch (Section 4.1), so that relatively uncertain decisions still receive stronger reinforcement under success and weaker penalties under failure even when absolute entropy is low. To this end, EAPO achieves the highest average avg@32 and pass@32 on both reasoning backbones in Table 2, whereas GRPO barely changes pass@32 from the initial model.

[1] https://huggingface.co/allenai/Olmo-3-7B-Think-DPO#evaluation

AuthorsOpenReview ↗

Response to Reviewer qW2L (2/2)

W2-2. Exploration evidence beyond diversity and stylistic proxies

We appreciate the comment that answer entropy and epistemic markers do not by themselves establish better reasoning search. Beyond pass@k, we directly measure productive exploration with strategy coverage, where EAPO discovers and preserves more distinct successful strategies than all baselines. Specifically, we utilize LLM judges to summarize the solution strategy of each response and cluster them by strategy (using GPT-6-Luna and GPT-6.1-Sol, respectively). Strategy coverage@k is the fraction of all correct strategies found by any method that a method recovers with correct responses within kk samples, so it cannot be inflated by diverse incorrect outputs.

Table A.2. Strategy coverage@k (%) on Qwen3-4B-Base (full curve in Figure 4)

kInitialGRPOEntropyAdvHAPO80/20RLRTEAPO
18.612.812.712.311.116.321.6
824.135.033.430.026.639.345.5
3239.651.653.345.444.558.962.2

As shown in Table A.2, EAPO recovers the most successful strategies at every budget, indicating that its exploration gains come from a wider range of successful strategies rather than diverse incorrect outputs or stylistic changes. We have incorporated this analysis into the revision, with full results and experimental details provided in Sections 5.4 and C.4, and its pass@k gains likewise extend to additional backbones and benchmarks (Section D.7).

Final-answer diversity and epistemic markers serve as complementary proxies to direct measures such as pass@k and strategy coverage. Final-answer diversity captures whether the policy avoids converging to the same incorrect answer on challenging problems, even within the same strategy, while epistemic markers have been shown to be an effective measure of exploratory reasoning rather than a stylistic artifact in [2].

[2] Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty, Kim et al., arXiv 2026


Q3. Repeatability of success and failure during training

Thank you for raising this question. After training, the strategies behind high-entropy successes of the initial policy are reproduced successfully more often, while those behind its low-entropy failures recur far less often under EAPO than under any baseline. Specifically, we label each response of the initial Qwen3-4B-Base on AIME24/25/26 and HMMT26 as high- or low-entropy based on the entropy at its critical decision points, and use the strategy labels in Section 5.4 (obtained by the LLM judges as described in W2-2) to measure how often each policy reproduces the strategy of a high-entropy success with a correct answer, or that of a low-entropy failure with an incorrect answer.

Table A.3. Recurrence (%) of the strategies behind high-entropy successes and low-entropy failures of the initial Qwen3-4B-Base

Initial responseGRPOEntropyAdvHAPO80/20RLRTEAPO
High-entropy success (↑)35.337.337.837.738.638.8
Low-entropy failure (↓)34.033.934.236.129.323.6

As shown in Table A.3, EAPO reproduces the strategies behind high-entropy successes most often among all methods, while reducing the recurrence of failed strategies far more than all baselines, consistent with its asymmetric design that strengthens reinforcement at uncertain decisions in successful responses and concentrates penalties on confident decisions in failed responses.


Q4. Epistemic markers and actual recovery

Thank you for this question on what the rise in epistemic markers reflects. The marker frequency is already normalized by length, and inserting such cues into failed responses leads to actual recovery, indicating that the markers are not merely a by-product of longer traces. To examine their link to recovery, we insert a short reflection cue (e.g., "Wait, let me reconsider this step.") at low-entropy windows of newly drawn failed responses of the initial model following Section C.1, and compare 64 continuations from the same prefix with and without the cue.

Table A.4. Accuracy of continuations from low-entropy windows of failed responses, with and without a reflection cue

ModelResampled (%)Intervened (%)Δ\Delta (pp)
Qwen3-4B-Base15.925.8+10.0
Qwen3-8B-Base10.517.4+6.9

As shown in Table A.4, the cue significantly improves accuracy on both models, showing that such cues can redirect the policy from a failing path toward a correct one, consistent with [2], where epistemic markers are shown to be an effective measure of exploratory reasoning.

[2] Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty, Kim et al., arXiv 2026

AuthorsOpenReview ↗

Gentle Follow-Up on Our Responses

Thank you once again for taking the time to review our manuscript. We have carefully addressed all the concerns raised in your review and provided detailed responses to fully resolve each of the points raised.

As the discussion period is coming to a close, we would like to kindly follow up on our previous response in case you have not yet had a chance to review it. We appreciate the many demands on your time and would be grateful for any opportunity you may have to review our responses.

Thank you once again for your time and consideration.

Thank you again for your time and constructive feedback. As the discussion period is coming to a close, we wanted to follow up in case you have not yet had a chance to look at our response. We have incorporated all of the reviewers’ comments into the revision, as summarized above. We understand how busy this period is, and we would appreciate any further comments you may have.

Thank you for your consideration.

Comments on the paper

3 comments

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

We thank the reviewers for their time and constructive feedback. We are glad that the reviewers found:

  1. A practical and lightweight method. EAPO redistributes only the existing advantage using policy entropy (qW2L), without auxiliary models, privileged information, or additional rollouts (cd2D), and is easy to integrate into existing RLVR pipelines (Havs, cd2D).
  2. A clear asymmetric design with persuasive ablations. The asymmetric idea is clear (qW2L), the sign–entropy ablations are persuasive (Havs, qW2L), and the resampling experiment offers a useful empirical basis (cd2D).
  3. Consistent improvements across backbones. EAPO achieves the highest avg@32 and pass@32 across all four backbones (cd2D), with gains on both base and reasoning models (Havs) and over strong entropy-based baselines (qW2L).
  4. Generalization beyond the training setup. The improvements transfer to Reasoning Gym tasks (Havs, cd2D), with further analyses on full fine-tuning, model scaling, and response length (Havs).

We summarize the main points below:

  1. Robustness across training and evaluation seeds. Across three training runs on Qwen3-4B-Base, EAPO achieves the best mean avg@32 on all four held-out benchmarks, with gaps exceeding the standard deviations. Across three evaluation seeds on all four backbones, it remains the best on most configurations. (Havs W3 & Q3, cd2D W2 & Q2, qW2L W1 & Q1)
  2. Exploration as discovering distinct successful strategies. We directly measure successful strategies with LLM-judged labels, beyond proxies such as final-answer diversity and epistemic markers. EAPO recovers 62.2% of all successful strategies within 32 samples, against 58.9% for RLRT and 51.6% for GRPO. This does not merely come from solving more problems, as EAPO still finds more distinct strategies per problem when the problems and the number of correct responses are matched. Its pass@k gains also extend to additional backbones and benchmarks. (Havs W2-2 & W2-3 & Q2, cd2D W1-3, qW2L W2-2)
  3. Distinction from EntropyAdv. EntropyAdv attenuates penalties at uncertain positions through an outcome-agnostic bonus without strengthening the correction of confident failures, whereas EAPO couples entropy with the advantage sign to concentrate correction on them. The low-entropy failures of the initial policy recur at 33.9% under EntropyAdv, comparable to GRPO (34.0%), and at 23.6% under EAPO. (Havs W1-1 & Q1, qW2L Q3)
  4. Comparison with closely related methods. EAPO outperforms related methods that leverage uncertainty at different granularities (UCAS, GEPO, and LESS) on all benchmarks, reaching 20.2% avg@32 on AIME24 against 14.6% for the strongest of them. A3PO, reproduced with its released code and original settings, led to reward collapse, as did replacing the entropy term of EAPO with sampled-token surprisal. (Havs W1-2, cd2D Q3)
  5. Failed responses remain recoverable from low-entropy windows. Inserting a short reflection cue at low-entropy windows of failed responses improves continuation accuracy from 15.9% to 25.8% on Qwen3-4B-Base and from 10.5% to 17.4% on Qwen3-8B-Base, showing that a correct continuation remains reachable once the policy departs from its confident behavior, where EAPO concentrates penalties. (Havs W2-1, cd2D W1-1 & Q1, qW2L Q4)

We have further updated the manuscript, reflecting all concerns raised by the reviewers, with changes shown in orange:

  • Sections 1, 6, and B: clarified the distinction between EAPO and existing entropy-based methods, including EntropyAdv. (Havs W1-1 & Q1)
  • Abstract, Section 5.4, Figure 4, and Section C.4 (Figures 5 and 6): added strategy coverage with LLM-judged labels and judge prompts. (Havs W2-2 & Q2, cd2D W1-3, qW2L W2-2)
  • Section D.6 (Table 13): added the number of distinct strategies under matched problems and correct responses. (Havs W2-2 & Q2)
  • Section D.5 (Table 12) and Section G (Figure 13): added the recurrence of high-entropy successes and low-entropy failures, with a case study. (Havs W1-1 & Q1, qW2L Q3)
  • Sections 3 and D.3 (Table 10): added the reflection-cue intervention at low-entropy windows of failed responses. (Havs W2-1, cd2D W1-1 & Q1, qW2L Q4)
  • Sections B and D.1 (Table 9): extended the discussion of UCAS and LESS, and added the comparison with UCAS, GEPO, and LESS. (Havs W1-2, cd2D Q3)
  • Section D.2: reported the reproduction of A3PO. (Havs W1-2, cd2D Q3)
  • Section D.10 (Figure 10, Tables 15 and 16): added 80/20 and RLRT to the training curves, held-out results across three training runs, and results across three evaluation seeds. (Havs W3 & Q3, cd2D W2 & Q2, qW2L W1 & Q1)
  • Sections 5.4 and D.11 (Figure 11): added pass@k on additional backbones and benchmarks. (Havs W2-3)
  • Section C.1: specified the decoding configuration of the resampling experiment. (cd2D W3 & Q4)
AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewers,

Thank you very much for your thoughtful comments and valuable feedback. Your suggestions have helped us substantially improve our manuscript.

As the discussion period draws to a close, we would like to follow up on our responses and additional revisions. We have incorporated further clarifications and improvements to address the concerns raised during the discussion.

We would appreciate it if you could review our follow-up comments and revisions and let us know whether your remaining concerns have been adequately addressed. We hope that our revisions have fully addressed your concerns, and we would greatly appreciate any further feedback or suggestions.

Thank you again for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.