acceptodds
Under review as a conference paper at ICLR 2027

Exploration-Preserving Policy Optimization

Abstract

Reinforcement learning with verifiable rewards improves reasoning, while the allocation of learning signal shapes which solutions remain accessible under repeated sampling. Group-relative objectives assign equal advantages to equally rewarded responses, making aggregate credit proportional to sampled mode frequency. We introduce Exploration-Preserving Policy Optimization (ExPPO), a lightweight advantage-shaping rule that redistributes credit using prompt-relative, length-normalized response surprisal and prompt pass rate. ExPPO combines bounded shaping with shared normalization to preserve verifier polarity and approximately maintain each prompt group’s total absolute sequence-advantage mass. Our analysis characterizes response-level credit allocation alongside sampled mode updates, deriving local conditions for gains in entropy and correct-mode discovery. Experiments show improved in-domain and out-of-domain reasoning coverage, higher aggregate response accuracy, and strong coverage at large sampling budgets. A controlled multi-answer evaluation further demonstrates increased correct-mode yield and gains in diversity among verified-correct responses.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-training, Large Language Models, Reinforcement Learning, RLVR, Exploration, OOD Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GBScOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Exploration-Preserving Policy Optimization (ExPPO), motivated by the observation that group-relative RL can concentrate credit on solution modes that are already sampled frequently. ExPPO reallocates response-level advantage using a prompt-relative rarity signal based on length-normalized sequence surprisal together with the prompt pass rate. The shaping preserves the sign of the original advantage while increasing credit for relatively surprising correct responses and modifying the treatment of other common/rare correct and incorrect responses. The method is evaluated on a controlled multi-answer task, mathematical reasoning benchmarks, and coding tasks, where it generally improves correct-solution coverage and diversity relative to GRPO and entropy-regularized baselines.

Critical strengths

I find the paper's problem formulation compelling. In particular, it makes an important distinction between maintaining generic policy entropy and preserving diversity within the support of correct solutions. For reasoning models trained with verifier-based rewards, the goal is not simply to keep the policy uncertain, but to prevent premature concentration on a small subset of already-discovered correct reasoning modes. This perspective is valuable because conventional entropy regularization is reward-agnostic: it may encourage exploration of unlikely incorrect continuations as well as useful alternative solutions, whereas ExPPO explicitly reallocates verifier-conditioned credit toward responses that are both promising and underrepresented.

The proposed formulation is also practically appealing because it modifies the existing GRPO training process using statistics that are already available during rollout generation, without requiring additional reward models or external evaluation mechanisms. The sign-preserving advantage reshaping is a reasonable design choice, as it aims to improve exploration while maintaining the original distinction between correct and incorrect responses.

The controlled multi-answer experiment is another strong aspect of the validation. Since the task provides multiple verifiably correct semantic solution modes, it allows the authors to evaluate whether the method improves correct-mode discovery rather than merely increasing token-level entropy or surface-form diversity. This provides a more meaningful evaluation of exploration behavior in reasoning RL than standard entropy-based metrics alone.

Critical weaknesses

My main concern is that the empirical validation does not fully establish the connection between the proposed rarity signal and the phenomenon the paper aims to address. ExPPO is motivated by preserving diversity among semantic solution modes, but the method operates on response-level length-normalized sequence surprisal. A low-probability trajectory is not necessarily a rare reasoning strategy: different surface realizations of the same underlying solution approach can have substantially different surprisal values, while a genuinely distinct reasoning mode may have high aggregate probability due to multiple paraphrased realizations. Therefore, the current formulation may conflate trajectory-level novelty with semantic-level diversity.

In addition, the evaluation does not fully rule out simpler explanations for the observed improvements. Since ExPPO modifies the advantage distribution to favor rare successful trajectories, the gains could partly come from a general exploration incentive rather than specifically addressing correct-mode collapse. Stronger comparisons against methods that explicitly optimize solution diversity or coverage would help clarify whether ExPPO provides benefits beyond generic exploration regularization.

Questions for authors
  1. On the controlled multi-answer task, can the authors directly measure the relationship between response surprisal and empirical semantic-mode frequency? For example, reporting correlation between surprisal and negative log mode frequency, or within-mode versus between-mode surprisal variation, would materially affect my confidence in the central rarity proxy.

  2. Since surprisal is computed at the response level, how sensitive is ExPPO to surface-form variation? For example, if two responses correspond to the same reasoning strategy but differ substantially in wording or verbosity, does ExPPO risk allocating additional credit to stylistic variants rather than genuinely different solution modes?

3 comments

Thank you for highlighting the gap between response surprisal and semantic solution rarity. We add direct mode-frequency correlations, controlled method-versus-paraphrase tests, and a whitespace sensitivity check. We also add Cue-GRPO and compare correct-mode yield and diversity across methods at the same sampling budget.

W1/Q1/Q2. Mode rarity, reasoning methods, and surface-form sensitivity

Surprisal reflects both the probability of a mode and the probability of a wording within that mode. We added direct frequency measurements and controlled tests of mathematical methods and paraphrases, using mathematics-trained Qwen2.5-7B checkpoints.

Mode frequency. Each verified final answer vector defines a mode. On 64 multi-answer questions, we use 256 responses per question with a 2,048-token cap. We calculate partial Pearson correlations between surprisal and negative log estimated mode probability, adjusting both for log length and formatting controls. Correlations use correct responses within each question and fold; we average the two reciprocal-fold estimates, then average equally over eligible questions.

ExPPO stepEligible questionsPartial Pearson correlationBetween-mode variance share
10060/64+0.24042.9%
50064/64+0.22643.3%
90050/64+0.25341.1%

The positive correlations show that rarer correct modes tend to have higher surprisal. Mode identity accounts for 41–43% of the variance.

Methods versus paraphrases. We constructed 64 mathematical questions with two verified methods and four paraphrases per method, matching lengths within 10% under both tokenizers. Method comparisons hold wording style fixed, and wording comparisons hold the method fixed. Both strategy tests below use step-900 checkpoints.

After adjusting for log length, we average the question-level contrast C=(dm−dw)/(dm+dw)C=(d_m-d_w)/(d_m+d_w), where dmd_m and dwd_w are mean squared surprisal differences from changing the method and wording. Positive values mean a larger method effect.

MethodLength-adjusted contrastPositive questionsExact-length subset contrast, 39 questions
ExPPO+0.55860/64+0.793
GRPO+0.53260/64+0.745

Within- and between-strategy variation. A separate GCD test uses 64 questions, three verified strategies per question, and four paraphrases per strategy:

MethodLength-adjusted between-strategy variance shareQuestions with between-strategy variance greater than within-strategy variance
ExPPO66.0%50/64
GRPO65.1%49/64

Both models are sensitive to differences between the tested methods.

Surface edits can still change credit. A whitespace-only control, with reasoning and group normalization fixed, gives a mean absolute change of 0.0177 normalized advantage units over 64 anchors from 45 questions. ExPPO bounds the raw correction by λ∣Ai∣λ|A_i| and preserves the advantage sign (Eqs. (8)–(9)).

W2. Comparisons with methods targeting diversity or coverage

We added Cue-GRPO, which explicitly reweights correct responses by strategy-cluster rarity. The comparison also includes GRPO-UR's emphasis on unlikely correct responses, F-GRPO's success-rate weighting, and REPO-R's entropy feedback.

We use the existing step-900 checkpoints and the first 256 responses per question on the same 64-question task.

MethodDistinct correct modes/questionEffective correct modes/questionCorrect-pair diversity (%)
GRPO3.9692.69450.97
F-GRPO3.7812.23840.50
GRPO-UR2.2502.03838.55
Cue-GRPO3.6252.62950.28
REPO-R4.3443.54868.59
ExPPO5.0003.82069.29

Effective modes are exponentiated correct-mode entropy, with zero for unsolved questions. Mode yields average over all 64 questions. Correct-pair diversity is the fraction of pairs of correct response samples, drawn without replacement, whose modes differ; repeated answers remain separate samples. It averages equally over the common questions with at least two correct responses under every method. All methods use the same response budget.

ExPPO leads all three measures. Compared with Cue-GRPO, it discovers 1.375 more distinct / 1.191 more effective modes per question, with 19.01 points higher correct-pair diversity. Compared with REPO-R, conditional diversity is close, while mode yields are higher.

The paper also reports a separate Phi-3.5-mini comparison at shared K=128K=128 across AIME24, AIME25, Minerva, and OlympiadBench (Figure 4). Coverage is the percentage of questions with at least one correct response, averaged equally across benchmarks:

MethodFour-task ID coverage (%)
F-GRPO12.17
GRPO-UR44.34
REPO-R40.30
ExPPO51.03

Thank you for the additional mode-frequency analyses, paraphrase controls, and diversity-baseline comparisons. These results help address my concerns about the rarity signal, although its generalization beyond the controlled tasks remains uncertain. I maintain my original rating of 3.

Thank you for the follow-up. We agree that the rarity signal needs further validation beyond controlled tasks. Our paper also evaluates broader OOD performance on GPQA, MMLU-Pro, Safety, and TruthfulQA-MC across different models and training settings. We will conduct more experiments on broader tasks to directly evaluate the rarity signal and report the results in the paper.

Reviewer_kMDZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how credit allocation in reinforcement learning with verifiable rewards (RLVR) shapes which solutions remain accessible under repeated sampling. The authors observe that group-relative objectives such as GRPO assign equal advantage to equally rewarded responses, making aggregate credit proportional to sampling frequency, so rare-but-correct modes receive systematically fewer updates—aggravating support concentration, entropy decline, and out-of-distribution forgetting. They propose ExPPO, a lightweight advantage-shaping rule that redistributes credit using prompt-relative, length-normalized response surprisal (group-standardized and tanh-bounded), gated by prompt pass rate: low-pass-rate groups upweight rare correct responses, while high-pass-rate groups strengthen penalties on common errors; a shared normalization then keeps each prompt group's total absolute advantage mass comparable, preserving verifier polarity. ExPPO plugs into GRPO and DAPO using existing rollout statistics with only O(G) overhead. The analysis, in a KL-local exponential-tilt model, establishes polarity preservation, surprisal-ordered response preference, empirical surprisal alignment, and derives local conditions for entropy gains and correct-mode discovery.

Critical strengths

S1: Lightweight and practical: reuses existing rollout log-probabilities with no extra sampling, only O(G) group reductions, and integrates cleanly into GRPO/DAPO.

S2: Well-motivated shaping: rarity emphasized in low-pass-rate groups, common errors in high-pass-rate groups; bounded shaping preserves verifier polarity, avoiding entropy bonuses' indiscriminate rewarding of unlikely tokens.

S3: Substantial theory: polarity preservation, surprisal ordering, empirical alignment, and local gains for entropy and correct-mode discovery are derived in a KL-local model.

S4: Carefully controlled protocol: matched budgets, three independent training seeds for Phi-3.5-mini, and paired-bootstrap CIs in the controlled evaluation.

Critical weaknesses

W1: Accuracy does not universally lead: 33.50 vs F-GRPO's 35.51 on Qwen2.5-7B, and only 10.59 vs REPO-R's 64.94 ID accuracy on Phi-3.5-mini; the abstract's "higher aggregate accuracy" claim needs qualification.

W2: The boundary with the closest baseline REPO-R is thin (ΔD +0.16; nearly identical OOD coverage on Phi-3.5-mini), so the novelty margin needs sharper analysis.

W3: No sensitivity analysis for λ, shared normalization, σ_min, or budget K—parameters that directly control the claimed gains.

W4: Small model scale: the largest backbone is 7B; the benefits are unvalidated on larger reasoning models where RLVR matters most.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for recognizing the method’s strengths and identifying areas needing clarification. We clarify the accuracy comparisons and gains over REPO-R, and narrow the abstract’s accuracy claim to competitive performance. We also summarize the existing sensitivity and sampling-budget results, add a numerical-floor check, and report new 14B experiments.

W1. Accuracy comparisons and the abstract claim

We agree that ExPPO does not have the highest accuracy in every comparison. On the controlled Qwen2.5-7B task, the original Table 2 reports accuracy of 33.50% versus F-GRPO's 35.51%. In the added step-900 evaluation with K=256K=256 responses per question on 64 questions, ExPPO discovers more distinct modes (5.000 versus 3.781) and effective modes (3.820 versus 2.238). A mode is a verified answer vector; effective modes are exponentiated correct-mode entropy, with zero for unsolved questions.

MethodDistinct correct modes/questionEffective correct modes/questionCorrect-pair diversity (%)
GRPO3.9692.69450.97
F-GRPO3.7812.23840.50
GRPO-UR2.2502.03838.55
Cue-GRPO3.6252.62950.28
REPO-R4.3443.54868.59
ExPPO5.0003.82069.29

Mode yields average over all 64 questions. Correct-pair diversity is the fraction of pairs of correct response samples, drawn without replacement, whose modes differ; repeated answers remain separate samples. It averages equally over the common questions with at least two correct responses under every method. All methods use the same response budget.

For Phi-3.5-mini, Table 4 gives 10.59 ± 0.27% ID accuracy for ExPPO and 9.64 ± 0.19% for REPO-R. Table 10 reports eight-benchmark aggregate accuracy of 31.43 ± 0.72% for ExPPO, 31.32 ± 2.37% for GRPO-UR, and 30.90 ± 0.08% for REPO-R.

We will revise the abstract to: “ExPPO improves repeated-sample coverage and correct-mode yield while maintaining competitive response accuracy.”

W2. How substantial is the improvement over REPO-R?

We agree that the original conditional-diversity gap of +0.16 points is small, and Phi OOD coverage is essentially tied. The larger gains are in ID coverage, Qwen OOD coverage, and correct-mode yield:

Original-paper comparisonREPO-RExPPODifference
Qwen2.5-7B ID coverage, Table 343.45 ± 0.9046.14 ± 1.94+2.69
Qwen2.5-7B OOD coverage, Table 362.17 ± 0.1876.96 ± 0.54+14.79
Phi ID coverage, Table 421.94 ± 1.8925.67 ± 1.31+3.73
Phi OOD coverage, Table 466.35 ± 0.3866.33 ± 1.59−0.02

Coverage is the percentage of questions solved by at least one response. The ID mean covers AIME24/25 (K=16K=16), Minerva (K=8K=8), and OlympiadBench (K=4K=4); OOD covers GPQA, MMLU-Pro, Safety, and TruthfulQA-MC (K=4K=4).

On the controlled Qwen task at K=256K=256 in the same added evaluation, REPO-R obtains 4.344 distinct / 3.548 effective modes, below ExPPO's results in W1. The larger-budget benchmark comparison is in W3 below.

REPO-R controls token entropy through feedback. ExPPO instead uses prompt-relative response scores, pass-rate gating, and shared normalization to control how the group advantage is allocated (Section 4 and Appendix E).

AuthorsOpenReview ↗

Response (2/2)

W3. Sensitivity to the coefficient, normalization, numerical floor, and sampling budget

Table 9 contains the coefficient sweep and component ablations:

Phi-3.5-mini, step 300ID coverage (%)OOD coverage (%)
GRPO16.6853.15
λ=0.10λ=0.1021.6862.56
λ=0.20λ=0.20, default25.6766.33
λ=0.30λ=0.3021.8370.23
λ=0.50λ=0.5022.5568.67
λ=0.75λ=0.7522.8171.53
λ=1.00λ=1.0023.7066.32
Without final normalization, λ=0.20λ=0.2022.0265.85
Without pass-rate gate, λ=0.20λ=0.2022.8365.89
Without tanh bound, λ=0.20λ=0.2022.7265.44

The default row is the three-training-seed mean; the other settings are single runs. All tested positive coefficients exceed GRPO on both averages. The default is strongest on ID, while larger coefficients can improve OOD further.

We also recomputed advantages on 20 saved banks, each containing 1,024 groups of size 16:

Numerical floorMaximum relative group-advantage L1L_1 change
10−10,10−8,10−6,10−4,10−310^{-10},10^{-8},10^{-6},10^{-4},10^{-3}0%
0.013.76%
0.18.28%

The advantages are unchanged across the small-floor range.

Figures 3 and 4 examine sampling budgets. At shared K=128K=128, Figure 4 reports:

MethodFour-task Phi ID coverage (%)
F-GRPO12.17
GRPO-UR44.34
REPO-R40.30
ExPPO51.03

ExPPO's lead over REPO-R is 10.73 points at this budget.

W4. Does ExPPO help above 7B?

We added Qwen2.5-14B-Instruct results after mathematics training, using checkpoint 300.

DomainBenchmarkBase coverage (%)GRPO coverage (%)ExPPO coverage (%)
IDAIME24, K=16K=1635.5643.3346.67
IDAIME25, K=16K=1632.2233.3340.00
IDOlympiadBench, K=4K=457.5353.4859.21
IDMacro, 3 benchmarks41.7743.3848.63
OODGPQA, K=4K=460.3460.0463.84
OODSafety, K=4K=485.8083.8085.60
OODTruthfulQA-MC, K=4K=480.9580.8583.48
OODMacro, 3 benchmarks75.7074.9077.64

ExPPO improves over GRPO on all six benchmarks, by 5.25 ID / 2.74 OOD coverage points on the three-task averages.

Reviewer_kMDZOpenReview ↗

maintain my original score.

I have read the authors' response and acknowledge that it addresses my questions. However, it does not change my evaluation, and I maintain my original score.

Thank you for your constructive review and feedback.

Reviewer_qRfEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates exploration degradation in GRPO, where equally rewarded responses receive identical advantages, potentially favoring frequently sampled response modes over rare correct solutions. To address this issue, the authors propose ExPPO, an advantage-shaping method that redistributes response-level credit using prompt-relative surprisal and group pass rate, with bounded shaping and shared normalization. Theoretical analyses establish several properties of the proposed rule and characterize conditional local effects on entropy and correct-mode discovery. Experiments across three language models and mathematical and code reasoning tasks report improvements in coverage, response accuracy, and correct-mode diversity compared with several GRPO-based baselines.

Critical strengths

S1. A practical and lightweight approach to credit redistribution. The paper addresses a relevant issue in GRPO through response-level advantage shaping. The proposed method is relatively simple to implement, introduces limited computational overhead, and can be integrated into existing RLVR frameworks.

S2. Relatively broad empirical evaluation with targeted exploration analysis. The experiments cover multiple models, reasoning tasks, and ID/OOD benchmarks. The controlled multi-answer evaluation and additional ablations provide useful evidence regarding correct-mode discovery and response-specific credit allocation.

Critical weaknesses

W1. Limited conceptual novelty beyond existing approaches. The problems of distribution sharpening and exploration degradation, as well as rarity-aware credit assignment, advantage modification, and prompt-success-dependent updates, have close precedents. Although ExPPO offers a lightweight integration of these ideas and demonstrates empirical benefits over several related methods, the paper does not sufficiently establish why this particular combination constitutes a substantive conceptual advance rather than an effective integration of established techniques. This limits the perceived novelty of the contribution, while acknowledging that some relevant studies may be concurrent.

W2. Insufficient justification and validation of the exploration signal and gating design. Length-normalized response surprisal measures average token-level unexpectedness under the rollout policy, but need not reliably reflect the rarity of a semantic solution mode or its exploration value. High surprisal may result from unusual wording, local errors, or incoherent content rather than a genuinely distinct reasoning strategy. Although the controlled evaluation demonstrates improvements in correct-answer diversity, it does not directly validate this proposed connection. In particular, the rationale for softening penalties on high-surprisal incorrect responses remains insufficiently established, as preserving such responses may not necessarily facilitate subsequent discovery of correct solutions. Moreover, while the ablation results support the usefulness of pass-rate gating, they do not establish why the proposed gating direction is preferable to alternative schedules, such as prioritizing the consolidation of reliable solutions at low pass rates. Overall, the central design remains substantially heuristic, with limited evidence isolating the mechanisms underlying its empirical gains.

Questions for authors
  1. Could the authors clarify the substantive conceptual contribution of ExPPO relative to existing rarity-aware credit assignment, advantage shaping, and prompt-success-dependent methods, while accounting for potentially concurrent work?

  2. Under what conditions does length-normalized response surprisal reliably indicate semantic solution rarity or useful exploration? Could the authors provide more direct evidence distinguishing uncommon reasoning strategies from unusual wording or other sources of token-level unpredictability?

  3. What evidence supports the benefit of softening penalties on high-surprisal incorrect responses? In particular, how can this mechanism be distinguished from merely preserving probability mass on unproductive failure modes?

  4. While Table 9 supports pass-rate gating relative to removing it, why is the proposed gating direction preferable to alternative schedules, such as reversed or constant gating?

  5. It would be helpful to provide empirical evidence on whether the positive residual alignment condition underlying Eq. (19) holds at different stages of training, perhaps using the controlled multi-answer setting where correct modes can be explicitly identified.

3 comments
AuthorsOpenReview ↗

Response (1/3)

Thank you for your careful review. We clarify the contribution and add a Cue-GRPO comparison, direct rarity tests, and measurements of the theoretical alignment condition. We also test alternative gates and rare-negative attenuation, explaining the latter through a clip mechanism motivation.

W1/Q1. What is new beyond rarity-aware credit and success-rate shaping?

ExPPO controls how a prompt group's learning signal is divided among correct and incorrect responses, while approximately preserving its total absolute sequence advantage. The difference from related methods is the allocation rule:

MethodHow it allocates learning signal
GRPO-HAdds a token-entropy bonus.
GRPO-URUses response-likelihood ranks to favor unlikely correct responses through reward shaping.
F-GRPOScales all advantages in a prompt group by the same success-dependent factor.
REPO-RRescales token advantages using token log probability and entropy feedback.
Cue-GRPOReweights correct responses using structural strategy clusters and their frequencies.
UCPOAdds a conditional-uniformity penalty over correct outputs.
ExPPOApplies bounded, success-gated corrections to both reward classes using prompt-relative response surprisal, then normalizes their total absolute advantage.

ExPPO needs no strategy labels or clustering. Proposition 2 establishes advantage-sign preservation, and Eq. (19) identifies when the correction improves local correct-mode discovery at a matched KL budget. Rarity weighting and rare-negative attenuation have precedents, including Delightful Policy Gradient [2]; our contribution is this allocation rule and its theoretical and empirical analysis.

We added a Cue-GRPO comparison using Qwen2.5-7B step-900 checkpoints on 64 multi-answer questions, with 256 responses per question. Each verified final answer vector defines a mode.

MethodDistinct correct modes/questionEffective correct modes/questionCorrect-pair diversity (%)
Cue-GRPO3.6252.62950.28
ExPPO5.0003.82069.29

Effective modes are exponentiated correct-mode entropy, with zero for unsolved questions. Mode yields average over all 64 questions. Correct-pair diversity is the fraction of pairs of correct response samples, drawn without replacement, whose modes differ; it averages equally over common questions with at least two correct responses under every method. ExPPO improves all three measures against this explicit strategy-rarity baseline.

W2/Q2. Does surprisal distinguish solution methods from wording?

Surprisal reflects both mode probability and the probability of a particular wording within that mode. It is more informative about mode rarity when lengths and within-mode wording probabilities are comparable.

Mode frequency. On the 64-question multi-answer task, we use 256 responses per question. We report partial Pearson correlations between surprisal and negative log estimated mode probability, adjusting both for log length and formatting controls. We calculate correlations among correct responses within each question and fold, average the two reciprocal-fold estimates, then average equally over eligible questions.

ExPPO 7B stepEligible questionsPartial Pearson correlationBetween-mode variance share
10060/64+0.24042.9%
50064/64+0.22643.3%
90050/64+0.25341.1%

Rarer correct modes have higher surprisal on average at all three checkpoints. Mode identity explains 41–43% of the variance, so within-mode variation also matters.

Methods versus paraphrases. We constructed 64 mathematical questions with two verified methods and four paraphrases per method, matching lengths within 10% under both tokenizers. Method comparisons hold wording style fixed; wording comparisons hold the method fixed. After adjusting for log length, we report the mean question-level contrast:

C=(dm−dw)/(dm+dw)C = (d_m - d_w)/(d_m + d_w)

Here dmd_m and dwd_w are mean squared surprisal differences from changing the method and wording, respectively. Positive values mean a larger method effect.

7B checkpoint 900Length-adjusted contrastPositive questionsExact-length subset contrast, 39 questions
ExPPO+0.55860/64+0.793
GRPO+0.53260/64+0.745

The questions cover linear systems, quadratics, integer sums, and GCD; the exact-length subset covers the last three. Both models distinguish methods from paraphrases in this test.

AuthorsOpenReview ↗

Response (2/3)

W2/Q3. Does softening rare incorrect responses help discover correct solutions?

We removed the high-surprisal incorrect-response correction while retaining shared normalization. We reran this ablation on another cluster; the “Remove rare-incorrect” column reports the rerun. The table also includes the gate variants discussed in Q4. All runs use Phi-3.5-mini at step 300 with λ=0.2λ=0.2.

Coverage is the percentage of questions with at least one correct response at the stated budget KK. Macro rows are unweighted means of the four benchmarks in each domain. For “Without pass-rate gate,” it equals Constant 1.

DomainBenchmarkFull ExPPO (paper)Remove rare-incorrectReversed gateConstant 0.5Without pass-rate gate
IDAIME24, K=16K=1617.783.330.003.3316.67
IDAIME25, K=16K=1613.3310.000.000.003.33
IDMinerva, K=8K=844.1240.0712.1338.2443.75
IDOlympiadBench, K=4K=427.4625.334.8920.0027.56
IDMacro25.6719.684.2615.3922.83
OODGPQA, K=4K=448.5148.4430.5837.0546.65
OODMMLU-Pro, K=4K=469.5265.7137.1448.5765.71
OODSafety, K=4K=475.5373.6069.3072.8076.20
OODTruthfulQA-MC, K=4K=471.7470.0350.1559.0675.00
OODMacro66.3364.4546.7954.3765.89

Full ExPPO has higher coverage on every benchmark than the rare-incorrect ablation. A separate matched comparison also favors retaining the correction, by 9.85 ID / 9.23 OOD coverage points. On the multi-answer task, those two checkpoints give the following results at 256 responses per question:

MethodDistinct correct modes/questionEffective correct modes/question
Full ExPPO, fresh control0.6720.600
Remove rare-incorrect correction0.6090.572

Retaining the correction increases correct-mode discovery. OOD response accuracy is lower in the matched comparison, showing a tradeoff between response accuracy and coverage.

Why soften penalties on rare incorrect responses? In PPO/GRPO, the clipped objective is LCLIP(θ)=E^t[min⁡(rt(θ)A^t,clip⁡(rt(θ),1−ϵ,1+ϵ)A^t)]L^{\mathrm{CLIP}}(\theta)=\hat{\mathbb{E}}_t\left[\min\left(r_t(\theta)\hat{A}_t,\operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t\right)\right]. For negative advantages (A^t<0\hat{A}_t<0), the minimum selects the more negative term, equivalently A^tmax⁡(rt(θ),1−ϵ)\hat{A}_t\max(r_t(\theta),1-\epsilon). Thus, the negative contribution remains unclipped when rt(θ)>1+ϵr_t(\theta)>1+\epsilon, while repeated negative updates across optimization steps can continue suppressing incorrect responses even after individual updates reach their clipping limits, and repeatedly applying strong negative updates to already unlikely responses can push their probabilities toward zero, removing potentially useful exploration paths. ExPPO reduces the penalty for high-surprisal incorrect responses while keeping their advantages negative. This prevents premature suppression of rare responses without rewarding errors, allowing the policy to retain alternative paths that may lead to correct solutions later. This motivation is consistent with analyses of [1] and [2], the ablation tests its benefit here.

W2/Q4. Why this gate rather than reversed or constant gating?

The proposed gate has the highest reported ID and OOD macro coverage in the Q3 table. Reversed gating has the lowest macro coverage, while constant 0.5 gating and the no-gate ablation lie between these settings.

The gate is g=1−ρg=1-ρ for above-average-surprisal responses and g=ρg=ρ for below-average-surprisal responses, where ρρ is the prompt pass rate. Its direction preserves learning from successes on difficult prompts and reduces repeated reinforcement of common successes on easy prompts:

SituationGateEffect before shared normalization, λ=0.2λ=0.2
Hard prompt, ρ=0.1ρ=0.1: common correct0.1Retains at least 98% of positive credit.
Hard prompt, ρ=0.1ρ=0.1: rare incorrect0.9Softens the penalty by up to 18%.
Easy prompt, ρ=0.9ρ=0.9: common correct0.9Reduces repeated positive credit by up to 18%.
Easy prompt, ρ=0.9ρ=0.9: rare incorrect0.1Retains at least 98% of the penalty.

Rare correct responses receive their strongest boost at low pass rates; common incorrect responses receive their strongest penalty increase at high pass rates.

AuthorsOpenReview ↗

Response (3/3)

Q5. Is the residual-alignment condition in Eq. (19) observed during training?

We measured the covariance between the residual rarity correction and the discovery utility V128V_{128} at steps 100, 500, and 900. This test uses 64 questions, 64 targets per question; targets preserve reward proportions and within-reward length bins.

Mode probabilities use all 248 responses in the opposite reference fold, with additive 0.5 smoothing over every valid mode and one failure category. Within each question and target fold, we remove the correction's component aligned with the GRPO advantage, then calculate covariance over both correct and incorrect responses. We average the two folds and then the 64 questions equally. Rarity correlations use the estimator defined in Q2.

ExPPO 7B stepResidual-alignment covariance, K=128K=128Rarity correlation
100+0.232+0.257
500+0.372+0.221
900+0.221+0.186

The covariance is positive at all three stages, supporting the alignment condition in this experiment. Eq. (19) connects this condition to local gains in correct-mode discovery at matched KL for small positive shaping strength.

[1] Zhang et al. Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs. ICLR 2026

[2] Ian Osband et al. Delightful policy gradient

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

General Response

First, we would like to thank all reviewers for their careful and constructive feedback on our work. During the rebuttal period, we conducted additional experiments to address the concerns or questions posed by the reviewers.

We summarize the reviewers’ concerns and the evidence provided in our rebuttal below.

Reviewer qRfE — contribution, rarity signal, and design justification. We clarified ExPPO’s contribution through a comparison with related methods: bounded, sign-preserving credit redistribution across correct and incorrect responses under a shared absolute-advantage budget. We added direct rarity and wording tests, rare-incorrect ablations, and reversed/constant gate comparisons. Full ExPPO achieves 25.67% ID coverage, compared with 19.68% without rare-incorrect attenuation and 4.26% with reversed gating. We also measured positive residual alignment at steps 100, 500, and 900, providing empirical support for the condition in Eq. (19).

Reviewer kMDZ — accuracy, gains over REPO-R, sensitivity, and model scale. We corrected the Phi ID accuracy comparison to 10.59% versus REPO-R’s 9.64% and narrowed the abstract’s claim to competitive accuracy. We highlighted larger coverage gains, including +10.73 points over REPO-R at K=128, summarized the coefficient, normalization, and sampling-budget analyses, and added a numerical-floor check. New 14B experiments improve over GRPO on all six evaluated benchmarks, with +5.25 ID / +2.74 OOD coverage points.

Reviewer GBSc — semantic rarity, wording sensitivity, and diversity-oriented baselines. We added direct mode-frequency correlations and within-/between-mode analyses. Correlations remain positive after controlling for length and formatting, and method changes affect surprisal more than paraphrasing on 60/64 mathematical questions. We also added Cue-GRPO, which explicitly reweights correct-strategy clusters. ExPPO achieves 5.000 versus 3.625 distinct correct modes per question and 69.29% versus 50.28% correct-pair diversity, and leads all three reported mode-discovery and diversity measures across the six compared methods.

Reviewer kMDZ subsequently confirmed that our response addresses their questions and maintained their weak-acceptance score.

Reviewer GBSc found the additional analyses helpful and maintained their rating of 3 (weak acceptance). We will conduct further experiments to validate the rarity signal with more diverse tasks.

We hope this summary helps the committee assess the added evidence and clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.