- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Exploration-Preserving Policy Optimization (ExPPO), motivated by the observation that group-relative RL can concentrate credit on solution modes that are already sampled frequently. ExPPO reallocates response-level advantage using a prompt-relative rarity signal based on length-normalized sequence surprisal together with the prompt pass rate. The shaping preserves the sign of the original advantage while increasing credit for relatively surprising correct responses and modifying the treatment of other common/rare correct and incorrect responses. The method is evaluated on a controlled multi-answer task, mathematical reasoning benchmarks, and coding tasks, where it generally improves correct-solution coverage and diversity relative to GRPO and entropy-regularized baselines.
Critical strengths
I find the paper's problem formulation compelling. In particular, it makes an important distinction between maintaining generic policy entropy and preserving diversity within the support of correct solutions. For reasoning models trained with verifier-based rewards, the goal is not simply to keep the policy uncertain, but to prevent premature concentration on a small subset of already-discovered correct reasoning modes. This perspective is valuable because conventional entropy regularization is reward-agnostic: it may encourage exploration of unlikely incorrect continuations as well as useful alternative solutions, whereas ExPPO explicitly reallocates verifier-conditioned credit toward responses that are both promising and underrepresented.
The proposed formulation is also practically appealing because it modifies the existing GRPO training process using statistics that are already available during rollout generation, without requiring additional reward models or external evaluation mechanisms. The sign-preserving advantage reshaping is a reasonable design choice, as it aims to improve exploration while maintaining the original distinction between correct and incorrect responses.
The controlled multi-answer experiment is another strong aspect of the validation. Since the task provides multiple verifiably correct semantic solution modes, it allows the authors to evaluate whether the method improves correct-mode discovery rather than merely increasing token-level entropy or surface-form diversity. This provides a more meaningful evaluation of exploration behavior in reasoning RL than standard entropy-based metrics alone.
Critical weaknesses
My main concern is that the empirical validation does not fully establish the connection between the proposed rarity signal and the phenomenon the paper aims to address. ExPPO is motivated by preserving diversity among semantic solution modes, but the method operates on response-level length-normalized sequence surprisal. A low-probability trajectory is not necessarily a rare reasoning strategy: different surface realizations of the same underlying solution approach can have substantially different surprisal values, while a genuinely distinct reasoning mode may have high aggregate probability due to multiple paraphrased realizations. Therefore, the current formulation may conflate trajectory-level novelty with semantic-level diversity.
In addition, the evaluation does not fully rule out simpler explanations for the observed improvements. Since ExPPO modifies the advantage distribution to favor rare successful trajectories, the gains could partly come from a general exploration incentive rather than specifically addressing correct-mode collapse. Stronger comparisons against methods that explicitly optimize solution diversity or coverage would help clarify whether ExPPO provides benefits beyond generic exploration regularization.
Questions for authors
-
On the controlled multi-answer task, can the authors directly measure the relationship between response surprisal and empirical semantic-mode frequency? For example, reporting correlation between surprisal and negative log mode frequency, or within-mode versus between-mode surprisal variation, would materially affect my confidence in the central rarity proxy.
-
Since surprisal is computed at the response level, how sensitive is ExPPO to surface-form variation? For example, if two responses correspond to the same reasoning strategy but differ substantially in wording or verbosity, does ExPPO risk allocating additional credit to stylistic variants rather than genuinely different solution modes?
3 comments
Thank you for the additional mode-frequency analyses, paraphrase controls, and diversity-baseline comparisons. These results help address my concerns about the rarity signal, although its generalization beyond the controlled tasks remains uncertain. I maintain my original rating of 3.
Thank you for the follow-up. We agree that the rarity signal needs further validation beyond controlled tasks. Our paper also evaluates broader OOD performance on GPQA, MMLU-Pro, Safety, and TruthfulQA-MC across different models and training settings. We will conduct more experiments on broader tasks to directly evaluate the rarity signal and report the results in the paper.
Thank you for highlighting the gap between response surprisal and semantic solution rarity. We add direct mode-frequency correlations, controlled method-versus-paraphrase tests, and a whitespace sensitivity check. We also add Cue-GRPO and compare correct-mode yield and diversity across methods at the same sampling budget.
W1/Q1/Q2. Mode rarity, reasoning methods, and surface-form sensitivity
Surprisal reflects both the probability of a mode and the probability of a wording within that mode. We added direct frequency measurements and controlled tests of mathematical methods and paraphrases, using mathematics-trained Qwen2.5-7B checkpoints.
Mode frequency. Each verified final answer vector defines a mode. On 64 multi-answer questions, we use 256 responses per question with a 2,048-token cap. We calculate partial Pearson correlations between surprisal and negative log estimated mode probability, adjusting both for log length and formatting controls. Correlations use correct responses within each question and fold; we average the two reciprocal-fold estimates, then average equally over eligible questions.
The positive correlations show that rarer correct modes tend to have higher surprisal. Mode identity accounts for 41–43% of the variance.
Methods versus paraphrases. We constructed 64 mathematical questions with two verified methods and four paraphrases per method, matching lengths within 10% under both tokenizers. Method comparisons hold wording style fixed, and wording comparisons hold the method fixed. Both strategy tests below use step-900 checkpoints.
After adjusting for log length, we average the question-level contrast C=(dm−dw)/(dm+dw), where dm and dw are mean squared surprisal differences from changing the method and wording. Positive values mean a larger method effect.
Within- and between-strategy variation. A separate GCD test uses 64 questions, three verified strategies per question, and four paraphrases per strategy:
Both models are sensitive to differences between the tested methods.
Surface edits can still change credit. A whitespace-only control, with reasoning and group normalization fixed, gives a mean absolute change of 0.0177 normalized advantage units over 64 anchors from 45 questions. ExPPO bounds the raw correction by λ∣Ai∣ and preserves the advantage sign (Eqs. (8)–(9)).
W2. Comparisons with methods targeting diversity or coverage
We added Cue-GRPO, which explicitly reweights correct responses by strategy-cluster rarity. The comparison also includes GRPO-UR's emphasis on unlikely correct responses, F-GRPO's success-rate weighting, and REPO-R's entropy feedback.
We use the existing step-900 checkpoints and the first 256 responses per question on the same 64-question task.
Effective modes are exponentiated correct-mode entropy, with zero for unsolved questions. Mode yields average over all 64 questions. Correct-pair diversity is the fraction of pairs of correct response samples, drawn without replacement, whose modes differ; repeated answers remain separate samples. It averages equally over the common questions with at least two correct responses under every method. All methods use the same response budget.
ExPPO leads all three measures. Compared with Cue-GRPO, it discovers 1.375 more distinct / 1.191 more effective modes per question, with 19.01 points higher correct-pair diversity. Compared with REPO-R, conditional diversity is close, while mode yields are higher.
The paper also reports a separate Phi-3.5-mini comparison at shared K=128 across AIME24, AIME25, Minerva, and OlympiadBench (Figure 4). Coverage is the percentage of questions with at least one correct response, averaged equally across benchmarks: