acceptodds
Under review as a conference paper at ICLR 2027

Exploration Prior: Semantically Guided Exploration for Reinforcement Learning of Language Model Reasoning

Abstract

Reinforcement learning with verifiable rewards (RLVR) substantially improves the reasoning capabilities of large language models, yet effective exploration remains challenging as policy optimization can progressively concentrate probability mass within a narrow region of the continuation space. We introduce Exploration Prior (EP), a framework that uses the model's natural-language conditioning capability to provide semantic guidance during RL training. EP uses lightweight, answer-free path-switching instructions to construct an intervention-conditioned continuation distribution as an exploration prior. Rather than specifying a correct reasoning path, the intervention shifts the continuation distribution toward alternative reasoning directions, while task rewards determine which sampled continuations are reinforced through policy optimization. From a probabilistic-inference perspective, the reward tilts this prior toward high-reward continuations, yielding a posterior that is approximated through reverse-KL-regularized policy optimization. Across five mathematical reasoning benchmarks and three model backbones, EP consistently outperforms standard RLVR and representative exploration- and prior-based baselines. On Qwen3-4B, EP improves the mean Avg@16 from 46.6% to 51.9% and the mean Pass@16 from 67.8% to 71.5% over GRPO. Moreover, its margin over GRPO in Pass@ becomes more pronounced at larger sampling budgets, indicating broader coverage of successful reasoning trajectories.

Primary area:
reinforcement learning
Keywords:
large language models, reasoning, exploration, RLVR

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_wdKiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates effective exploration in reinforcement learning with verifiable rewards (RLVR), where policy optimization may progressively concentrate probability mass on a narrow set of reasoning trajectories, limiting exploration. To address this issue, the authors propose Exploration Prior (EP), which leverages the language model's natural-language conditioning capability to provide semantic guidance. Specifically, EP constructs an exploration prior using answer-free path-switching instructions. From a probabilistic inference perspective, task rewards tilt this prior toward high-reward reasoning trajectories, yielding a posterior distribution that is approximated through reverse-KL-regularized policy optimization. Experiments across multiple mathematical reasoning benchmarks and model backbones demonstrate improvements in reasoning performance, trajectory diversity, and successful-solution coverage.

Critical strengths
  1. This paper addresses the challenge of insufficient exploration in RLVR training and explores natural-language conditioning as a new perspective for guiding reasoning exploration.

  2. This paper demonstrates performance improvements across multiple mathematical reasoning benchmarks and model backbones, supported by exploration analyses, ablation studies, and hyperparameter sensitivity analyses.

Critical weaknesses
  1. This paper argues that existing RLVR exploration methods primarily rely on entropy, uncertainty, and trajectory diversity signals but lack explicit semantic guidance, thereby motivating the proposed Exploration Prior (EP). However, this motivation lacks sufficient theoretical justification and empirical support. First, the paper fails to clearly define effective exploration or clarify the relationship between trajectory diversity, reasoning path coverage, and exploration effectiveness. Second, the absence of explicit semantic guidance does not imply that existing methods cannot discover diverse, high-reward reasoning trajectories. The authors neither thoroughly analyze the specific failure modes of existing approaches nor demonstrate why semantic guidance would lead to more valuable exploration. Consequently, the paper fails to establish a convincing connection between the substantive limitations of existing methods and the design of EP, leaving its core motivation and necessity insufficiently justified.
  2. This paper constructs an exploration prior through natural-language interventions and uses the resulting token-level probability shifts to guide policy updates. However, the authors do not adequately explain how these probability shifts translate into effective reasoning exploration. Path-switching instructions may induce changes in textual expression, reasoning strategies, or error correction, but these behaviors do not necessarily provide the same exploration value. Although task rewards are incorporated into policy optimization, the paper lacks a direct analysis of the reasoning behaviors induced by different interventions and their associated reward improvements. Moreover, the existing textual diversity and Pass@K analyses are insufficient to demonstrate that EP discovers more valuable reasoning trajectories than existing exploration methods. Consequently, the effectiveness and distinctive contribution of the proposed path-switching mechanism remain insufficiently supported by empirical evidence.
Questions for authors

see Critical Weaknesses

2 comments

Response to W1: Motivation and Effective Exploration(Part 1/2)

Thank you for your insightful suggestion, which helps us clarify our methodology and strengthen the empirical analysis. We agree that the lack of explicit semantic guidance does not imply that existing methods cannot discover diverse, high-reward trajectories. Our motivation is instead based on the limitations of their exploration signals.

1. Definition of effective exploration. We define effective exploration as the ability to discover high-reward reasoning trajectories under a limited sampling budget. Trajectory diversity measures differences between generated sequences, whereas strategy diversity captures differences in the underlying reasoning approaches. Exploring distinct reasoning strategies can facilitate the discovery of successful solutions. However, greater trajectory diversity does not necessarily imply broader strategy coverage, as diverse trajectories may still follow the same underlying reasoning approach [2].

2. Limitations of entropy-based exploration. Section 4.3 (“Relation to Entropy Regularization”) shows that, given a problem and a fixed prefix, an entropy bonus over a fixed finite set of reasoning continuations is equivalent, up to an additive constant, to a reverse-KL penalty toward a uniform prior over that set. It therefore does not explicitly distinguish between semantically different exploration directions, such as rephrasing an existing solution and attempting an alternative reasoning approach.

3. Limitations of diversity-based exploration. Methods such as DIVER [1] encourage exploration through textual and equational diversity rewards. However, these trajectory-level signals do not necessarily reflect differences in underlying reasoning strategies. A recent study [2] provides empirical evidence that DIVER primarily increases diversity within existing approaches rather than expanding the coverage of distinct reasoning strategies. These findings highlight a potential gap between trajectory-level diversity and strategy-level exploration.

4. Motivation of EP. These limitations motivate us to introduce an explicit semantic preference into the exploration prior. Rather than indiscriminately encouraging randomness or trajectory dissimilarity, EP uses path-switching instructions to bias the policy toward alternative reasoning continuations. Verifiable rewards complement this guidance by reinforcing successful trajectories. EP thus provides a directional exploration signal without requiring knowledge of the correct solution.

5. Additional empirical evidence. We examine how the choice of instruction affects Qwen3-4B performance and measure strategy coverage at four training checkpoints.

First, we replace path-switching instructions with general reasoning instructions (e.g., “Continue reasoning”), while keeping the remaining configuration unchanged. The table also includes a path-switching variant that removes explicit self-correction cues, discussed in W2. Each cell reports Avg@16 / Pass@16 (%). The mean gives equal weight to each benchmark.

MethodAIME 2024AIME 2025AMC23Minerva MathHMMT Feb 2025Mean
GRPO45.2 / 76.736.7 / 66.786.6 / 97.542.9 / 54.821.7 / 43.346.6 / 67.8
General instruction46.5 / 73.342.3 / 73.387.3 / 97.544.2 / 55.518.3 / 43.347.7 / 68.6
EP (original)54.4 / 76.745.4 / 73.389.8 / 97.544.4 / 56.625.6 / 53.351.9 / 71.5
EP (without self-correction cues)57.9 / 80.046.7 / 76.789.1 / 97.542.6 / 55.525.2 / 46.752.3 / 71.3

The original EP instructions outperform general reasoning instructions by 4.2 percentage points in mean Avg@16 and 2.9 points in mean Pass@16.

Second, we analyze the evolution of strategy diversity throughout training. Following the strategy-identification criteria in [2], we distinguish mathematical strategies based on three dimensions: Mathematical Tools, Definitions/Structures, and Representational Viewpoints. We use DeepSeek Flash v4.1 as the judge model to identify and categorize distinct strategies. At each step, we sample 8 rollouts for each of the 64 problems. To reduce the confounding effect of differences in accuracy, we primarily report strategy diversity on problems with at least two correct trajectories, ensuring a sufficient number of successful samples for meaningful strategy comparison. We additionally report results across all problems with at least one correct trajectory. Importantly, our strategy identification allows multiple distinct strategies to be counted within a single trajectory, rather than restricting each trajectory to a single strategy.

Response to W1: Motivation and Effective Exploration(Part 2/2)

MethodStrategy diversity (problems with ≥2 correct trajectories)Strategy diversity (all problems with correct trajectories)
step30step60step120step200step30step60step120step200
GRPO1.231.331.261.291.211.341.241.28
GRPO+Entropy1.321.311.251.271.331.261.251.26
Neutral General Instruction(Continue Reasoning.)1.351.241.281.281.281.241.261.28
EP1.351.441.241.271.321.481.231.25

We observe a strategy coverage advantage at Step 60. Among problems with at least two correct trajectories, EP covers 1.44 distinct strategies per problem on average, compared with 1.33 for GRPO, 1.31 for GRPO+Entropy, and 1.24 for general instructions. The same ordering holds when including problems with at least one correct trajectory. Together with the higher benchmark Pass@16 at the selected checkpoint, these results provide complementary evidence that EP promotes strategy-level exploration during training and improves the effectiveness of the learned policy under a fixed inference sampling budget.

We will clarify these distinctions and incorporate the additional analyses in the revised manuscript.

Criteria for identifying distinct strategies. For each problem, the judge identifies and groups mathematically equivalent approaches across all correct trajectories. Following the criteria in [2], distinct strategies are identified based on three dimensions:

  • Mathematical tools (e.g., calculus or geometry).
  • Definitions or structures (e.g., explicit formulas or recurrence relations).
  • Representational viewpoints (e.g., reasoning about a geometric locus or expressing constraints in vector form).

References

[1] Hu et al. Diversity-Incentivized Exploration for Versatile Reasoning. ICLR 2026.

[2] Lee et al. Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning. EMNLP 2026.

Response to W2: Mechanism and Effectiveness of Path-Switching Guidance

Thank you for raising the possibility that a path-switching instruction can also prompt the model to check its work. We address this through the policy update and an additional instruction ablation.

1. How the prior enters the policy update. Each sampled token receives the trajectory's reward-based GRPO advantage. From the intervention position onward, EP adds a scaled, clipped log-probability ratio comparing the instruction-conditioned prior with the current policy on the same sampled token. The sum of the two terms enters the GRPO update. Because the probability of a continuation is the product of its conditional token probabilities, these updates can change which reasoning paths the policy samples in subsequent rollouts.

2. Removing explicit self-correction cues. Some of the original path-switching instructions also incorporate self-correction cues. We remove these cues and replace the original instruction set with the following twelve path-switching instructions.

Try a different approach.
Use another method to solve the problem.
Explore a different reasoning path.
Consider another solution strategy.
Switch to a different approach.
Solve the problem from a different perspective.
Adopt a different reasoning strategy.
Use a different mathematical approach.
Explore another way to solve the problem.
Proceed with a different solution method.
Reformulate the problem from a different perspective.
Pursue a different line of reasoning.

Across the five benchmarks in W1, removing explicit self-correction cues yields 52.3 Avg@16 and 71.3 Pass@16, close to the original EP scores of 51.9 and 71.5. These results suggest that the benefits of path-switching guidance do not depend on explicit self-correction cues.

3. Evidence of effective strategy-level exploration. Under the same sampling budget, EP improves mean Pass@16 from 67.8 to 71.5 and mean Avg@16 from 46.6 to 51.9 over GRPO. We agree that these metrics alone cannot distinguish broader strategy coverage from improved reliability within existing strategies.

To address this limitation, the additional strategy-level analysis at training step 60, reported in our response to W1, shows that EP covers more distinct mathematical strategies among correct rollouts than GRPO, GRPO+Entropy, and the general-instruction variant.

Together, these results provide complementary evidence that path-switching guidance improves exploration effectiveness and broadens strategy coverage among successful trajectories. A finer-grained analysis of different intervention-induced reasoning behaviors and their individual contributions to reward improvements is left for future work.

We will incorporate these clarifications and additional analyses into the revised manuscript.

Reviewer_xkWMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the loss of reasoning diversity during reinforcement learning with verifiable rewards. Exploration Prior uses instructions such as “try a different approach” to guide GRPO training. The model first generates responses normally. A frozen copy of the current model then scores the same responses with an instruction inserted partway through the reasoning. The resulting changes in token probabilities provide an additional training signal. This guidance is gradually removed during training, and no additional model or instruction is needed at inference time.

Critical strengths

S1. I find the idea intuitively reasonable. Their method can enable exploration through a simple reasoning instruction inserted during training to weigh the importance of learning signals.

S2. The reported gains are meaningful. The improvements at larger sampling budgets are encouraging because they suggest that the benefit extends beyond making already-likely answers more frequent.

Critical weaknesses

W1. It is unclear whether encouraging the model to switch approaches is what drives the improvement. The authors compare their instruction with unrelated text, such as “The room stays quiet.” However, they do not compare it with a general reasoning instruction, such as “Continue reasoning carefully.” Would this simpler instruction work just as well? This comparison would help show whether the benefit comes specifically from encouraging alternative approaches or more generally from using a helpful instruction to guide training.

It is also unclear whether EP finds different ways to solve a problem or simply produces differently worded responses. Higher Pass@K shows better success with multiple attempts, but this could come from using the same approach more reliably. The BLEU-based measure mainly captures differences in wording. I would like to see an analysis of the generated solutions showing whether EP actually uses a wider range of valid approaches to the same problem.

W2. The problem studied is a known phenomenon in the literature. I agree that the EP method is one way to mitigate the diversity collapse issue, but its novelty is my main concern.

It is unclear whether EP offers an improvement over closely related methods. I could find the following methods at first glance. ICE [1] already uses reasoning instructions to encourage exploration during RL, without requiring those instructions at inference time. OPCD [2] scores model-generated responses using a teacher-given additional context. SAGE [3] also modifies the reference distribution in the KL penalty to encourage exploration.

EP’s main difference seems to be using a “try another approach” instruction partway through a response to guide GRPO updates. I find this interesting, but the paper does not compare it directly with these alternatives. The SDPO comparison also needs more explanation. The authors use a modified version, and it is unclear whether its poor performance comes from the method itself or how it was implemented and tuned. I would like to see a comparison with a close alternative under the same training setup, including the additional training cost of EP.

[1] Dilkes et al. Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy.

[2] Ye et al. On-Policy Context Distillation for Language Models.

[3] Lee et al. SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs.

Questions for authors

Q1. What do the authors see as the main methodological advance over instruction-conditioned exploration and on-policy context distillation combined with GRPO?

Q2. Can the authors clarify how GRPO+SDPO was implemented and tuned, particularly the construction of its teacher context and the strength and schedule of its training signal?

Q3. Have the authors compared the path-switching instruction with a general reasoning instruction while keeping everything else fixed? Do they have an analysis showing that EP produces more distinct valid solution strategies, rather than primarily changing wording or improving the success rate of existing strategies?

The reported gains are promising, and I see value in the proposed idea. I currently lean toward weak rejection because the advantage over closely related methods and the specific contribution of path switching remain unclear. I would consider increasing my score if the rebuttal provides convincing evidence addressing these concerns.

2 comments

Thank you for your thoughtful review and for recognizing the potential of EP. We provide additional instruction ablations and reasoning-strategy analyses, compare EP directly with GRPO+OPCD, and quantify its training overhead.

Response to W1 and Q3: Path-Switching Instructions and Strategy Diversity

Instruction ablation. On Qwen3-4B, we replace the path-switching instructions with general reasoning instructions such as “Continue reasoning,” keeping all other training settings fixed. Across five benchmarks, the general instruction improves mean Avg@16 / Pass@16 from 46.6 / 67.8 for GRPO to 47.7 / 68.6. EP achieves 51.9 / 71.5, a further gain of 4.2 and 2.9 percentage points, respectively. We also remove explicit self-correction cues while retaining requests such as “Try a different approach.” This variant achieves 52.3 / 71.3, comparable to the original EP on both metrics. Per-benchmark results and replacement instructions

Distinct correct approaches. At four training checkpoints, we sample eight rollouts per problem across 64 problems. For each problem, a model judge groups correct solutions by their mathematical tools, structures, and representations, counting equivalent derivations together regardless of wording. Among problems with at least two correct rollouts, EP covers 1.44 distinct strategies per problem at step 60, compared with 1.24 for the general instruction, 1.33 for GRPO, and 1.31 for GRPO+Entropy. At steps 120 and 200, EP and GRPO exhibit comparable strategy diversity (1.24 vs. 1.26 and 1.27 vs. 1.29). These results support broader strategy coverage at step 60, when prior guidance is active. All four checkpoints, both correctness thresholds, and judging criteria

Response to W2 and Q1: Comparison with ICE, OPCD, and SAGE

EP samples rollouts from the current policy conditioned only on the problem. It selects a high-entropy position and uses a frozen policy snapshot to rescore the sampled suffix with an answer-free path-switching instruction inserted at that position. For each suffix token, EP adds the scaled, clipped log-probability shift to the reward-based GRPO advantage. This enables instruction-guided policy updates directly on unconditioned on-policy rollouts.

  • ICE also produces an instruction-free inference policy. It uses an external model to propose strategy instructions, trains on instruction-conditioned rollouts, and transfers successful behaviors through filtered self-distillation. In contrast, EP uses instructions only for scoring, directly guiding RL updates on unconditioned rollouts without a separate behavior transfer objective.
  • OPCD also scores student-generated continuations under additional context. Its context conveys experience extracted from previous solutions for the student to internalize. In contrast, EP provides exploration guidance at a specific reasoning step, jointly optimizing instruction-induced token preferences and verifiable rewards.
  • SAGE uses token entropy to reshape the reference distribution in its KL penalty. In EP, entropy determines where to intervene, while the instruction determines the direction of exploration. Our GRPO+Entropy baseline implements KL-Cov as a representative uncertainty-driven exploration method. We additionally compare EP directly with GRPO+OPCD as a closely related context-distillation method.

Direct comparison with GRPO+OPCD. We use Qwen3-4B in non-thinking mode, preserve each method's conditioning mechanism, and apply the same checkpoint selection criterion to GRPO, GRPO+OPCD, and EP. Every 10 steps, we evaluate mean Avg@4 on AMC23 and AIME25 and select the highest-scoring checkpoint. Across five benchmarks, mean Avg@16 / Pass@16 is 31.56 / 51.18 for GRPO+OPCD, 46.6 / 67.8 for GRPO, and 51.9 / 71.5 for EP. GRPO+OPCD exhibits declining training rewards under both EMA settings (0 and 1.0). We report the better-performing setting, EMA = 0, consistent with the original OPCD configuration. Full comparison and per-benchmark results

Response to W2 and Q2: GRPO+SDPO Setup and EP Training Cost

Teacher context. We incorporate an SDPO distillation term into GRPO. The student receives the original problem. For each rollout, the teacher additionally receives one verifier-approved solution uniformly sampled from the other rollouts for the same problem, followed by an instruction to solve the problem step by step. If no other correct rollout is available, we skip the distillation term for that rollout. Thus, when a group contains exactly one correct rollout, that solution supervises all other rollouts in the group. Exact teacher prompt and sampling procedure

Signal strength and schedule. We set the GRPO+SDPO distillation coefficient to 0.1 throughout all 200 training steps. EP uses the same coefficient for the first 60 steps, followed by linear annealing to zero over the next 10 steps. Figure 2a shows that the GRPO+SDPO training score drops sharply before the two schedules diverge. Under the same constant coefficient of 0.1, EP achieves 40.1 / 58.5 in mean Avg@16 / Pass@16, compared with 28.3 / 47.1 for GRPO+SDPO. These results indicate that EP outperforms GRPO+SDPO even under identical coefficient schedules. While annealing improves EP's performance, its effect on GRPO+SDPO remains untested.

Measured training cost. On four H100 GPUs, 200 training steps took 16 h 24 min for GRPO and 18 h 20 min for EP (65.60 and 73.33 GPU-hours, respectively). EP thus incurs an additional 1 h 56 min of training time (11.8%). The additional scoring is performed only during training, with no additional inference-time computation or sampling requirements.

Reviewer_vvgrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets a known weakness of RL with verifiable rewards for LLM reasoning: as GRPO training goes on, the model settles on a few reasoning paths and rarely samples others that could earn reward.

Exploration Prior (EP) adds a small token-level bonus to GRPO. For each rollout, it picks an uncertain point mid-response, inserts a generic phrase that gives no answer, such as "Wait, let's try a different approach.", and asks a frozen copy of the current model how likely the rest of the rollout is with that phrase in context. Tokens the phrase makes more likely get a small bonus; tokens it makes less likely get a small penalty. The bonus is clipped and switched off after the first 70 of 200 training steps. The paper explains this as KL regularization toward a "prior" defined by the phrase.

On five math benchmarks, EP beats GRPO and three alternatives. On Qwen3-4B, mean Avg@16 (accuracy averaged over 16 samples) rises from 46.6 to 51.9 and mean Pass@16 (solved at least once in 16) from 67.8 to 71.5, with gains in the same direction on Qwen3-8B and OLMo3-7B-Instruct. Random sentences in place of the phrases remove the gain.

Critical strengths

S1. A cheap, answer-free method with the right control. EP needs no teacher model, no reference solutions and no extra sampling: a fixed list of 12 phrases and one extra forward pass per rollout. It also runs the key control: replacing the phrases with 11 neutral sentences such as "The room stays quiet." drops mean Avg@16 from 51.9 to 44.8, below GRPO's 46.6 (Table 2). So the gain depends on what the phrase says, not on merely changing the context.

S2. Large gains on the hardest benchmarks, in the same direction on three models. The clearest gains are in Avg@16 on AIME, the metric least affected by luck because it averages 16 samples per problem: +9.2 on AIME24 and +8.7 on AIME25 for Qwen3-4B (Table 1), and +6.8 and +9.6 for Qwen3-8B (Table 4). While the bonus is on, EP's policy entropy rises from 0.28 to 0.51 without lowering training reward, while GRPO's stays at 0.27-0.30 (Figure 2). On OLMo3-7B-Instruct, GRPO lowers AIME24 Pass@16 from the base model's 83.3 to 76.7, while EP raises it to 90.0, which supports the claim of broader coverage.

Critical weaknesses

W1. The gains are not shown to exceed run-to-run noise, and the settings look tuned on the test sets.

No result reports seeds or error bars. The Pass@16 gain over GRPO is small in absolute terms: 3.3 of its 3.7 points come from 5 problems on AIME25 and HMMT25. The paper's own second evaluation disagrees with Table 1: in Figure 3(a), EP solves 34 of the 60 AIME25 and HMMT25 problems at 16 samples, where Table 1 implies 38, while GRPO solves 33 in both. The method is also very sensitive to its settings, which were evaluated on the test benchmarks with no validation split described. Changing the clip from 0.3 to 0.1 or 0.6 costs 4.8 and 3.9 points of Avg@16, a larger or fixed bonus weight puts EP below GRPO, and a smaller weight beats the default. That is surprising for a bonus of at most 0.03 per token, under a tenth of the smallest nonzero GRPO advantage. Either it acts mainly on prompts where every rollout gets the same reward, where it is the only signal, or single-run noise is large.

W2. The gain may come from steering Qwen3 back toward the thinking mode the authors switched off, rather than from guided exploration, and the experiments cannot tell these apart.

Thinking mode is disabled for the Qwen3 models in both training and evaluation, which is stated only in Appendix E. The 12 path-switching phrases (Appendix A), such as "Wait, let's try a different approach.", read like the self-correction language of thinking mode, so the bonus may mainly reward the model for moving back toward behavior it already has. Three observations fit this reading. The gain is smaller, though still positive, on OLMo3-7B-Instruct, which has no thinking switch (+1.6 Avg@16, against +5.3 on Qwen3-4B and +4.5 on Qwen3-8B). After the bonus is off at step 70, EP's entropy keeps rising and stays above GRPO's to the end, as a learned shift in behavior would. And the method as run is an early nudge, not the exploration the theory describes: the bonus only reweights tokens the model already sampled, never generates the continuation after the phrase, and when kept on for all of training it ends below GRPO (40.1 vs 46.6). The random-sentence control cannot separate these readings, and the comparisons that could are missing: thinking mode on, a reasoning phrase that does not ask to switch paths, and the same phrases inserted into sampled rollouts (Q1, Q3).

Questions for authors

Q1. Thinking mode (this bears on every result). Thinking mode is disabled for the Qwen3 models in both training and evaluation, which is stated only in Appendix E. The 12 path-switching phrases (Appendix A), such as "Wait, let's try a different approach." and "Maybe I made a mistake. Let me try a different approach.", read like the self-correction language reasoning models produce in thinking mode. So EP may mostly steer a model with thinking switched off back toward its own thinking-mode behavior, which the random-sentence control cannot rule out. Please report GRPO and EP with thinking mode on (or explain why that is not meaningful under the 4K-token training rollout cap), how often "Wait" or "Alternatively" style switches appear in each method's outputs, and why the Avg@16 gain is smaller on OLMo3-7B-Instruct, which has no thinking switch (+1.6), than on Qwen3-4B (+5.3) and Qwen3-8B (+4.5).

Q2. Seeds and the second evaluation. Can you report mean and standard deviation over at least 3 training seeds for GRPO and EP on Qwen3-4B? Figure 3(a) shows EP solving 34 of the 60 AIME25 and HMMT25 problems at 16 samples, while Table 1 implies 38; GRPO is 33 in both. Are these separate sampling runs, and is Pass@K computed from one draw of K samples or with the unbiased estimator?

Minor suggestions (do not affect the score).

(a) In Table 5, the mean Avg@16 for clip threshold c = 0.6 is printed as 50.0, but its five cells average 48.0. Please also soften "performs robustly" (G.1) and "relatively insensitive" (G.2).

(b) Fix the missing subject on line 63 ("baselines. improves the mean"), the unresolved "(?)" citation on line 96, the two near-duplicate sentences on lines 114-120, and "Table F in Appendix D" on line 346, which should be Table 4 in Appendix F.

(c) Add a limitations paragraph: math only, no seeds, 200 training steps, thinking mode off.

2 comments

Thank you for your thorough review and recognition of EP's strengths. We appreciate your insightful comments and address your concerns below.

Response to W1 and Q2

Thank you for highlighting the discrepancy at K=16K=16.

Why Table 1 and Figure 3(a) differ. Yes, these were separate sampling runs with the same base seed and decoding parameters. Table 1 used 16 rollouts per problem and counted a problem as solved if any rollout was correct. For Figure 3(a), we generated a separate pool of 256 rollouts per problem and drew one random subset of size KK for each point on the curve. At K=16K=16, EP solved 38 of the 60 AIME25 and HMMT25 problems in Table 1 and 34 in the figure. GRPO solved 33 in each run. As the reviewer observes, EP's margin on these two benchmarks therefore depends on the sampling draw.

Pass@K with 256 rollouts. To reduce sampling variability, we recomputed Pass@K on Qwen3-4B using all 256 rollouts per problem. For a problem with cc correct rollouts, the standard unbiased estimator 1−(256−cK)/(256K)1-\binom{256-c}{K}/\binom{256}{K} averages over all possible subsets of size KK. We also expanded the evaluation from AIME25 and HMMT25 to include AIME24 and AMC23. We omitted Minerva Math because generating 256 rollouts for each of its 272 problems, with a maximum generation length of 32,768 tokens, was computationally costly. The following table reports Pass@K (%) across the four benchmarks.

KKBaseGRPOGRPO w/ Ref-KLGRPO+EntropyGRPO+SDPOEP
130.0648.1546.7949.2225.8954.37
236.1855.3952.9756.3631.6461.23
441.7561.5658.6761.9036.4466.65
847.4766.7163.8965.8940.6571.10
1653.1670.6867.9769.0344.4574.89
3258.4173.9471.1872.1048.2577.91
6463.4077.1374.3975.0452.5480.34
12868.3980.2277.7377.3257.4282.16
25673.7582.5080.2178.5462.0883.33

EP achieves the highest Pass@K across all nine sampling budgets. Notably, EP with 8 samples outperforms GRPO with 16 samples, suggesting improved sampling efficiency. We will update Figure 3(a) and the corresponding discussion to reflect these results.

Variation across training seeds. We trained both methods on Qwen3-4B with seeds 42 (Table 1), 3407, and 123. Each score (%) weights the five benchmarks equally. The last column gives the mean and sample SD across seeds.

MetricMethodSeed 42Seed 3407Seed 123Mean ± SD
Avg@16GRPO46.646.547.746.9 ± 0.7
Avg@16EP51.952.052.252.0 ± 0.2
Pass@16GRPO67.866.669.367.9 ± 1.4
Pass@16EP71.572.071.371.6 ± 0.4

EP improved both metrics in all three paired runs. The mean gain was 5.1 percentage points for Avg@16 (range 4.5–5.5) and 3.7 for Pass@16 (range 2.0–5.4).

Checkpoint selection.

Regarding checkpoint selection, DAPO [1] observes that higher training rewards do not necessarily translate into better evaluation performance and monitors accuracy on the AIME test set throughout RL training. Following prior work on benchmark-based evaluation during training [2,3], we evaluated intermediate checkpoints every 10 training steps and selected the checkpoint with the highest mean Avg@4 across AMC23 and AIME25. Nevertheless, we acknowledge the potential selection bias introduced by this procedure. We will clarify the selection protocol and moderate the robustness claims in Appendix G. We applied this same rule to GRPO, GRPO+OPCD, and EP. Checkpoint selection and per-benchmark results for GRPO, GRPO+OPCD, and EP.

Effect of the exploration bonus. Although the per-token bonus is small relative to nonzero GRPO advantages, it provides a systematic preference over reasoning continuations across repeated policy updates. Its influence therefore cannot be assessed solely from its instantaneous magnitude. For groups with identical rewards, the EP term can still provide a learning signal when the GRPO reward advantage is zero. From the intervention position onward, the update adds a scaled, clipped token-level log-probability ratio to that advantage. How the exploration prior enters the GRPO update.

references

[1] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale

[2] Zhang et al. Reinforcement-aware Knowledge Distillation for LLM Reasoning.

[3] Wang et al. Reinforcement Learning for Reasoning in Large Language Models with One Training Example.

Response to W2 and Q1

Thinking mode under the 4K-token training cap. With thinking mode enabled on Qwen3-4B, approximately 89% of sampled trajectories reached the 4K-token cap and were truncated. Most trajectories therefore ended without a completed solution for the verifier to score, making positive rewards sparse. Under this cap, a GRPO–EP comparison would depend heavily on how often each policy produces a final answer before truncation, making it difficult to isolate the effect of EP on exploration. We therefore used thinking mode off for both methods during training and evaluation and will state this setting in the main experimental setup.

The reviewer raises a plausible explanation for the Qwen3 results. An instruction such as “Wait, let’s try a different approach” both cues self-correction and asks for a change of approach. We tested whether the gains survive removing the self-correction cues. In a separate analysis, we examined the mathematical strategies in successful rollouts during training.

Does EP's benefit depend on Qwen3's thinking mode? On OLMo3-7B-Instruct, which has no thinking-mode switch, EP improves mean Avg@16 over GRPO from 50.3 to 51.9 (+1.6 points) and mean Pass@16 from 71.5 to 75.2 (+3.7 points). Notably, the Pass@16 gain matches that on Qwen3-4B. Differences in training recipes and initial reasoning behavior may contribute to the smaller Avg@16 gain. Although this comparison cannot isolate these factors, it suggests that EP's benefits are not specific to Qwen3's thinking-mode configuration.

Does the wording of the instruction explain the gain? On Qwen3-4B, we replaced all 12 original instructions with direct requests to change approach, such as “Use another method to solve the problem.” These instructions omit “Wait,” “Alternatively,” and suggestions that the model made a mistake. A second control used general reasoning instructions such as “Continue reasoning,” without explicitly encouraging a change of approach. Across five benchmarks, removing the self-correction cues yielded 52.3 Avg@16 and 71.3 Pass@16, close to the original EP results of 51.9 and 71.5. General reasoning instructions yielded 47.7 and 68.6. An explicit request to change approach thus retained gains of 4.6 and 2.7 percentage points over general reasoning instructions even after we removed the cues associated with thinking mode. Instruction ablation results by benchmark and all 12 replacement instructions.

Do the rollouts contain different mathematical approaches? At four training checkpoints, we sampled eight rollouts per problem on 64 problems. A judge grouped correct solutions by their mathematical tools, structures, and representations, counting equivalent derivations as the same strategy even when their wording differed. At step 60, while the bonus was active, EP yielded 1.44 distinct strategies per problem among problems with at least two correct rollouts, compared with 1.33 for GRPO, 1.31 for GRPO+Entropy, and 1.24 for general reasoning instructions. At steps 120 and 200, after the bonus had been annealed to zero, EP and GRPO had similar strategy counts (1.24 vs. 1.26 and 1.27 vs. 1.29). The evidence for broader strategy coverage is therefore strongest while the bonus is active. The later entropy difference alone cannot establish a lasting increase in strategy diversity. Strategy diversity at all four checkpoints, both correctness thresholds, and judging criteria.

Together, the instruction ablations and strategy-level analysis suggest that path-switching semantics contribute to EP's improvements beyond explicit self-correction cues, with evidence of broader mathematical strategy coverage during the guided phase.

Response to Minor Suggestions We thank the reviewer for identifying these issues. We will correct the mean Avg@16 for c=0.6c=0.6 in Table 5 from 50.0 to 48.0, address the textual, citation, and cross-reference errors, and moderate the robustness claims in Appendix G. We will also add a limitations paragraph discussing the focus on mathematical reasoning, the limited training horizon of 200 steps, the thinking-mode configuration, and the absence of a separate validation split.

Reviewer_1GybOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Exploration Prior (EP), a method intended to improve exploration during reinforcement learning with verifiable rewards. EP samples reasoning trajectories from the current policy, selects an intermediate position from a set of high-entropy tokens, and inserts an answer-free instruction encouraging a change in reasoning direction. A frozen snapshot of the current policy then scores the original continuation under this modified context. The resulting conditional distribution serves as a prior rather than a rollout policy.

The training update combines the GRPO reward advantage with a clipped token-level log-likelihood difference between the intervention-conditioned prior and the unconditioned policy. The prior coefficient is used early in training and subsequently annealed to zero. The paper motivates this procedure through reward-tilted posterior inference and an approximate reverse-KL-regularized objective.

Experiments use DAPO-MATH-17K for training and evaluate three model backbones on five mathematical reasoning benchmarks. On Qwen3-4B, EP improves mean Avg@16 from 46.6 to 51.9 and mean Pass@16 from 67.8 to 71.5 relative to GRPO. The paper also reports cross-model results, Pass@K and diversity analyses, and ablations of intervention semantics, placement, coefficient scheduling, and signal clipping.

I view the main contribution as the construction and integration of a language-conditioned exploration prior into RLVR. The posterior formulation provides an interpretation of the method, rather than a substantially new inference framework.

Critical strengths
  1. EP uses the model’s existing response to natural-language instructions without requiring reference solutions, an external teacher, or additional correctness feedback. Importantly, training trajectories remain sampled from the unmodified policy; the intervention is used only to construct the auxiliary distribution. This is a useful alternative to encouraging diversity through entropy alone. The method changes the relative preference among continuations using a reasoning-related instruction, rather than indiscriminately promoting a more diffuse distribution. The prior construction is a worthwhile practical idea even independently of the Bayesian interpretation.

  2. In Table 1, EP outperforms both GRPO and the strongest non-EP baseline by mean Avg@16, GRPO+Entropy. Relative to the latter, the gains are 3.4 percentage points in Avg@16 and 6.5 points in Pass@16. Table 4 reports improvements in aggregate performance on Qwen3-8B and OLMo3-7B-Instruct as well, although improvements are not uniform across individual benchmarks.The simultaneous gains in average correctness and successful-solution coverage are more persuasive than increases in entropy or textual diversity alone. These performance results are the strongest evidence in favor of acceptance.

Critical weaknesses
  1. The semantic controls are insufficient. Unrelated random sentences may disrupt reasoning, making them a weak control for path-switching instructions. A coherent, non-switching instruction and an unconditioned policy prior are needed to isolate the proposed mechanism.

  2. Keeping the coefficient at 0.1 throughout training applies substantially more cumulative regularization than the default schedule. This comparison does not establish whether annealing itself is responsible for the improvement.

  3. The main results lack statistics across independent training seeds. Sampling 16 responses per problem does not address training instability, particularly for the smaller gains reported on OLMo3.

  4. Computational efficiency remains unclear. EP requires additional intervention-conditioned scoring, but the paper reports neither training overhead nor compute-matched comparisons. Equal rollout budgets alone do not establish an efficiency advantage over GRPO.

  5. Higher Pass@K and BLEU-based diversity are consistent with improved exploration, but do not establish that EP discovers meaningfully different successful reasoning strategies rather than more varied formulations of similar solutions.

Questions for authors
  1. What distinguishes path switching from generic context-conditioned regularization? Under otherwise identical settings, how does EP compare with: An unconditioned frozen-policy prior. A coherent, length-matched reasoning instruction that does not encourage changing paths. If these controls recover most of the improvement, I would interpret the contribution primarily as context-conditioned regularization or online self-distillation, rather than a path-switching-specific exploration mechanism.

  2. Does the schedule advantage persist when cumulative regularization strength is matched?

  3. How many independent training seeds were used for each result? Please provide cross-seed statistics for the main Qwen3-4B comparison and the OLMo3 generalization result, together with the training-time or computational overhead relative to GRPO. Stable gains under a comparable compute budget would materially strengthen the case for acceptance.

  4. How much learning is driven by the prior when the reward advantage provides no distinction? Appendix H.3 explicitly acknowledges that the practical gradient omits future-token likelihood-shift terms; I do not regard this as an undisclosed derivation error. The implemented update additionally uses prior-signal clipping, group-normalized reward advantages, and PPO clipping.

2 comments

Thank you for your comments. We provide additional instruction controls, compare schedules with matched cumulative coefficients, and evaluate GRPO and EP across three seeds.

Response to W1 and Q1: Path-Switching Instructions and Prior Controls

On Qwen3-4B, we replace the path-switching instructions with general reasoning instructions such as “Continue reasoning,” keeping all other training settings fixed. Across five benchmarks, this control recovers only 1.1 of EP's 5.3 percentage-point gain over GRPO in Avg@16 and 0.8 of its 3.7-point gain in Pass@16. Removing explicit self-correction cues while retaining the request to switch approaches (e.g., “Try a different approach”) preserves the gains on both metrics.

InstructionMean Avg@16 (%)Mean Pass@16 (%)
GRPO46.667.8
General reasoning47.768.6
Path switching51.971.5
Path switching without self-correction cues52.371.3

These results support the specific contribution of path-switching instructions beyond general reasoning guidance. The general reasoning instructions are coherent, although their lengths are not strictly matched to those of the path-switching instructions. Per-benchmark results and replacement instructions

For the unconditioned prior, the timing of snapshot refresh is important. Table 1 includes a fixed pre-RL reference (GRPO w/ Ref-KL), which yields 46.8 Avg@16 and 64.7 Pass@16. EP instead refreshes its snapshot at the start of each policy update. If we remove the instruction and score the same token under the same prefix, the snapshot and current policy agree at the moment of refresh, resulting in a zero log-probability shift. A subsequent shift can arise only as the policy moves away from that snapshot. We have not conducted a separate experiment with this refreshed, unconditioned prior.

Response to W5: Strategy Diversity in Correct Rollouts

At four checkpoints, we sample eight rollouts per problem across 64 problems. A model judge groups correct derivations by mathematical tools, structures, and representations, counting equivalent approaches together even when their wording differs. Among problems with at least two correct rollouts, EP averages 1.44 distinct strategies per problem at step 60, compared with 1.24 for the general reasoning instruction, 1.33 for GRPO, and 1.31 for GRPO+Entropy. After annealing, EP and GRPO exhibit comparable strategy diversity at steps 120 (1.24 vs. 1.26) and 200 (1.27 vs. 1.29). These results provide evidence that EP broadens the coverage of successful reasoning strategies during the active exploration phase. All four checkpoints, both correctness thresholds, and judging criteria

Response to W2 and Q2: Annealing at Matched Cumulative Strength

We run a constant-coefficient control with β=0.0325\beta=0.0325 for 200 steps. Its cumulative coefficient is 200×0.0325=6.5200\times0.0325=6.5, matching the nominal area under the default schedule (0.1 for 60 steps, followed by linear annealing to zero over 10 steps). We monitor mean Avg@4 on AIME25 and AMC23 every 10 steps. With a constant coefficient of 0.0325, the monitored score again drops sharply late in training, consistent with our previous observation under a constant coefficient of 0.1. This suggests that the timing of prior guidance matters beyond its cumulative coefficient. Given the late performance decline, we did not conduct the full five-benchmark evaluation for the 0.0325 run and therefore cannot quantify its final Avg@16 or Pass@16 gap.

Response to W3, W4, and Q3: Variation Across Training Seeds and Training Cost

We train GRPO and EP on Qwen3-4B with three independent seeds (42, 3407, and 123). The table reports the mean and sample standard deviation across seeds, with equal weight assigned to each of the five benchmarks within each run.

Qwen3-4BMean Avg@16 (%)Mean Pass@16 (%)
GRPO46.9 ± 0.767.9 ± 1.4
EP52.0 ± 0.271.6 ± 0.4

EP consistently improves both metrics across all three paired runs. The gains range from 4.5 to 5.5 percentage points in Avg@16 and from 2.0 to 5.4 points in Pass@16. The other main comparisons include one reported training run per method and backbone. On OLMo3-7B-Instruct, EP improves Avg@16 from 50.3 to 51.9 and Pass@16 from 71.5 to 75.2. With one run per method, however, the variability across training seeds on OLMo3 remains unmeasured. Individual seed results and backbone comparison

On four H100 GPUs, 200 training steps took 16 h 24 min for GRPO (65.60 GPU-hours) and 18 h 20 min for EP (73.33 GPU-hours). EP thus incurs an additional 1 h 56 min of training time (11.8%) under the same rollout budget. Training-time comparison

Response to Q4: Prior Updates in Groups with Identical Rewards

When all rollouts for a problem are correct, or all are incorrect, every group-normalized reward advantage is zero. While the prior is active, its clipped token-level log-probability shift is then the only advantage signal from that group. It applies from the intervention position onward and enters the same PPO-clipped surrogate. With the default clip of 0.3 and initial coefficient of 0.1, its magnitude is at most 0.03 per suffix token. Once the coefficient reaches zero, these groups contribute no policy-gradient signal. Token-level update and clipping

We have not quantified how much these updates contribute to the performance gain. To isolate their contribution, we plan to repeat EP with the prior disabled only for groups with identical rewards, keeping it active for mixed-reward groups under the usual schedule. We plan to include this ablation in the final version.

Reviewer_8DhzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Exploration Prior (EP) inserts an answer-free “try a different approach” prompt at a high-entropy position and uses the same model to rescore the remaining response. This guidance is gradually reduced during GRPO training. Rollouts are generated without the extra prompt. EP outperforms GRPO on average across three models and five math benchmarks.

Critical strengths
  1. The method is simple and needs no stronger teacher. The guidance comes from the model itself, without full reference solutions.
  2. The gains extend beyond Qwen3-4B. EP achieves higher average scores than GRPO on all three models.
Critical weaknesses
  1. The changes appear incremental. Conceptually, EP ≈ OPCD-style contextual self-distillation + GRPO + a “try a different approach” prompt inserted during reasoning. The main changes are the prompt, its placement, and how its weight is reduced. The paper cites OPCD but does not compare with it under matched conditions, so the benefit of these changes is unclear.
  2. The baselines need clearer descriptions.Section 5.1 does not specify whether GRPO+Entropy uses Clip-Cov or KL-Cov, or give its parameters. It also leaves unclear how GRPO+SDPO selects successful answers or handles groups with no correct answers. These details matter for reproducing the baselines and interpreting GRPO+SDPO’s decline in Table 1 and Figure 2.
Questions for authors
  1. How does EP compare with OPCD + GRPO using identical prompts, models, and data, with comparable training compute?
  2. What algorithm and parameters does GRPO+Entropy use? How does GRPO+SDPO construct prompts, select successful solutions, and handle groups with no correct responses?
1 comment

Thank you for your careful review. We clarify EP's distinction from OPCD and provide a direct comparison with GRPO+OPCD. We also describe the implementations of GRPO+Entropy and GRPO+SDPO.

Response to W1 and Q1: Comparison with OPCD

Conceptual distinction. Experience internalization and exploration priors address different challenges in reasoning optimization. OPCD aims to internalize previously accumulated experiential knowledge extracted from self-generated solution traces through context-conditioned distillation, without requiring ground-truth labels. In contrast, EP directly guides the current policy's exploration behavior through semantic priors, with verifiable rewards reinforcing effective trajectories.

Since the two methods rely on different conditioning mechanisms, enforcing identical full prompts would alter their intended objectives. We therefore preserve each method's original conditioning mechanism while keeping the other experimental settings consistent.

Empirical comparison. To provide an empirical comparison, we implemented GRPO+OPCD using Qwen3-4B in non-thinking mode. We observed that its training reward consistently deteriorated as training progressed. We further evaluated two EMA settings, 0 (as used in the original OPCD paper) and 1.0 (as used in our paper). Both settings exhibited similar degradation trends, suggesting that the deterioration cannot be attributed solely to the choice of EMA coefficient. We report the results with EMA=0, which achieved better performance and follows the original OPCD configuration.

For checkpoint selection, DAPO [1] observes that higher training rewards do not necessarily translate into better evaluation performance and therefore monitors accuracy on the AIME test set throughout RL training. Following prior work on benchmark-based checkpoint selection [2,3], we evaluated checkpoints every 10 training steps and selected the checkpoint with the highest mean Avg@4 across AMC23 and AIME25. The same evaluation frequency and selection criterion were applied consistently to GRPO, GRPO+OPCD, and EP to ensure a fair comparison. We report the results across all five mathematical benchmarks below:

Each cell reports Avg@16 / Pass@16 (%). Best scores for each metric are in bold.

MethodAIME24AIME25AMC23MinervaMathHMMT Feb 2025Average
Qwen3-4B (before training)21.0 / 46.720.2 / 43.367.8 / 90.040.3 / 53.312.3 / 23.332.3 / 51.3
GRPO45.2 / 76.736.7 / 66.786.6 / 97.542.9 / 54.821.7 / 43.346.6 / 67.8
GRPO+OPCD19.38 / 46.6720.63 / 43.3366.41 / 90.0039.32 / 52.5712.08 / 23.3331.56 / 51.18
EP54.4 / 76.745.4 / 73.389.8 / 97.544.4 / 56.625.6 / 53.351.9 / 71.5

These results suggest that, under our experimental setting, GRPO+OPCD does not yield consistent improvements, whereas EP provides a more effective way to guide policy exploration by combining semantic priors with verifiable rewards. We emphasize that the two methods target different objectives, and our results should not be interpreted as evidence that OPCD is generally ineffective.

[1] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale

[2] Zhang et al. Reinforcement-aware Knowledge Distillation for LLM Reasoning.

[3] Wang et al. Reinforcement Learning for Reasoning in Large Language Models with One Training Example.

Response to W2 and Q2: GRPO+Entropy and GRPO+SDPO Setup

GRPO+Entropy. We use KL-Cov rather than Clip-Cov, following the official verl recipe. We adopt the default hyperparameters specified in the recipe: ppo_kl_coef = 1 and kl_cov_ratio = 0.002.

GRPO+SDPO. The student receives only the original problem. For the teacher, we preserve the original system and preceding messages, and replace the final user message with the following template:

{Original problem}

Here is a reference solution to this problem:

{Selected successful trajectory}

Now solve this problem step by step.

A successful trajectory is one that receives a positive correctness reward from the verifier. For each rollout, we uniformly sample a successful trajectory from the other rollouts in the same group, explicitly excluding the rollout itself. If no other successful trajectory is available, we skip the SDPO loss for that rollout. In particular, when a group contains exactly one successful trajectory, only that trajectory's SDPO loss is skipped, while other rollouts can still use it as the teacher reference. If a group contains no correct responses, all its GRPO advantages are zero and no SDPO distillation signal is available; thus, the group contributes no policy update.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.