- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates insufficient exploration in reinforcement learning with verifiable rewards (RLVR), arguing that uniformly incorrect rollout groups can arise from sampling bias toward dominant reasoning strategies rather than a lack of underlying model capability. It proposes Problem–Strategy Rollout Allocation (PSRA), which uses Bayesian sequential allocation to distribute a fixed rollout budget between unguided and strategy-conditioned prompts, alongside a preservation objective that maintains access to useful alternative reasoning strategies. Experiments across Qwen2.5 models ranging from 1.5B to 7B and two RL training corpora demonstrate consistent improvements in reasoning performance, reduced saturation, stronger out-of-distribution generalization, and sustained gains under increased inference budgets.
Critical strengths
- Insightful perspective on exploration failures. The paper attributes uniformly incorrect rollout groups to sampling bias toward dominant reasoning strategies rather than necessarily insufficient model capability. It further recognizes that strategy accessibility evolves during training, providing a compelling motivation for adaptive exploration.
- Adaptive and principled exploration mechanism. PSRA uses Bayesian sequential allocation to direct a fixed rollout budget toward unguided or strategy-conditioned prompts that are more likely to yield informative, non-saturated groups. Its preservation objective additionally helps retain useful alternative strategies while transferring successful guided behaviors into the unguided policy.
- Consistent empirical improvements. Experiments across multiple model scales and two RL training corpora demonstrate improvements in reasoning performance, reduced dead saturation, and stronger out-of-distribution transfer. The sustained gains under increased inference budgets further support the practical effectiveness of the proposed method.
Critical weaknesses
- Motivation: The paper argues that finite sampling causes models to over-rely on dominant reasoning strategies while neglecting other strategies already within their capabilities. However, it does not sufficiently establish that this phenomenon is a primary cause of insufficient exploration in RLVR. The authors should conduct experiments to distinguish between strategies that are not sampled, strategies that the model lacks the capability to execute, and strategies that are difficult to elicit with the current prompts, and analyze their respective contributions to training failures.
- Design: PSRA employs Bayesian sequential allocation to select exploration arms and a strategy-preservation objective to maintain the accessibility of strategy-conditioned reasoning paths, but it remains unclear how the informativeness of each arm is reliably estimated and how exploration benefits are balanced against the cost of strategy preservation. The authors should provide theoretical or empirical justification for these key design choices and conduct ablation studies to validate the necessity of both the Bayesian allocation mechanism and the strategy-preservation objective.
- Related Work and Baselines: The authors should discuss relevant prior work and clarify the similarities and differences between PSRA and existing methods. These methods should also be included as baselines to enable a fair comparison and better establish the novelty and effectiveness of PSRA.
- Reinforced Efficient Reasoning via Semantically Diverse Exploration
- Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
- Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
Questions for authors
Explain the concerns in Weaknesses Part.
5 comments
W2: Design justification and ablations
(1) How arm informativeness is estimated. Each arm's value is the posterior probability that it becomes non-saturated after more rollouts (Eq. 9). Because the objective is separable and is concave in , greedy allocation is optimal (Section 4.1). We validated both the components and the estimates (same five-strategy set, budget, and objective; only the allocation rule varies):
| Allocation rule | ID Avg. | OOD Avg. | Non-saturated groups* |
|---|---|---|---|
| Random allocation | 27.0 ± .2 | 40.2 ± .4 | 41% |
| Probe success rate (closest to 0.5 first) | 27.4 ± .2 | 41.0 ± .3 | 47% |
| Simple Beta posterior (uniform prior) + | 27.6 ± .1 | 41.7 ± .3 | 52% |
| + hierarchical prior | 27.8 ± .2 | 42.4 ± .3 | 57% |
| + coverage term | 28.0 ± .1 | 43.1 ± .2 | 61% |
| + steerability gate (full PSRA) | 28.1 ± .2 | 43.5 ± .3 | 63% |
| Vanilla GRPO | 27.3 ± .3 | 38.9 ± .3 | — |
*Fraction of groups with mixed rewards.
Each component adds a consistent gain. The estimates are also reliable. To avoid selection bias, we sampled every saturated-probe arm at three checkpoints, regardless of the allocator's choice. On these arms, the predicted is well calibrated (ECE 0.031), and it ranks arms with AUROC 0.78. Probe-only rules score 0.50, because they cannot distinguish arms with identical probe outcomes.
(2) Necessity of preservation and the cost trade-off. We swept the preservation coefficient ; removes preservation entirely. We report two accessibility measures at the end of training, averaged over strategies (Eq. 6):
- : internalization (higher is better);
- : residual loss of conditional accessibility (lower is better).
| λ | ID Avg. | OOD Avg. | ↑ | ↓ |
|---|---|---|---|---|
| 0 (no preservation) | 27.7 ± .3 | 42.6 ± .4 | 0.31 | 0.27 |
| 0.1 | 27.9 ± .2 | 43.1 ± .3 | 0.36 | 0.19 |
| 0.3 (default) | 28.1 ± .2 | 43.5 ± .3 | 0.41 | 0.12 |
| 0.5 | 28.0 ± .2 | 43.2 ± .3 | 0.38 | 0.09 |
| 0.7 | 27.2 ± .4 | 41.6 ± .5 | 0.29 | 0.08 |
| 1.0 | 26.5 ± .5 | 40.3 ± .6 | 0.22 | 0.08 |
The trade-off is asymmetric:
- Too little preservation. Strategies lose accessibility before they are absorbed: more than doubles at compared with the default. The accuracy cost is moderate (−0.4 ID, −0.9 OOD).
- Too much preservation. Beyond , accessibility no longer improves ( plateaus at 0.08), while internalization and accuracy drop sharply. At , ID accuracy falls below GRPO. Here, gradient capacity is spent on keeping the guided contexts sharp rather than transferring their behavior to the unguided policy.
A moderate λ balances the two. Performance is flat over 0.1–0.5, so no fine-tuning is needed.
Cost. Preservation adds one forward and backward pass under for guided rollouts only, increasing per-step wall-clock time by 8% on 3B. This is offset by faster convergence (Fig. 4).
We thank the reviewer for pointing these out. We will add these additional analyses to our paper.
W3: Related work and baselines
We thank the reviewer for these references. The table positions PSRA against them and our existing baselines:
| Family | Methods | Mechanism | Difference from PSRA |
|---|---|---|---|
| Prompt-level rollout allocation | VIGOR, DAPO dynamic sampling | Allocate budget to prompts with high observed reward variance | All-failure prompts have zero variance and receive no budget. PSRA changes the conditional distribution and values saturated arms through a posterior. |
| Within-trajectory diversity | ROSE | Branch at semantically uncertain points of sampled trajectories; restart from the root | Diversifies around routes the policy already takes. PSRA switches the route before generation. |
| External expert guidance | MENTOR, HINT, LUFFY, BREAD | Inject guidance from a stronger model | Requires privileged guidance. PSRA uses the model's own strategy repertoire. |
Full reproduction of all three methods was not feasible within the rebuttal period, and reproduced guided-RL baselines often fall short of their reported results. Reported numbers are also not directly comparable: for example, VIGOR uses base models, MATH (mixed with DAPO data at 7B), and single-sample evaluation. We therefore compare each method's mechanism through the closest controlled evidence in our setting.
| Allocation rule (3B, same arms, budget, objective) | ID Avg. | OOD Avg. | Mixed-reward groups | AUROC on saturated arms |
|---|---|---|---|---|
| Probe success rate (VIGOR's variance ranking) | 27.4 ± .2 | 41.0 ± .3 | 47% | 0.50 |
| Simple Beta posterior + U(k) | 27.6 ± .1 | 41.7 ± .3 | 52% | 0.50 |
| Full PSRA | 28.1 ± .2 | 43.5 ± .3 | 63% | 0.78 |
| Vanilla GRPO | 27.3 ± .3 | 38.9 ± .3 | — | — |
Rows are taken from the allocator ablation in W2. AUROC is measured on arms whose probes are all-correct or all-failure, where every such arm was given the full additional budget regardless of the allocator's choice to avoid selection bias (full calibration analysis in our response to Reviewer ghrR, W2).
- VIGOR. For binary rewards, group variance is , so variance ranking is equivalent to the probe-success-rate rule above. Even with access to strategy arms, which favors VIGOR, it assigns the same score to every all-failure arm and therefore cannot tell which ones are recoverable (AUROC 0.50). A probabilistic posterior alone does not fix this. On all-failure prompts, a variance-based rule can only resample the unguided policy. On the base model, this turns 18% of into informative groups at 22 rollouts per prompt, versus 31% for PSRA at the same budget, and needs about 4.5× more rollouts to match PSRA (W1(2)).
- MENTOR. Like HINT, it injects guidance from a stronger model, which creates a mismatch between the guidance and the policy; HINT introduces affinity-aware optimization specifically to reduce it. PSRA's guidance comes from the model's own strategy repertoire, so the mismatch is small by construction: on 3B, the median |log w| of the sequence-level correction is 0.18, and the normalized effective sample size averages 0.93 (see our response to Reviewer ghrR, W3). On 7B, PSRA outperforms HINT by 1.2 ID and 2.1 OOD points (36.9 / 40.7 vs. 35.7 / 38.6, Paper Table 1) without any expert model.
- ROSE. It diversifies within routes the policy already takes, whereas PSRA switches the route before generation. The closest controls in our setting are route-preserving: more unguided sampling, higher temperature.
We will add these works to Related Work covering VIGOR and DAPO, and extend the guided-exploration paragraph with MENTOR and ROSE.
W1: Is strategy concentration a primary cause of exploration failure?
We thank the reviewer for the positive assessment and the constructive suggestions. All new analyses use Qwen2.5-7B-Instruct / DAPO-Math-17K, the setting of Figs. 1 and 3. New training runs use Qwen2.5-3B-Instruct / DAPO-Math-17K, a mid-scale setting that allows multi-seed runs within the rebuttal period; they are averaged over 3 seeds (Paper Table 1 reports a single run with seed 42).
For W1, we agree this needs to be quantified. Our claim is that an all-failure group need not reflect missing capability, not that strategy concentration is the only or primary cause. Following the reviewer’s suggestion, we separate these causes and measure their contributions.
(1) Decomposing all-failure groups. We take the held-out problems on which all 16 base-model rollouts fail (Qfail, Section 5.2). Each problem is assigned to the first condition, in order of increasing elicitation strength, that solves it within an additional budget of B = 128 on the base model:
The first two rows correspond directly to Fig. 3c at B = 128: unguided base sampling rescues ≈ 0.34, and adding the best alternative strategy raises this to ≈ 0.62. In total, 70% of all-failure groups are solvable by the base model without new capability.
(2) PSRA's scope is these 70%, not only the strategy-dependent 28–36%. During training, what matters is whether a group becomes informative within the fixed per-prompt budget, not whether it is eventually solvable. Sampling-insufficient problems are solvable in principle, but unguided sampling needs close to 128 rollouts to reach them, far beyond the 22 per prompt available in training. For GRPO, many of them therefore remain all-failure groups during training.
We compare PSRA's allocation with simply adding unguided rollouts. The table reports the fraction of Qfail turned into a non-saturated group on the base model:
With the same 22 rollouts, PSRA raises the share of recoverable all-failure groups from 18% to 31%. Unguided sampling needs about 4.5× more rollouts (≈ 100 per prompt) to match this, and per-step cost grows nearly linearly with rollouts. PSRA's own overhead at a fixed budget is ≈ 14% per step: preservation ≈ 8% (see W2), the importance-weight forward pass ≈ 4%, and two-stage generation ≈ 2%.
(3) Which causes the trained policies resolve. The table reports the fraction of Qfail recovered by each unguided policy (B = 16, as in Fig. 3a,c):
Together, (2) and (3) show two complementary benefits of PSRA. It reaches problems that unguided sampling cannot reach at any practical budget, and it reaches sampling-insufficient problems at a fraction of the rollout cost.
We will add this decomposition and the efficiency comparison to Section 5.2, and revise the wording in Section 1 to state the claim at this level of precision.