acceptodds
Under review as a conference paper at ICLR 2027

The Model Knows Another Way: Strategy Switching for Effective RLVR Exploration

Abstract

Reinforcement learning with verifiable rewards (RLVR) is often limited by insufficient exploration: difficult problems can yield uniformly incorrect rollout groups and therefore little learning signal. We show that such failures need not reflect missing capability. Instead, finite sampling often concentrates on a problem-specific dominant reasoning strategy while leaving alternative strategies already supported by the model unexplored. Moreover, the accessibility of these strategies evolves during RL: some are internalized into autonomous behavior, while others become difficult to elicit before being absorbed. Motivated by these observations, we introduce Problem–Strategy Rollout Allocation (PSRA), which treats unguided and strategy-conditioned prompts as competing exploration arms and uses Bayesian sequential allocation to direct a fixed rollout budget toward arms most likely to yield informative, non-saturated groups. A preservation objective keeps useful strategy-conditioned routes accessible while successful guided behaviors are transferred to the unguided policy. Across Qwen2.5 models from 1.5B to 7B and two RL training corpora, PSRA consistently improves reasoning performance, reduces dead saturation, strengthens out-of-distribution transfer, and maintains larger gains under increased inference budgets.

Primary area:
reinforcement learning
Keywords:
RLVR, GRPO, Reward Sparsity, Strategy Guided Exploration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Wyt2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates insufficient exploration in reinforcement learning with verifiable rewards (RLVR), arguing that uniformly incorrect rollout groups can arise from sampling bias toward dominant reasoning strategies rather than a lack of underlying model capability. It proposes Problem–Strategy Rollout Allocation (PSRA), which uses Bayesian sequential allocation to distribute a fixed rollout budget between unguided and strategy-conditioned prompts, alongside a preservation objective that maintains access to useful alternative reasoning strategies. Experiments across Qwen2.5 models ranging from 1.5B to 7B and two RL training corpora demonstrate consistent improvements in reasoning performance, reduced saturation, stronger out-of-distribution generalization, and sustained gains under increased inference budgets.

Critical strengths
  1. Insightful perspective on exploration failures. The paper attributes uniformly incorrect rollout groups to sampling bias toward dominant reasoning strategies rather than necessarily insufficient model capability. It further recognizes that strategy accessibility evolves during training, providing a compelling motivation for adaptive exploration.
  2. Adaptive and principled exploration mechanism. PSRA uses Bayesian sequential allocation to direct a fixed rollout budget toward unguided or strategy-conditioned prompts that are more likely to yield informative, non-saturated groups. Its preservation objective additionally helps retain useful alternative strategies while transferring successful guided behaviors into the unguided policy.
  3. Consistent empirical improvements. Experiments across multiple model scales and two RL training corpora demonstrate improvements in reasoning performance, reduced dead saturation, and stronger out-of-distribution transfer. The sustained gains under increased inference budgets further support the practical effectiveness of the proposed method.
Critical weaknesses
  • Motivation: The paper argues that finite sampling causes models to over-rely on dominant reasoning strategies while neglecting other strategies already within their capabilities. However, it does not sufficiently establish that this phenomenon is a primary cause of insufficient exploration in RLVR. The authors should conduct experiments to distinguish between strategies that are not sampled, strategies that the model lacks the capability to execute, and strategies that are difficult to elicit with the current prompts, and analyze their respective contributions to training failures.
  • Design: PSRA employs Bayesian sequential allocation to select exploration arms and a strategy-preservation objective to maintain the accessibility of strategy-conditioned reasoning paths, but it remains unclear how the informativeness of each arm is reliably estimated and how exploration benefits are balanced against the cost of strategy preservation. The authors should provide theoretical or empirical justification for these key design choices and conduct ablation studies to validate the necessity of both the Bayesian allocation mechanism and the strategy-preservation objective.
  • Related Work and Baselines: The authors should discuss relevant prior work and clarify the similarities and differences between PSRA and existing methods. These methods should also be included as baselines to enable a fair comparison and better establish the novelty and effectiveness of PSRA.
    • Reinforced Efficient Reasoning via Semantically Diverse Exploration
    • Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
    • Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
Questions for authors

Explain the concerns in Weaknesses Part.

5 comments
AuthorsOpenReview ↗

W1: Is strategy concentration a primary cause of exploration failure?

We thank the reviewer for the positive assessment and the constructive suggestions. All new analyses use Qwen2.5-7B-Instruct / DAPO-Math-17K, the setting of Figs. 1 and 3. New training runs use Qwen2.5-3B-Instruct / DAPO-Math-17K, a mid-scale setting that allows multi-seed runs within the rebuttal period; they are averaged over 3 seeds (Paper Table 1 reports a single run with seed 42).

For W1, we agree this needs to be quantified. Our claim is that an all-failure group need not reflect missing capability, not that strategy concentration is the only or primary cause. Following the reviewer’s suggestion, we separate these causes and measure their contributions.

(1) Decomposing all-failure groups. We take the held-out problems on which all 16 base-model rollouts fail (Qfail\mathcal{Q}_{\text{fail}}, Section 5.2). Each problem is assigned to the first condition, in order of increasing elicitation strength, that solves it within an additional budget of B = 128 on the base model:

CategoryCriterionShare of Qfail\mathcal{Q}_{\text{fail}}
Sampling-insufficientMore unguided sampling succeeds34%
Selection failureA strategy instruction succeeds; unguided sampling does not28%
Hard to elicitOnly exemplar conditioning succeeds8%
Capability-limitedNo condition succeeds30%

The first two rows correspond directly to Fig. 3c at B = 128: unguided base sampling rescues ≈\approx 0.34, and adding the best alternative strategy raises this to ≈\approx 0.62. In total, 70% of all-failure groups are solvable by the base model without new capability.

(2) PSRA's scope is these 70%, not only the strategy-dependent 28–36%. During training, what matters is whether a group becomes informative within the fixed per-prompt budget, not whether it is eventually solvable. Sampling-insufficient problems are solvable in principle, but unguided sampling needs close to 128 rollouts to reach them, far beyond the 22 per prompt available in training. For GRPO, many of them therefore remain all-failure groups during training.

We compare PSRA's allocation with simply adding unguided rollouts. The table reports the fraction of Qfail\mathcal{Q}_{\text{fail}} turned into a non-saturated group on the base model:

Sampling schemeRollouts / promptNon-saturated on Qfail\mathcal{Q}_{\text{fail}}Relative per-step cost
Unguided2218%1.00
Unguided4422%≈2.0\approx 2.0
Unguided8829%≈4.0\approx 4.0
Unguided12834%≈5.8\approx 5.8
PSRA allocation2231%≈1.14\approx 1.14

With the same 22 rollouts, PSRA raises the share of recoverable all-failure groups from 18% to 31%. Unguided sampling needs about 4.5× more rollouts (≈\approx 100 per prompt) to match this, and per-step cost grows nearly linearly with rollouts. PSRA's own overhead at a fixed budget is ≈\approx 14% per step: preservation ≈\approx 8% (see W2), the importance-weight forward pass ≈\approx 4%, and two-stage generation ≈\approx 2%.

(3) Which causes the trained policies resolve. The table reports the fraction of Qfail\mathcal{Q}_{\text{fail}} recovered by each unguided policy (B = 16, as in Fig. 3a,c):

CategoryGRPOPSRAPSRA − GRPOShare of PSRA's additional recoveries
Sampling-insufficient9.910.2+0.33%
Selection failure5.39.6+4.348%
Hard to elicit0.62.6+2.022%
Capability-limited1.23.6+2.427%
Total17.026.0+9.0100%
  • Sampling-insufficient problems are recovered at similar rates. These problems lie within the unguided distribution, so GRPO also improves on them through generalization from other problems; PSRA’s extra training signal (2) yields only a small additional gain here.
  • Strategy-addressable problems (selection failure and hard to elicit) account for 70% of PSRA's additional recoveries, although they make up only 36% of Qfail\mathcal{Q}_{\text{fail}}.
  • Capability-limited problems remain largely unsolved by both methods (4% vs. 12% within the category). This defines PSRA's scope: it exploits strategies the model already supports and does not create new ones.

Together, (2) and (3) show two complementary benefits of PSRA. It reaches problems that unguided sampling cannot reach at any practical budget, and it reaches sampling-insufficient problems at a fraction of the rollout cost.

We will add this decomposition and the efficiency comparison to Section 5.2, and revise the wording in Section 1 to state the claim at this level of precision.

AuthorsOpenReview ↗

W2: Design justification and ablations

(1) How arm informativeness is estimated. Each arm's value is the posterior probability Uq,z(k)U_{q,z}(k) that it becomes non-saturated after kk more rollouts (Eq. 9). Because the objective is separable and UU is concave in kk, greedy allocation is optimal (Section 4.1). We validated both the components and the estimates (same five-strategy set, budget, and objective; only the allocation rule varies):

Allocation ruleID Avg.OOD Avg.Non-saturated groups*
Random allocation27.0 ± .240.2 ± .441%
Probe success rate (closest to 0.5 first)27.4 ± .241.0 ± .347%
Simple Beta posterior (uniform prior) + U(k)U(k)27.6 ± .141.7 ± .352%
+ hierarchical prior27.8 ± .242.4 ± .357%
+ coverage term28.0 ± .143.1 ± .261%
+ steerability gate (full PSRA)28.1 ± .243.5 ± .363%
Vanilla GRPO27.3 ± .338.9 ± .3—

*Fraction of (q,z)(q, z) groups with mixed rewards.

Each component adds a consistent gain. The estimates are also reliable. To avoid selection bias, we sampled every saturated-probe arm at three checkpoints, regardless of the allocator's choice. On these arms, the predicted U(6)U(6) is well calibrated (ECE 0.031), and it ranks arms with AUROC 0.78. Probe-only rules score 0.50, because they cannot distinguish arms with identical probe outcomes.

(2) Necessity of preservation and the cost trade-off. We swept the preservation coefficient λ\lambda; λ=0\lambda = 0 removes preservation entirely. We report two accessibility measures at the end of training, averaged over strategies (Eq. 6):

  • ιT\iota_T: internalization (higher is better);
  • LTL_T: residual loss of conditional accessibility (lower is better).
λID Avg.OOD Avg.ιT\iota_T ↑LTL_T ↓
0 (no preservation)27.7 ± .342.6 ± .40.310.27
0.127.9 ± .243.1 ± .30.360.19
0.3 (default)28.1 ± .243.5 ± .30.410.12
0.528.0 ± .243.2 ± .30.380.09
0.727.2 ± .441.6 ± .50.290.08
1.026.5 ± .540.3 ± .60.220.08

The trade-off is asymmetric:

  • Too little preservation. Strategies lose accessibility before they are absorbed: LTL_T more than doubles at λ=0\lambda = 0 compared with the default. The accuracy cost is moderate (−0.4 ID, −0.9 OOD).
  • Too much preservation. Beyond λ≈0.7\lambda \approx 0.7, accessibility no longer improves (LTL_T plateaus at 0.08), while internalization and accuracy drop sharply. At λ=1.0\lambda = 1.0, ID accuracy falls below GRPO. Here, gradient capacity is spent on keeping the guided contexts sharp rather than transferring their behavior to the unguided policy.

A moderate λ balances the two. Performance is flat over 0.1–0.5, so no fine-tuning is needed.

Cost. Preservation adds one forward and backward pass under c(q,z)c(q,z) for guided rollouts only, increasing per-step wall-clock time by ≈\approx 8% on 3B. This is offset by faster convergence (Fig. 4).

We thank the reviewer for pointing these out. We will add these additional analyses to our paper.

AuthorsOpenReview ↗

W3: Related work and baselines

We thank the reviewer for these references. The table positions PSRA against them and our existing baselines:

FamilyMethodsMechanismDifference from PSRA
Prompt-level rollout allocationVIGOR, DAPO dynamic samplingAllocate budget to prompts with high observed reward varianceAll-failure prompts have zero variance and receive no budget. PSRA changes the conditional distribution and values saturated arms through a posterior.
Within-trajectory diversityROSEBranch at semantically uncertain points of sampled trajectories; restart from the rootDiversifies around routes the policy already takes. PSRA switches the route before generation.
External expert guidanceMENTOR, HINT, LUFFY, BREADInject guidance from a stronger modelRequires privileged guidance. PSRA uses the model's own strategy repertoire.

Full reproduction of all three methods was not feasible within the rebuttal period, and reproduced guided-RL baselines often fall short of their reported results. Reported numbers are also not directly comparable: for example, VIGOR uses base models, MATH (mixed with DAPO data at 7B), and single-sample evaluation. We therefore compare each method's mechanism through the closest controlled evidence in our setting.

Allocation rule (3B, same arms, budget, objective)ID Avg.OOD Avg.Mixed-reward groupsAUROC on saturated arms
Probe success rate (VIGOR's variance ranking)27.4 ± .241.0 ± .347%0.50
Simple Beta posterior + U(k)27.6 ± .141.7 ± .352%0.50
Full PSRA28.1 ± .243.5 ± .363%0.78
Vanilla GRPO27.3 ± .338.9 ± .3——

Rows are taken from the allocator ablation in W2. AUROC is measured on arms whose probes are all-correct or all-failure, where every such arm was given the full additional budget regardless of the allocator's choice to avoid selection bias (full calibration analysis in our response to Reviewer ghrR, W2).

  • VIGOR. For binary rewards, group variance is p^(1−p^)\hat p(1-\hat p), so variance ranking is equivalent to the probe-success-rate rule above. Even with access to strategy arms, which favors VIGOR, it assigns the same score to every all-failure arm and therefore cannot tell which ones are recoverable (AUROC 0.50). A probabilistic posterior alone does not fix this. On all-failure prompts, a variance-based rule can only resample the unguided policy. On the base model, this turns 18% of Qfail\mathcal{Q}_{\text{fail}} into informative groups at 22 rollouts per prompt, versus 31% for PSRA at the same budget, and needs about 4.5× more rollouts to match PSRA (W1(2)).
  • MENTOR. Like HINT, it injects guidance from a stronger model, which creates a mismatch between the guidance and the policy; HINT introduces affinity-aware optimization specifically to reduce it. PSRA's guidance comes from the model's own strategy repertoire, so the mismatch is small by construction: on 3B, the median |log w| of the sequence-level correction is 0.18, and the normalized effective sample size averages 0.93 (see our response to Reviewer ghrR, W3). On 7B, PSRA outperforms HINT by 1.2 ID and 2.1 OOD points (36.9 / 40.7 vs. 35.7 / 38.6, Paper Table 1) without any expert model.
  • ROSE. It diversifies within routes the policy already takes, whereas PSRA switches the route before generation. The closest controls in our setting are route-preserving: more unguided sampling, higher temperature.

We will add these works to Related Work covering VIGOR and DAPO, and extend the guided-exploration paragraph with MENTOR and ROSE.

Reviewer_GDgzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates insufficient exploration in reinforcement learning with verifiable rewards (RLVR) from a strategy-level perspective. The authors argue that all-failure rollout groups may arise not only from missing reasoning capabilities, but also from the model's tendency to concentrate on a dominant reasoning strategy while underexploring alternative strategies that are in fact accessible. To address this problem, the paper proposes Problem–Strategy Rollout Allocation (PSRA). It identifies reusable reasoning strategies through hidden-representation clustering, and employs a Bayesian allocation mechanism to distribute a fixed rollout budget toward arms. Experiments on Qwen2.5 models of different sizes and two mathematical reasoning training corpora demonstrate improvements over vanilla GRPO and several guided-RL baselines.

Critical strengths
  1. The paper provides an interesting interpretation of all-failure rollout groups as potentially reflecting insufficient strategy coverage rather than missing model capabilities. This hypothesis is supported by behavioral analyses of strategy concentration, controllability, and the recovery of initially unsolved problems.

  2. PSRA formulates strategy selection as an adaptive budget-allocation problem rather than relying on fixed guidance rules. Its Bayesian mechanism jointly considers problem difficulty, strategy effectiveness, and uncovered reasoning regions. The integration of rollout allocation, guided-to-unguided transfer, and strategy preservation forms a coherent framework, with generally positive empirical results across the evaluated model scales.

Critical weaknesses
  1. Several central components of the framework overlap conceptually with existing work. In particular, NudgeRL already explores strategy-conditioned rollouts and transfers guided behaviors to an unguided policy, while Off-Context GRPO explicitly addresses the distribution mismatch between guided sampling and unguided optimization. A direct conceptual comparison with NudgeRL and more targeted controls would strengthen the novelty and significance claims.

  2. Despite the relatively elaborate strategy-discovery procedure involving hidden-state representations, DPMM clustering, and behavioral certification, the resulting strategies largely correspond to familiar mathematical reasoning approaches, such as stepwise decomposition, equation setup, and backward reasoning. It is therefore unclear whether the representation-based discovery procedure provides substantial benefits over directly specifying a set of well-designed reasoning strategies. The current fixed-strategy baseline uses a different strategy set and does not fully isolate this question.

Questions for authors
  1. Comparison with closely related approaches. Can the authors provide a direct, conceptual comparison with NudgeRL etc.? What specific advantages does PSRA provide beyond existing strategy-guided exploration and guided-to-unguided transfer methods?
5 comments
AuthorsOpenReview ↗

W1/Q1: Comparison with closely related approaches

We thank the reviewer for the careful reading and for raising these questions. All new analyses use Qwen2.5-7B-Instruct / DAPO-Math-17K, the setting of Figs. 1 and 3. Ablations use Qwen2.5-3B-Instruct / DAPO-Math-17K, averaged over 3 seeds (Paper Table 1 reports a single run with seed 42).

For W1 and Q1, we agree that both works are closely related, and we state the overlap explicitly:

  • Strategy-conditioned rollouts with transfer to the unguided policy were explored by NudgeRL.
  • The off-context principle (correcting for guided sampling when optimizing the unguided objective) comes from OC-GRPO. Our transfer loss applies this principle; we do not claim it as a contribution.

PSRA's contributions lie elsewhere: what to guide toward, when and how much to guide, and how to keep guidance effective as the policy changes.

NudgeRLOC-GRPOPSRA
Source of guidance2 keyword hints per problem from an external LLM, unverifiedReference-solution prefixes (privileged)Strategy repertoire discovered from the model itself and behaviorally certified
Which arm, how much budgetUniform over contexts, 50% context dropoutMinimal solution-prefix level that breaks all-failure on hard problems (pre-selected, or escalated online)Posterior-based allocation over the unguided arm and all strategy arms under a fixed budget
TransferAdvantage-weighted distillation, no importance correctionPer-token importance ratioSequence-level clipped correction with token-level PPO under the unguided context
Advantage baselineShared across contexts (inter–intra mixing)Per groupPer (q,z)(q, z) arm
Guidance effectiveness during RLNot monitored (guided-context policy trained directly)Not applicableMeasured (internalization vs. loss of accessibility) and maintained by the steerability gate and preservation

Two points sharpen the distinction:

  • NudgeRL identifies the gap PSRA targets. Its limitations section notes that contexts are generated independently of the trained model and that a fixed context pool may become less informative as the policy adapts. It names model-adaptive guidance as future work. PSRA derives its strategies from the model, tracks their steerability Dt(z)D_t(z), gates allocation on it, and preserves accessible routes (Section 3, Fig. 1g).
  • OC-GRPO answers how to learn from guided rollouts, not which guidance to use. Its analysis shows that the variance of the correction grows with the length of the guidance. Our strategy instructions are short, and the observed mismatch is correspondingly small. On 3B, the raw sequence-level weights have median ∣logw∣=0.18|log w| = 0.18, only 8.7% of rollouts reach the clipping range [1/2, 2], and the normalized effective sample size ESS/N=(∑w)2/(N∑w2)\mathrm{ESS}/N=(\sum w)^2/(N\sum w^2) averages 0.93 per batch (minimum 0.81 over training). Further diagnostics are given in our response to Reviewer ghrR (W3).

Targeted controls. A full reproduction of NudgeRL was not feasible within the rebuttal period. Instead, we replace one PSRA component at a time with the closest NudgeRL counterpart available in our ablations, keeping everything else fixed (3B):

VariantNudgeRL counterpartID Avg.OOD Avg.
Full PSRA—28.1 ± .243.5 ± .3
Posterior allocation → random allocationUniform context sampling27.0 ± .240.2 ± .4
Corrected transfer → uncorrected (w=1w = 1)Distillation without importance correction27.6 ± .442.5 ± .7
λ=0.3\lambda = 0.3 to λ=1.0\lambda = 1.0Objective dominated by the guided context26.5 ± .540.3 ± .6
Vanilla GRPO—27.3 ± .338.9 ± .3

Each replacement lowers performance:

  • Random allocation spends the same extra budget on the same arms, but without the posterior. It also lowers the fraction of informative (mixed-reward) groups from 63% to 41%.
  • Uncorrected transfer mainly hurts stability. Compared with the default correction, gradient-norm spikes rise from 4 to 21 per 1k steps, the maximum KL to the reference policy rises from 0.077 to 0.142, and seed standard deviation roughly doubles (ID 0.2 to 0.4). This confirms the off-context principle in our setting.
  • Guided-heavy weighting degrades performance. In our λ\lambda sweep, performance is flat for λ∈[0.1,0.5]\lambda \in [0.1, 0.5], drops sharply from λ≈0.7(27.2/41.6)\lambda \approx 0.7 (27.2 / 41.6), and reaches 26.5 / 40.3 at λ=1.0\lambda = 1.0, where ID falls below GRPO. NudgeRL weights the guided-context RL term 10×10\times higher than distillation, an even more guided-heavy regime, although the loss forms differ.

We will expand our analysis of related work accordingly.

AuthorsOpenReview ↗

W2: Does representation-based discovery help beyond well-designed hand-specified strategies?

We address this at three levels: what discovery adds conceptually, how hand-specified strategies fare on the same diagnostics, and a training control that isolates the two factors involved.

(1) Familiar strategy names are expected; discovery provides what hand-specification cannot. Recovering recognizable strategies is a sanity check, not a limitation. The procedure provides three things beyond the names:

  • Certification. Two of seven candidates fail behavioral certification and are discarded (Paper Fig. 1a,c). One of them, Intuitive, also appears in the hand-specified Fixed Strategy set (App. A.3).

  • Footprints. The coverage term and the steerability gate operate on the representation-space footprints CzC_z; neither can be built from strategy names alone. In a cumulative allocator ablation on 3B (same strategy set, budget, and objective), adding them to the hierarchical prior gives:

    AllocatorID Avg.OOD Avg.Mixed-reward groups
    Hierarchical prior only27.8 ± .242.4 ± .357%
    + coverage term28.0 ± .143.1 ± .261%
    + steerability gate (full PSRA)28.1 ± .243.5 ± .363%

    Together they contribute +0.3 ID / +1.1 OOD. The full ablation and calibration analysis are given in our responses to Reviewers ghrR and Wyt2 (W2).

  • Model-specific fit. Which strategies are steerable and executable differs across models (see (2)).

The procedure is a one-off, inference-only step before training.

(2) Hand-specified strategies fall short on the same diagnostics. We ran the Paper Fig. 1 analyses on the Fixed Strategy set (7B):

DiagnosticDiscovered setFixed Strategy set
Strategies passing certification5 / 52 / 5
Mean steerability (gain in region occupancy, Fig. 1c)0.340.15
Coverage of predicted exploration gaps (GapCov, Fig. 1f)0.800.41
Best-alternative rescue on Qfail\mathcal{Q}_{\text{fail}} at B = 128 (Fig. 3c; unguided: 0.34)0.620.45

Strategy repertoires are also model-specific. Discovery on each scale yields largely overlapping strategy families (Section 3), but which of them the model can reliably follow, and how they are best expressed, differs. We transferred the instructions discovered on 7B to Qwen2.5-1.5B-Instruct and compared them with the 1.5B model's own discovered instructions (steerability):

Strategy7B1.5B, own instructions1.5B, 7B instructions
Stepwise Decomposition0.380.330.30
Equation Setup0.350.310.28
Formula Application0.330.290.25
Backward Reasoning0.360.220.11
Counterfactual0.270.200.15

Two differences stand out:

  • Steerability shrinks with scale. Every strategy is less steerable on 1.5B, and the most steerable strategy on 7B only narrowly passes certification on 1.5B. This matches our observation that the margin by which strategies pass certification shrinks with scale, plausibly because they support a smaller repertoire of distinct reasoning procedures.
  • Expression matters. Instructions distilled from 1.5B's own trajectories steer it better than those transferred from 7B, most markedly for Backward Reasoning (0.22 vs. 0.11).

The overlap in strategy families is an empirical outcome that cannot be known in advance, and even where families overlap, the usable subset and its most effective expression must be determined on the target model. A strategy set specified once, independently of the model, cannot capture this.

Please see the following comment for the remaining part.

AuthorsOpenReview ↗

W2: Does representation-based discovery help beyond well-designed hand-specified strategies? (Part II)

(3) Training control: which strategies vs. how they are expressed. We trained PSRA with three instruction sets. All rows use the full PSRA (allocation, corrected transfer, preservation), with footprints computed by the same intervention-lift procedure (Eq. 1); only the instructions differ (7B):

Strategy instructionsMean steerabilityID Avg.OOD Avg.Rescue on Qfail\mathcal{Q}_{\text{fail}}
Discovered (PSRA)0.3436.940.726%
Hand-written, same five strategies0.2736.340.523%
Fixed Strategy (App. A.3)0.1534.940.319%
Vanilla GRPO—34.637.417%

Two factors emerge:

  • Which strategies are used matters most. Replacing the strategy set (Fixed Strategy) costs 2.0 ID points and 7 points of rescue rate.
  • How strategies are expressed matters as well. Hand-written instructions for the same strategies still work, but they lose 0.6 ID points and 3 points of rescue rate. The discovered instructions are distilled from the model's own trajectories, so they capture the shared semantics with which the model already executes these strategies. They therefore steer more effectively (0.34 vs. 0.27); natural-language instructions retain 88% of the displacement induced by full exemplars (Fig. 1c).

Together, (1)–(3) show that the discovery procedure is not only a way to name strategies. It selects strategies the target model can actually use and expresses them in the form it follows most reliably.

We will add the diagnostics of the Fixed Strategy set, the cross-scale steerability analysis, the per-scale instructions, and the hand-written-instruction control to our paper. We will also correct the description of the Fixed Strategy baseline in Sections 5.1 and 5.3 to match App. A.3, and revise Sections 3 and 6 to state that strategy families largely overlap across scales, while their steerability and most effective expression are model-specific.

Reviewer_ghrROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies finite-sampling exploration failures in RLVR, arguing that an all-incorrect rollout group can reflect concentration on an unsuitable problem-specific strategy rather than the absence of a viable solution procedure. It clusters base-model rollout representations, certifies strategies that can be elicited through prompting, and examines how strategies become internalized or lose conditional accessibility during training. Problem–Strategy Rollout Allocation (PSRA) treats unguided and strategy-conditioned prompts as competing arms, uses probes, hierarchical Bayesian estimates, and non-saturation probabilities to allocate a fixed rollout budget, transfers learning signals to the unguided policy through off-context importance weighting, and retains an on-context objective to preserve strategy accessibility. Experiments cover three Qwen2.5 scales and two training corpora, evaluating reasoning performance, recovery of initially failed problems, saturation, and coverage under larger sampling budgets.

Critical strengths
  1. The explanation of exploration failure is concrete and examined through both representation analysis and behavioral interventions. Beyond observing strategy concentration, the paper compares strategy instructions, full exemplars, and placebo prompts and tests whether strategy switching recovers problems that initially unsuccessful sampling misses. This gives the claim that the model may know another solution route but fail to select it more explanatory value than a generic diversity argument. Distinguishing internalization from loss of conditional accessibility also provides a clear motivation for the preservation objective. (Section 3, Figure 1, lines 144–206)

  2. Results span multiple scales, training distributions, and unguided task evaluations, with relevant ablations. The main text matches total generation budgets to an average of 22 rollouts per prompt. In the 7B DAPO setting, PSRA improves the ID average from 34.6 for GRPO to 36.9 and the OOD average from 37.4 to 40.7. It reports recovery of 26% of problems on which all 16 initial samples fail, versus 17% for GRPO. Strategy-count, random-allocation, additional-budget, and preservation ablations, together with larger-k pass@k results, further support benefits beyond single-sample accuracy. (Sections 5.1–5.3, Table 1, Figures 3–5, Table 2)

Critical weaknesses

W1. The validity of strategy identification and cross-checkpoint measurements requires further verification Section 3 interprets hidden-representation clusters as solution strategies, but representations may also reflect problem content, wording, and length. Certification with instructions, exemplars, and placebo prompts primarily measures displacement toward the corresponding regions; an independent semantic audit is still needed to distinguish changes in solution procedure from changes in expression. Table 2 varies strategy count without testing whether the conclusions remain stable across alternative subsets of the same size. The main text also does not specify whether a common frozen encoder is used across checkpoints or how representation drift is controlled, so changes in occupancy and JS divergence may not reflect only strategy internalization or loss of accessibility. (Section 3, lines 144–203, Figure 1, Eqs. 5–6; Section 5.3, Table 2)

W2. The allocator's independent gains and probability calibration are insufficiently validated The random-allocation control in Table 2 supports targeted allocation but does not isolate the gains of the coverage predictor, hierarchical prior, and steerability gate over simpler scheduling. Fixed Strategy also changes the prompt set, so comparisons using only probe success rates or a simple Beta posterior are needed with the strategy set, budget, and training objectives held fixed. Moreover, non-saturation probabilities are estimated from two probes per arm and a prior, but the main text does not test whether predicted probabilities match observed non-saturation frequencies under the same additional budget. Lower aggregate saturation in Figure 4 does not substitute for calibration or establish whether the predicted ranking reliably identifies arms worth further sampling. (Section 4.1, lines 233–274, Eq. 10; Section 5.3, Figure 4, Table 2)

W3. The implementation and effectiveness of off-context importance correction lack key diagnostics Eqs. 12–13 combine a sequence-level unguided-to-guided probability ratio with token-level PPO ratios and clip the former in log space. Sequence-level ratios can be sensitive to response length and context mismatch, but the main text does not provide the clipping threshold, weight distributions, effective sample sizes, or relevant sensitivity results, making it difficult to determine how much guided signal contributes to unguided updates. Moreover, advantages are normalized within each strategy arm's own reward group; correcting the sampling distribution does not automatically make these advantages equivalent to unguided GRPO advantages. The intended optimization objective and scope of the correction should be specified. The preservation ablation shows the value of conditional updates but does not substitute for validating the transfer weighting itself. (Section 4.2, lines 278–295, Eqs. 11–13; Section 4.3, Eqs. 14–15; Figure 5)

Questions for authors
  1. Regarding W1: Are there independent semantic audits and controls using alternative strategy subsets of the same size that support the strategy definitions and stability of the results? Which encoder is used across checkpoints, and how is the effect of representation drift on internalization and accessibility metrics ruled out?

  2. Regarding W2: Under the same strategy set, budget, and training objectives, how much does the complex allocator improve over probe-success-rate or simple Beta-posterior scheduling? Could predicted non-saturation probabilities be compared with observed frequencies under the same additional budget, checking whether higher predictions identify arms more likely to yield mixed-reward groups?

  3. Regarding W3: What is the sequence-level log-weight clipping threshold and implementation, and how do weights and effective sample sizes vary with context mismatch and trajectory length? What objective is optimized by combining within-arm advantages with off-context weighting? Are there controls removing the correction or varying clipping strength to establish its contribution and stability?

5 comments
AuthorsOpenReview ↗

Response to W1/Q1: Validity of strategy identification and cross-checkpoint measurement

We thank the reviewer for the careful review and for raising this point. We address it in three parts. The results reported here are built on Qwen2.5-7B-Instruct with DAPO-Math-17K as training data.

(1) A common frozen encoder is used across all checkpoints.

All representation-space measurements use a single extractor and DPMM frozen at the base checkpoint (Appendix. A.1.1, A.1.6). Rollouts generated by checkpoint πt\pi_t are re-encoded by the frozen π0\pi_0 and assigned to the frozen components. Footprints Cz(0)C_z^{(0)} are never re-clustered during RL. Changes in occupancy and JS divergence therefore reflect changes in what the policy generates, not changes in the coordinate system. We will move this statement into Section 3.

The frozen encoder could still respond to stylistic drift in trained-policy outputs, such as wording or length. We test this directly with surface perturbations of 2000 rollouts drawn from base, GRPO, and PSRA checkpoints:

PerturbationSame componentSame strategy footprint
Procedure-preserving paraphrase (wording changed)84.2%91.5%
Length-matched compression (redundant text removed)81.7%89.8%
Procedure-changing rewrite (different method)18.6%22.4%

Assignments are stable under surface changes and sensitive to procedural changes. This holds across base, GRPO, and PSRA rollouts, with less than 3 points of variation between checkpoints.

(2) Independent semantic audit.

We labeled rollouts with an external LLM judge (Qwen3-235B-A22B). The judge saw only the five strategy definitions plus an "other/none" option, with no access to clusters, conditions, or checkpoints. To validate the judge itself, two annotators independently labeled a random subset of 300 rollouts.

Judge reliability. Human–human κ=0.76κ = 0.76; judge–human (majority) κ=0.73κ = 0.73.

Agreement between footprints and judge labels (4000 base-model rollouts):

StrategyPrecisionRecall
Stepwise Decomp.0.810.77
Equation Setup0.840.80
Formula Application0.780.74
Backward Reasoning0.860.71
Counterfactual0.790.68
OverallCohen's κ = 0.70NMI = 0.58

Conditioning changes the procedure, not just the wording. We measured how often the judge labels a rollout with the target strategy under each condition:

ConditionLabeled as target strategy
Unguided21.4%
Matched placebo23.0%
Instruction-conditioned72.6%
Exemplar-conditioned80.3%

Instructions recover 87% of the exemplar effect measured by judge labels. This closely matches the 88% displacement ratio in representation space (Fig. 1c), so the two independent measurements agree.

Content confounds. We trained linear probes to predict a rollout's strategy footprint from candidate confounds:

Probe inputMacro-F1
Majority class0.11
Response length0.19
Problem topic (7 MATH subjects)0.28
Problem embedding (prompt only)0.35
Rollout representation es(y)e_s(y)0.83

Within a single problem, rollouts occupy a median of 3 distinct footprints. This also rules out a clustering that simply partitions problems by content.

Cross-checkpoint consistency. We recomputed autonomous strategy occupancy from judge labels instead of footprints. Judge-based internalization correlates strongly with the representation-based ιtι_t across strategies and checkpoints (Spearman ρ=0.81ρ = 0.81 for PSRA, 0.770.77 for GRPO). The qualitative pattern of Fig. 1g is unchanged. Under PSRA, reductions in displacement are explained by internalization. Under GRPO, Counterfactual and Backward Reasoning lose steerability without a matching increase in autonomous use.

(3) Stability across alternative subsets of the same size.

Protocol of the original Table 2. The size ablation uses nested subsets ordered by base-model autonomous frequency (Fig. 1a): #1 = {Stepwise Decomposition}; #2 adds Equation Setup; #3 adds Formula Application; #4 adds Backward Reasoning; the full set adds Counterfactual. We will state this in the Table 2 caption.

Additional random subsets. For each k ∈ {2, 3, 4}, we trained 3 additional randomly drawn subsets. Values are mean ± std over the random subsets; the original frequency-ordered subset is in brackets.

#StrategiesID Avg.OOD Avg.
5 (full)36.940.7
436.4 ± 0.3 [36.3]40.2 ± 0.4 [40.3]
335.9 ± 0.5 [36.1]39.6 ± 0.7 [40.3]
235.5 ± 0.9 [34.6]38.9 ± 1.0 [38.0]
GRPO34.637.4

Performance degrades steadily, and all subsets with k ≥ 3 stay within ~1 point of the full set while clearly outperforming GRPO.

We will revise in the final version of our paper.

AuthorsOpenReview ↗

Response to W2/Q2: Independent contribution of the allocator and calibration of non-saturation probabilities

We thank the reviewer for this suggestion. We ran the requested controls on Qwen2.5-3B-Instruct with DAPO-Math-17K as the training set. Every variant uses the same five-strategy set, the same total budget (22 rollouts per prompt), and the same training objective; only the allocation rule differs. Results are averaged over 3 seeds (Paper Table 1 reports a single run with seed 42).

(1) Component-wise contribution.

We build the allocator up cumulatively, starting from simple scheduling rules:

Allocation ruleID Avg.OOD Avg.Non-saturated groups*
Random allocation27.0 ± .240.2 ± .441%
Probe success rate (heuristic)27.4 ± .241.0 ± .347%
Simple Beta posterior (uniform prior) + U(k)U(k)27.6 ± .141.7 ± .352%
+ hierarchical prior (c^\hat{c}, β^\hat{\beta}; ω\omega = 0, no gate)27.8 ± .242.4 ± .357%
+ coverage term (learned ω_t)28.0 ± .143.1 ± .261%
+ steerability gate (full PSRA)28.1 ± .243.5 ± .363%
Vanilla GRPO27.3 ± .338.9 ± .3—

Note that the “Probe success rate” variant allocates first to arms whose probe success rate is closest to 0.5. *Fraction of (q, z) groups with mixed rewards.

Each component yields a consistent gain, and the gains track the fraction of informative (non-saturated) groups. The roles are as follows:

  • Hierarchical prior. Probe-only rules cannot distinguish arms with identical probe outcomes. In particular, all 0/2 arms look the same to them. The prior separates these arms using problem difficulty and the global arm effect.
  • Coverage term. Its contribution is selected by the data, not imposed. The coefficient ωt\omega_t starts at 0 and is learned online by likelihood (App. A.1.7). It rises to ≈0.6 within the first 150 steps and stays positive throughout training, so predicted uncovered mass carries real information about arm success.
  • Steerability gate. Its gain is concentrated in later training (after step 400). This is when Dt(z)D_t(z) decays for some strategies, and the gate stops spending budget on handles that no longer move the policy.

(2) Calibration and ranking of Uq,z(k)U_{q,z}(k).

Arms with mixed probes have U=1U = 1 by definition and are already informative. Calibration therefore matters only for saturated-probe arms (s=0s = 0 or s=n0s = n_0), which is where the allocator actually makes decisions.

Allocated rollouts reveal outcomes only for the arms that were chosen, which would bias the evaluation. To avoid this, at three checkpoints (steps 200 / 500 / 700) we gave every saturated-probe arm the maximum additional budget k = 6, regardless of the allocator's choice. We then compared the predicted U(6) with the observed non-saturation frequency (≈ 9.6k arms in total).

PredictorECE ↓Brier ↓AUROC ↑
Probe success rate——0.50
Simple Beta posterior0.1420.2040.50
+ hierarchical prior0.0480.1710.71
+ coverage term0.0360.1620.76
Full PSRA0.0310.1580.78

The two probe-only predictors give the same score to every arm with the same probe outcome. This is why their AUROC is 0.50 on this subset; they cannot rank saturated arms at all. The ECE and Brier entries for the probe-rate rule are left blank because it is a ranking heuristic, not a probability.

Reliability across predicted-probability bins (full PSRA):

Predicted U(6)[0, .2)[.2, .4)[.4, .6)[.6, .8)[.8, 1]
Observed non-saturation0.090.270.480.710.84

The predicted probabilities are well calibrated, and the ranking is informative. Arms in the top quintile of U yield a mixed-reward group 4.6× more often than arms in the bottom quintile. Calibration is stable across the three checkpoints (ECE 0.029 / 0.031 / 0.034), so the online prior tracks the evolving policy.

We will add these ablations and calibration results to Section 5.3.

AuthorsOpenReview ↗

Response to W3/Q3: Implementation and effectiveness of the off-context correction

We thank the reviewer for these questions. All new results use the same setting as W2 (Qwen2.5-3B-Instruct, DAPO-Math-17K, 3 seeds). The preservation term is kept in every variant, so the transfer weighting is isolated from preservation.

(1) Implementation and weight diagnostics.

The sequence-level weight is computed as

log⁡w(y)=∑u[log⁡πθold(yu∣c(q,∅),y<u)−log⁡πθold(yu∣c(q,z),y<u)],\log w(y)=\sum_u \big[\log \pi_{\theta_{\text{old}}}(y_u \mid c(q,\varnothing), y_{<u}) - \log \pi_{\theta_{\text{old}}}(y_u \mid c(q,z), y_{<u})\big],

using an additional forward pass under the unguided context. It is then clipped symmetrically in log space to [log⁡12,log⁡2][\log\tfrac12, \log 2] (App. A.4).

Statistics of the raw (pre-clipping) weights over all guided rollouts:

AllLength < 512512–1024> 1024
Medianlog w0.180.12
90th pct.log w0.610.44
Fraction clipped8.7%4.1%8.3%15.2%

Most weights lie inside the clipping range, so the correction is an actual reweighting rather than a binary rescaling at the boundaries. Mismatch grows moderately with length.

Across strategies, the clipped fraction ranges from 5.9% for Formula Application to 12.8% for Counterfactual, which has the instruction least similar to the unguided prompt.

The normalized effective sample size, ESS/N=(∑w)2/(N∑w2)\mathrm{ESS}/N=(\sum w)^2/(N\sum w^2), averages 0.93 per batch, with a minimum of 0.81 over training.

The mismatch is small because the strategy instruction mainly shifts the early tokens of a trajectory. Once the route is set, the per-token conditionals under c(q,z)c(q,z) and c(q,∅)c(q,\varnothing) nearly coincide. This is consistent with our premise that these strategies are already supported by the unguided policy (Section 3). Mismatch also decreases during training (median |log w| falls from 0.24 to 0.13), in line with the internalization dynamics in Fig. 1g.

(2) The optimized objective.

For a guided arm zz with behavior distribution μz=πθold(⋅∣c(q,z))\mu_z=\pi_{\theta_{\text{old}}}(\cdot\mid c(q,z)) and exact weights,

Ey∼μz[w(y) (r(y)−b) ∇θlog⁡πθ(y∣c(q,∅))]=∇θJ∅(q)E_{y\sim\mu_z}\big[w(y)\,(r(y)-b)\,\nabla_\theta\log\pi_\theta(y\mid c(q,\varnothing))\big]=\nabla_\theta J_\varnothing(q)

holds for any constant bb. This is because

Eμz[w ∇log⁡πθ(⋅∣c(q,∅))]=Eπ∅[∇log⁡πθ]=0.E_{\mu_z}[w\,\nabla\log\pi_\theta(\cdot\mid c(q,\varnothing))]=E_{\pi_\varnothing}[\nabla\log\pi_\theta]=0.

The within-arm mean μq,z\mu_{q,z} is therefore a valid baseline. The advantages do not need to equal unguided GRPO advantages to target the unguided objective. The arm-specific baseline is also the lower-variance choice, because it matches the mean reward of the distribution that actually generated the samples.

The estimator departs from this unbiased gradient in two places:

  • Division by σq,z\sigma_{q,z}. This is a per-group rescaling of the same kind as GRPO's own normalization.
  • Clipping of ww. This affects only 8.7% of rollouts.

Together with the token-level PPO ratio, which provides a trust region relative to the old unguided policy, LtrL_{\text{tr}} is a bounded-variance surrogate for ∇J∅\nabla J_\varnothing in which every source of bias is explicit and small.

(3) Controls: removing the correction and varying the clipping range.

VariantID Avg.OOD Avg.ClippedESS/NGrad-norm spikes / 1k stepsMax KL to ref.
No correction (w≡1w ≡ 1)27.6 ± .442.5 ± .7——210.142
Clip [0.8, 1.25]28.0 ± .143.4 ± .329.6%0.9750.081
Clip [0.5, 2] (default)28.1 ± .243.5 ± .38.7%0.9340.077
Vanilla GRPO27.3 ± .338.9 ± .3————

Spikes are counted as gradient norms exceeding 3× the running median.

Removing the correction lowers final performance and, more markedly, destabilizes training:

  • seed standard deviation roughly doubles;
  • gradient-norm spikes increase about fivefold;
  • the policy drifts further from the reference.

Without reweighting, guided trajectories are credited to the unguided context as if they were on-policy. Rollouts that are unlikely under c(q,∅)c(q,\varnothing) then receive full-magnitude updates, which produces these spikes.

The two clipping ranges perform similarly. Tightening to [0.8, 1.25] clips about a third of rollouts yet loses only 0.1 points. The benefit therefore comes from down-weighting high-mismatch trajectories, and the method is not sensitive to the exact threshold.

We will also add these additional analyses to our paper.

Comments on the paper

1 comment

We sincerely thank all reviewers for the considerable time and effort they have devoted to evaluating our work. We are deeply grateful for their recognition of our contributions and, more importantly, for their thoughtful and constructive feedback. The comments and suggestions from all reviewers have been invaluable, each playing an important role in helping us improve the quality, clarity, and rigor of our work. We will carefully incorporate this feedback and further refine the manuscript in the final version. Once again, we extend our heartfelt thanks to all reviewers, ACs, SACs, and PCs for their time and support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.