acceptodds
Under review as a conference paper at ICLR 2027

Boosting LLM Exploration via Weak-Model Guidance in RLVR

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@ for large . While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.

Primary area:
reinforcement learning
Keywords:
Weak-to-Strong Exploration, Mathematical Reasoning, Reinforcement Learning with Verifiable Rewards, Policy Entropy, Trajectory Completion, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cgDDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to mitigate entropy collapse in RLVR training and the resulting drop in pass@k at large k. The authors propose introducing partial solutions generated by weaker small models (Gemma-2-2B and LLaMA-3.2-1B) as prefixes during GRPO training. Specifically, the weak model first generates a complete response. The response is then truncated at the position with the largest entropy drop, measured by the magnitude of entropy change. The target policy model continues generation from this point, and its gradient updates are applied only to the completion it generates itself. Notably, no prefix is used at evaluation time; instead, the policy model directly generates the complete response. Experiments on mathematical tasks are conducted with two Qwen2.5 models.

Critical strengths
  1. The method is simple and fairly interesting. It introduces generations from another model to avoid diversity collapse during RL training, giving the policy an opportunity to learn alternative reasoning paths.
  2. The evaluation takes practical value into account by returning to the standard paradigm of direct generation by the policy model.
  3. Generating prefixes with a small model is relatively cheap and does not introduce much additional overhead.
Critical weaknesses
  1. Limited novelty. Using partial solutions as prefixes or hints in RLVR training and letting the model continue the remaining reasoning has already been explored in multiple works accepted at top conference [1–3]. The main difference in this paper is only that the prefix comes from a weak model and is interpreted as a “perturbation.” However, the paper cites only QuestA and does not compare against any prefix- or hint-based method in its experiments.
  2. No correctness check on weak-model prefixes. The paper does not perform any correctness check on the prefixes generated by the weak model. Two situations can be expected: (1) when the prefix is wrong, all 8 continuations often fail; (2) when the prefix is wrong, the subsequent reasoning deviates from the prefix’s logic and reaches the correct answer by chance, and this kind of error is also reinforced.
  3. Relation to data augmentation and off-policy training. This paper trains the model on states produced by an external model, which is closer to data augmentation or an off-policy state distribution. The paper should discuss this more.

References

[1] QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation. ICLR 2026.

[2] BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning. NeurIPS 2025.

[3] StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason. ACL 2026.

Questions for authors

See above.

1 comment

We thank the reviewer for the detailed comments on related work and the training mechanism.

Regarding the distinction from prior work, we agree that the original manuscript did not sufficiently discuss QuestA, BREAD, and StepHint. We will expand the related-work section and clarify that our novelty does not lie in the general idea of using partial trajectories as prefixes. Rather, we investigate a different role for auxiliary trajectories: instead of treating them primarily as correct hints or demonstrations, we use trajectories from models with different generation distributions as training-time conditioning contexts, even when their semantic guidance is imperfect. The target policy generates and learns from the continuation, while the auxiliary prefix itself is excluded from the policy loss. No auxiliary prefix is used during inference or evaluation.

Regarding incorrect or misleading prefixes, we agree that they can negatively affect the conditioning context and potentially lead to unproductive rollouts. In our implementation, all eight rollouts for a given question share the same prefix. If all eight completions receive zero reward, their group-normalized advantages are zero, and this group provides no effective policy-gradient signal under the standard GRPO formulation. Such cases therefore waste rollout computation and reduce the effective training signal. However, if the target policy successfully continues from a prefix and obtains a non-zero reward relative to other completions, the generated suffix can still receive a useful learning signal. Importantly, the prefix itself is never directly optimized. We also retain 80% question-only GRPO rollouts, which allows the target policy to continue learning from the original question distribution rather than relying entirely on auxiliary contexts.

Regarding the on-policy nature of the method, we agree that describing the entire training procedure as standard question-only on-policy GRPO would be imprecise. The suffix is sampled from the current target policy conditioned on an externally generated prefix, and its policy loss is computed using the target policy. However, the context distribution includes externally generated prefixes and therefore differs from the original question-only rollout distribution. We will describe the method more accurately as GRPO with a mixed context distribution, or mixed-context trajectory augmentation. Unlike conventional distillation, our method does not optimize the target policy to imitate the auxiliary prefix, nor does it require auxiliary-model token probabilities for the prefix. The learning objective is applied only to the target-generated continuation.

We will revise the method description to make this distinction explicit and discuss the potential negative effects of unproductive prefixes, including reduced effective rollout efficiency.

Reviewer_k82eOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the loss of response diversity caused by increasingly concentrated response distributions during reinforcement learning with verifiable rewards (RLVR). To mitigate this issue, the authors propose leveraging initial trajectories generated by a weaker model to enhance entropy-based diversity during training. Experiments conducted primarily with the Qwen2.5-7B model demonstrate substantial improvements in pass@kk metrics, suggesting that the proposed approach can improve the diversity of candidate solutions and increase the likelihood of finding correct answers among multiple sampled responses.

Critical strengths

Strength 1: An Interesting and Counterintuitive Approach to Mitigating Entropy Collapse

The paper investigates an important and persistent issue in RLVR: entropy collapse and the resulting loss of response diversity. The authors propose an interesting and somewhat counterintuitive idea: instead of using a stronger model to guide a weaker one, they leverage trajectories generated by a weaker model to encourage diversity in a stronger model's exploration process. This reverse guidance paradigm is conceptually intriguing and offers a different perspective on addressing the loss of diversity during RLVR training.

Strength 2: An Intuitive and Lightweight, Training-Free Design

The proposed method is intuitive and relatively simple to implement. By using partial trajectories generated by a weaker model as prefixes to intervene in the stronger model's exploration process during training, the approach encourages exploration without requiring additional model training or a complex optimization procedure for the guiding model. This training-free design is a practical strength, as it provides a straightforward way to introduce alternative reasoning trajectories into the exploration process while keeping the overall training framework largely unchanged.

Critical weaknesses

Weakness 1: Limited Model Diversity in the Evaluation

The experimental evaluation relies exclusively on Qwen2.5-7B, making it difficult to assess whether the proposed method generalizes across different model architectures, model scales, or base-model capabilities. Experiments on additional models are needed to establish the robustness and general applicability of the approach, rather than demonstrating its effectiveness only within a single model configuration.

Weakness 2: Insufficient Analysis of Prefix Quality and Its Contribution to Learning

2.1. Unverified Prefix Quality and Unclear Sources of Improvement. The authors randomly sample 200 prefixes and classify them using DeepSeek-V4-Flash, without human verification. However, the observation that most prefixes in the training data are of low quality does not establish that these low-quality prefixes contribute positively to learning. The observed performance gains could still be driven primarily by a small subset of high-quality prefixes. A more rigorous analysis, such as evaluating the effect of prefixes stratified by quality or comparing against controlled mixtures of high- and low-quality prefixes, is needed to determine which types of prefixes actually drive the improvements.

2.2. Potential Confounding in the Training-Entropy Analysis. Figure 3 compares training entropy across different prefix-injection ratios. However, these settings involve different input conditions, making the interpretation of the entropy differences ambiguous. Higher entropy under prefix injection may simply reflect greater uncertainty when continuing externally supplied or less familiar prefixes, rather than indicating that the model has preserved a richer distribution of reasoning paths. The current analysis does not clearly disentangle input-induced uncertainty from genuine improvements in the diversity of the model's learned reasoning behavior. Additional evaluations under matched input conditions, or direct measurements of response diversity on a common held-out prompt set without prefix injection, would help clarify this distinction.

Weakness 3: Missing Analysis of Computational Overhead and Efficiency

The proposed method introduces additional computational requirements, including generating auxiliary trajectories and evaluating the target base model's entropy to determine where prefixes should be truncated. However, the paper does not report the associated computational overhead, such as total GPU-hours, preprocessing costs, or the overall training budget.

Without a comprehensive accounting of these costs, it is difficult to assess the method's practical efficiency and its trade-off between performance gains and additional computation. Reporting the incremental cost relative to standard RLVR training, together with the resulting performance improvement, would provide a more complete evaluation of the method's practical value.

Questions for authors

Q1. Do the Benefits Generalize Across Model Families and Scales?

The evaluation is limited to Qwen2.5-7B and Qwen2.5-Math-7B. Could the authors provide results on target models from a different model family or at different parameter scales to assess whether the benefits generalize beyond the current setting?

Q2. Do Low-Quality Prefixes Actually Contribute to the Improvements?

Could the authors manually validate a subset of the prefix-quality labels and conduct quality-controlled ablations using correct, uninformative, and misleading prefixes separately? Matching the training questions, prefix lengths, and training budgets across conditions would help isolate the effect of prefix quality. In particular, such experiments could determine whether low-quality prefixes genuinely contribute to the observed gains or whether the improvements are primarily driven by the high-quality subset.

Q3. Does Higher Training Entropy Reflect Better Exploration or Merely More Difficult Inputs?

In Figure 3, is policy entropy measured over each run's own mixture of prefix-conditioned and question-only inputs? If so, differences in input distributions may confound the comparison.

Could the authors evaluate intermediate and final checkpoints on a shared, fixed set of question-only prompts and report entropy alongside pass@kk? This would help distinguish uncertainty induced by difficult external prefixes from genuine improvements in the model's exploration behavior without prefix guidance.

Q4. What Is the Computational Overhead, and Does Entropy-Based Truncation Justify Its Cost?

Could the authors report the computational costs of auxiliary trajectory generation, entropy-based truncation, and RL training separately, together with the total GPU-hours?

Furthermore, under a matched total computational budget, does the proposed method retain its advantages over GRPO and SimKO? Does entropy-based truncation provide a meaningful benefit over random truncation when the remaining experimental conditions and computational budgets are controlled?

These comparisons would clarify whether the method's performance gains justify its additional computational overhead and whether entropy-based truncation is essential to its effectiveness.

1 comment

We thank the reviewer for identifying several important limitations concerning generalization, prefix quality, and the interpretation of entropy.

Regarding generalization, we acknowledge that all target models in our current experiments belong to the Qwen2.5 family. Although we investigate auxiliary trajectories from different model families, including Gemma and LLaMA, this does not establish generalization across target-model families. We will explicitly narrow our claims and identify cross-family evaluation as an important direction for future work.

Regarding prefix quality, we agree that the current evaluation is insufficient to establish a causal relationship between semantic quality and exploration. In our analysis of 200 sampled Gemma prefixes, DeepSeek-V4-Flash classifies 71% as providing no substantial guidance or being potentially misleading. Together with the source-model ablation, this observation suggests that the semantic quality of auxiliary guidance alone may not fully account for the performance gains. In particular, prefixes from Gemma-2-2B achieve a higher Pass@128 than prefixes from Qwen2.5-1.5B and Qwen2.5-7B in our experiments.

Nevertheless, we acknowledge two limitations: the prefix-quality labels are generated by an LLM judge without human verification, and the remaining high-quality prefixes may contribute disproportionately to the observed gains. Moreover, we have not controlled prefix quality and length independently. Thus, we will avoid claiming that low-quality or misleading prefixes themselves have been proven to improve exploration. A more precise interpretation is that auxiliary trajectories with imperfect semantic guidance can still be useful in our training framework, and that their usefulness may depend on factors beyond semantic correctness.

Regarding the entropy analysis, we agree that higher entropy under prefix-conditioned inputs does not necessarily indicate increased diversity under the original question-only distribution. The entropy measurements in our current figures are obtained under their respective conditioning contexts. Therefore, higher entropy may partly reflect additional uncertainty induced by unfamiliar or different contexts, rather than an intrinsic increase in the target policy’s question-only exploration. We will revise the discussion to describe these results as conditional-uncertainty observations, not direct evidence of increased learned diversity. The improvements in large-k performance are consistent with our exploration hypothesis, but do not by themselves establish the proposed mechanism causally.

Regarding computational overhead, our approach requires auxiliary trajectory generation and target-model entropy scoring during preprocessing. The generated prefixes are then reused during RL training, avoiding repeated auxiliary generation and entropy computation during each rollout. We acknowledge that these preprocessing stages introduce additional computational costs, and that our current evaluation does not provide a complete accelerator-hour or token-level cost accounting. We will clarify this limitation and avoid describing the overhead as negligible. A compute-matched comparison would be valuable future work.

Reviewer_uUfrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether partial reasoning trajectories from weaker models can improve exploration during RLVR. It mixes ordinary question-only GRPO training with continuation from auxiliary prefixes, using a target-model entropy-drop heuristic to select where to truncate those prefixes. Prefix tokens are excluded from the policy loss, and evaluation uses the original question without auxiliary guidance. On two Qwen2.5 7B models, the method improves average large-k coverage over vanilla GRPO, while single-sample and per-benchmark results are mixed. The paper also examines training entropy, prefix quality, and choices of prefix source and truncation.

Critical strengths
  1. A simple intervention with no auxiliary-model requirement at evaluation. The method changes the training inputs without introducing a new policy objective, additional supervised fine-tuning, or inference-time guidance. Its use of imperfect auxiliary trajectories is a worthwhile direction to explore, especially when a stronger teacher is unavailable.

  2. The empirical presentation is reasonably careful. The paper includes Pass@k Optimization and SimKO rather than comparing only against vanilla GRPO. It also reports per-benchmark results and ablations of prefix source, injection probability, and truncation. The discussion acknowledges mixed results and treats distributional perturbation as a hypothesis. This makes the paper more credible than a claim of universal improvement from another training modification.

Critical weaknesses
  1. The methodological increment is narrow, and the analysis does not sufficiently establish the distinctive contribution. External trajectories and training-time guidance are already established directions, including LUFFY (which the paper discusses in Appendix A). This method is different: the auxiliary text is a conditioning prefix rather than a demonstration being directly optimized, and the source model need not be strong. I therefore do not regard it as simply the same algorithm. However, the main contribution then rests on what is specifically gained from weak, cross-model prefixes and the entropy-based truncation rule.
  2. The practical gains are not strong or robust enough to compensate for the limited methodological novelty. Gemma prefixes improve average pass@128 over GRPO by 2.95 points on Qwen2.5-7B and 1.92 points on Qwen2.5-Math-7B. These are positive results, but the corresponding pass@1 changes are only 0.72 and 0.03 points. On the Math model, SimKO is stronger on both metrics: 41.64/74.41 versus 40.11/71.22 for pass@1/pass@128. Both prefix variants also remain below their respective base models in pass@128, although they retain the substantial single-sample gains from RL; this is partial coverage recovery, not evidence of no benefit.
Questions for authors
  1. What isolates the value of weak cross-model guidance? Do you have a comparison that controls prefix length and quality while varying the source or distributional mismatch? Alternatively, can a focused quality-stratified experiment establish whether the low-quality prefixes themselves contribute to the improvement? Either would materially strengthen the main interpretation.
  2. Is the entropy-based truncation worth its additional cost? How stable is its 0.52-point pass@128 advantage over random truncation across independent training runs, and what are the preprocessing time and token costs?
  3. When should a practitioner choose this method? Given the SimKO results on Qwen2.5-Math-7B, what supported condition makes weak-model prefixes the preferable choice? A focused comparison of performance and total training/preparation cost would help. I am not requesting a broad new benchmark suite.
1 comment

We thank the reviewer for the thoughtful comments. We agree that the use of partial reasoning trajectories or hints as prefixes has been explored in prior work, and we will revise the related-work discussion to better distinguish our study from LUFFY, QuestA, BREAD, and StepHint.

Our central contribution is not to introduce another method for providing better reasoning guidance, but to investigate whether auxiliary trajectories from models with different generation distributions can serve as useful exploration contexts for a target policy, even when their semantic guidance is imperfect. Unlike approaches that primarily use auxiliary trajectories as demonstrations or correct hints, our method studies their role as alternative conditioning contexts during RL training. The auxiliary prefix is not included in the policy loss; only the continuation generated by the target policy is optimized. Furthermore, prefixes are used only during training, while evaluation follows the standard question-only setting.

Our empirical observations are consistent with this perspective. For Qwen2.5-7B, using Gemma-2-2B prefixes improves Pass@128 from 67.76 to 70.71, while Pass@1 changes from 38.29 to 39.01. We also evaluate 200 sampled Gemma prefixes using DeepSeek-V4-Flash, which classifies 71% as providing no substantial guidance or being potentially misleading. In addition, our source-model ablation shows that Gemma-2-2B prefixes achieve a higher Pass@128 (70.71) than Qwen2.5-1.5B (67.22) and Qwen2.5-7B (68.79) prefixes. These observations suggest that semantic guidance quality alone may not explain the observed gains.

However, we acknowledge that these experiments do not isolate the effect of generation-distribution differences from prefix quality and length. The prefix-quality evaluation is also based on an LLM judge rather than human annotation. Therefore, we will present these findings as empirical observations supporting our hypothesis, rather than as causal evidence that misleading prefixes independently improve exploration.

Regarding the comparison with SimKO, we agree that SimKO achieves stronger results on Qwen2.5-Math-7B (Pass@1/Pass@128: 41.64/74.41) than our method (40.11/71.22). We do not intend to claim that our approach outperforms SimKO or replaces methods explicitly optimizing Pass@k. Instead, the two approaches address different questions: SimKO focuses on improving Pass@k, whereas our study investigates whether changing the training-time context distribution can provide additional exploration opportunities. Our results demonstrate partial recovery over vanilla GRPO, rather than superiority over existing optimization methods.

Finally, we agree that the improvement from entropy-based truncation over random truncation is modest (0.52 Pass@128 points: 70.71 vs. 70.19). We will clarify that entropy-based truncation is a practical, label-free heuristic rather than the central contribution of the paper. We will also better acknowledge its additional preprocessing cost, including auxiliary trajectory generation and target-model entropy scoring. These computations are performed before RL training, and the resulting prefixes are reused during training; nevertheless, we do not claim that the overhead is negligible. We will clarify these limitations and discuss the trade-off between implementation cost and potential exploration benefits.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.