acceptodds
Under review as a conference paper at ICLR 2027

Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) is a scalable paradigm for improving the mathematical reasoning of large language models, but it is fundamentally limited by exploration: the policy can only improve on trajectories it has already sampled. Sampling more rollouts alleviates this problem, but at a prohibitive computational cost, while objective-level modifications offer little control over what is explored. We propose NudgeRL, a framework for structured, diversity-driven exploration in RLVR whose core component, Strategy Nudging, conditions each rollout on a lightweight strategy-level context, without requiring the context generator to solve the problem itself. To learn from such exploration, we decompose the advantage into inter- and intra-context terms and add a policy correction term that transfers discovered behaviors back to the base policy. Across five mathematical reasoning benchmarks, NudgeRL with 8 rollouts matches the strongest GRPO baseline using 32 rollouts with roughly fewer total tokens and less training compute. On code generation, it also outperforms GRPO with 64 rollouts using only one-eighth as many. These results establish strategy-guided exploration as a compute-efficient alternative to brute-force rollout scaling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, RLVR

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_GBxTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NudgeRL, a framework for improving exploration efficiency in RLVR through lightweight strategy-level hints. It combines context-conditioned rollout generation, inter- and intra-context advantage estimation, and a policy learning objective that transfers learned behaviors to the unconditioned policy. Experiments on mathematical reasoning and code generation show that NudgeRL matches or outperforms GRPO baselines with substantially fewer rollouts and lower estimated training compute.

Critical strengths
  • Good computational efficiency. The experimental results show promising effectiveness of NudgeRL. For example, with only 8 rollouts, NudgeRL matches the strongest GRPO (rollout batch size=32) baseline on average across multiple math benchmarks, using approximately 5x fewer tokens and 2.9x less estimated training compute.
  • Comprehensive analysis of method mechanism. This paper conducts experiments over two model families for the math reasoning task, and extend to the domain of code generation, where NudgeRL can still outperforms GRPO. Moreover, this paper also includes ablations on context dropout, hint selection, advantage weighting, and policy correction.
Critical weaknesses
  • Insufficient comparison with closely related methods. The experiments primarily compare against rollout-scaled GRPO, without directly evaluating relevant hint-based approaches such as NuRL [1], SAGE [2], RLTF-SD[3]. The simplified oracle-prefix baseline does not fully address this gap, leaving the advantages of the proposed strategy-guided exploration over existing methods unclear.
  • Limited evaluation of compute allocations. Combining Table 1 and Table 3, it is clearly evident that NudgeRL reduces computational overhead while maintaining performance. However, I suggest that the perspective on cost control or scaling for GRPO is not comprehensive. The configurations of current experiments use the same number of training steps, leaving alternative allocations of the GRPO budget unexplored. For instance, under a fixed budget, we can use less rollout batch size for more training steps. And in this paper, NudgeRL-8 uses approximately 44% more compute than GRPO-8, but the paper does not evaluate whether allocating this additional budget to more GRPO training steps could narrow the performance gap. Given that NudgeRL does not significantly improve task performance, I suggest including more fair comparisons to highlight its contributions to efficiency.

[1] Chen, Justin Chih-Yao et al. “Nudging the Boundaries of LLM Reasoning.”

[2] Liao, Baohao et al. “Self-Hinting Language Models Enhance Reinforcement Learning.”

[3] Song, Yu-Da et al. “Expanding the Capabilities of Reinforcement Learning via Text Feedback.”

Typos:

  • p.4, L180–181: objective that learns … and distill(s) effective strategies
  • p.9, L464–465: via (a) policy correction augmented RL objective
Questions for authors

Please refer to the Weaknesses.

Reviewer_VRtROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NUDGERL, which guides RLVR exploration with strategy prompts. It combines prompt dropout, within- and between-group advantage decomposition, and policy correction to help the unprompted policy learn from useful trajectories found with prompts. Training samples a total of (N=8)(N=8) responses per problem. Math and coding experiments show that NUDGERL matches or outperforms GRPO while using fewer rollouts.

Critical strengths
  1. The motivation and mechanism are intuitive, and the training results show clear gains in both performance and efficiency.
Critical weaknesses
  1. The prompts in Section 5.1 are generated for each problem by GPT-4o-mini. Although they are not filtered for correctness, they still provide external strategy information. The ablations on prompt sources and method components lack a control using one fixed prompt per problem, standard GRPO advantages, and the same policy correction. This makes it hard to separate the gains from exploring multiple strategies from those of simply training with prompts.

  2. The evidence for gains on initially difficult problems is still indirect. The paper includes challenging benchmarks such as AIME and APEX, along with pass@k analysis, but does not break down gains by the initial policy’s success rate. It is therefore unclear whether the method systematically helps on problems the initial policy struggles to solve.

Questions for authors
  1. With (N=8)(N=8) and the same prompt source, dropout rate, policy correction, and unprompted evaluation setup, how does standard GRPO with one fixed prompt per problem compare with full NUDGERL?

  2. Using the existing initial-policy samples, could you report gains separately for problems with low success rates and the rest? For problems with no successes among those samples, does NUDGERL solve more of them than GRPO in the final unprompted evaluation?

Reviewer_a8xsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NudgeRL, a framework for improving exploration efficiency in reinforcement learning with verifiable rewards (RLVR). Instead of increasing the number of rollouts, the method uses strategy-level hints generated by an external LLM to encourage the policy to explore different reasoning approaches. To learn from these context-conditioned rollouts, the authors introduce an Inter-Intra Group Advantage estimator that accounts for both trajectory-level rewards and context-level performance. A policy correction objective is further used to transfer the discovered behaviors to the original policy, so that no additional hints are needed at inference time.

The method is evaluated on five mathematical reasoning benchmarks using Qwen3-4B and Olmo3-7B, as well as on LiveCodeBench using Qwen2.5-Coder-7B. With 8 rollouts, NudgeRL achieves comparable average performance to GRPO with 32 rollouts on mathematical reasoning, while using approximately 5× fewer tokens and 2.9× less estimated training compute. On code generation, it also outperforms GRPO with 64 rollouts. The paper includes ablations on context dropout, hint selection, advantage weighting, and policy correction.

Critical strengths
  1. A practical approach to improving exploration without increasing the rollout budget.

The central idea of assigning different strategy hints to individual rollouts is simple but useful. Rather than relying on stochastic sampling to discover alternative reasoning paths, NudgeRL explicitly encourages the policy to explore different approaches. I also appreciate that the method does not require complete reference solutions or oracle reasoning traces. The policy correction objective addresses an important practical issue: how to benefit from guided exploration during training without requiring external guidance at inference time. This makes the overall framework relatively easy to incorporate into existing RLVR pipelines.

  1. Encouraging empirical results across different models and tasks.

The evaluation covers two model families for mathematical reasoning and an additional code-generation setting. The improvement on LiveCodeBench is particularly encouraging, with pass@1 increasing from 0.317 for GRPO-64 to 0.335 for NudgeRL-8. The authors also account for the additional computation introduced by policy correction rather than comparing rollout counts alone. The ablations provide some evidence that both context dropout and policy correction matter, which strengthens the empirical case for the proposed framework.

Critical weaknesses
  1. The current experiments do not sufficiently isolate the contribution of strategy-guided exploration.

NudgeRL combines three components: externally generated strategy hints, modified advantage estimation, and policy correction. Although the paper includes ablations on individual hyperparameters, it is still difficult to determine how much of the improvement comes from exploring different strategies, as opposed to providing additional problem-specific knowledge through the hints.

In particular, the paper does not report a clean baseline that uses the same context-conditioned rollouts but standard GRPO advantage estimation and no policy correction. This is important because setting λ = 1 in Eq. (8) recovers the standard group-relative advantage, making it possible to isolate the effect of the other components. Similarly, a control using unrelated or randomly reassigned hints would help determine whether the gains require meaningful strategy guidance or simply arise from perturbing the prompt distribution.

This distinction matters for the paper's central claim. The experiments show that the complete framework works, but they do not yet convincingly establish that structured exploration, rather than auxiliary knowledge from the external model or the additional optimization objective, is the primary source of improvement.

  1. The evidence for the claimed efficiency advantage is not yet sufficiently robust.

The mathematical reasoning results are much closer than the abstract might suggest. On Qwen3-4B, NudgeRL achieves an average pass@1 of 0.489 compared with 0.487 for GRPO-32. On Olmo3-7B, the corresponding numbers are 0.285 and 0.281. Moreover, NudgeRL does not consistently outperform GRPO-32 on individual benchmarks. These results are compatible with the claim that NudgeRL reaches similar performance using less compute, but the absence of multiple training seeds or uncertainty estimates makes it difficult to assess the reliability of such small differences.

I am also concerned about the compute comparison. According to Table 3, NudgeRL-8 requires 0.492 EFLOPs, compared with 0.342 EFLOPs for GRPO-8 and 0.701 EFLOPs for GRPO-16. The reported efficiency advantage is primarily established by comparing against larger-rollout GRPO configurations trained for the same number of steps. It remains unclear how NudgeRL would compare against a GRPO baseline given the same total training compute, potentially with more optimization steps and a smaller rollout group.

Since compute efficiency is one of the main contributions, a compute-matched comparison and some evidence of run-to-run stability would substantially strengthen the paper. Comparisons with stronger exploration-oriented RLVR baselines, beyond GRPO rollout scaling and clipping adjustments, would also make the empirical positioning more convincing.

Questions for authors
  1. Can the authors provide a more direct component-wise comparison to isolate the benefit of Strategy Nudging?

In particular, I would be interested in the performance of context-conditioned sampling with standard GRPO advantages (λ = 1) and no policy correction (λ_PC = 0), as well as the performance when the proposed learning objective is used without meaningful strategy hints. The current ablations vary λ and λ_PC separately but do not report this full separation. It would also be useful to evaluate randomly reassigned or unrelated hints. These controls would help establish whether the benefit comes from strategy diversity itself, rather than additional information or generic prompt perturbations.

  1. How stable are the results across training seeds, and how does NudgeRL compare with GRPO under a matched compute budget?

Table 4 reports a single random seed, while the average performance differences over the strongest GRPO configurations are only 0.002 and 0.004 for the two mathematical reasoning models. Could the authors provide results across multiple seeds, ideally with uncertainty estimates? In addition, a GRPO configuration trained with approximately the same total FLOPs as NudgeRL would make the efficiency comparison more informative. It would be helpful to clarify whether the number of distinct training problems and the effective optimization batch size are comparable across rollout configurations.

  1. Could the authors clarify how negative advantages are handled in the policy correction objective?

Eq. (10) uses a signed advantage-weighted log-likelihood objective. As written, trajectories with negative advantages also contribute to the correction loss and explicitly decrease their likelihood under the original prompt. This differs somewhat from the description of policy correction as distilling useful or successful trajectories. Are negative advantages retained in the actual implementation, or are the weights clipped or otherwise transformed? Clarifying this would help explain the optimization mechanism and the relationship to the advantage-weighted distillation method discussed in Appendix C.

  1. Could the authors provide more details on how strategy hints are constructed and evaluated for code generation?

The appendix gives the hint-generation and prompting templates for mathematical reasoning, but the corresponding procedure for LiveCodeBench is less clear. Are the code-generation hints also produced by gpt-4o-mini using the same strategy-level formulation? How is correctness verified during training, and how sensitive is the result to hint quality? Given that code generation provides the clearest improvement over the strongest GRPO baseline, these details are important for assessing the generality and reproducibility of the method.

Reviewer_7P4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NudgeRL, a strategy-guided exploration framework for RLVR, aiming to address the limited exploration diversity of conventional methods such as GRPO. Rather than increasing the number of sampled rollouts, NudgeRL introduces lightweight, problem-specific strategy hints generated by an external LLM to encourage the policy to explore different reasoning approaches. The framework consists of three main components: (1) Strategy Nudging, which conditions rollouts on different strategy hints to improve exploration diversity; (2) Inter-Intra Group Advantage, which incorporates both within-context and across-context reward information for credit assignment; and (3) Policy Correction, which transfers useful behaviors discovered through context-conditioned exploration back to the original policy, allowing inference without external hints. Experiments across mathematical reasoning and code generation benchmarks show that NudgeRL achieves comparable or better performance than GRPO with substantially larger rollout budgets. The authors further report improvements in token efficiency and estimated training compute.

Critical strengths
  1. Well-motivated problem and an intuitive exploration framework. The paper addresses an important challenge in RLVR, i.e., conventional rollout sampling can repeatedly explore dominant reasoning patterns while failing to discover alternative, potentially successful strategies.

  2. Reasonably clear methodology and useful supporting analyses. The paper provides a clear description of the three components and their roles in the overall framework. The analyses of exploration diversity, context dropout, hint sampling strategies, and training efficiency provide useful insights into the method.

Critical weaknesses

A major concern is that the proposed approach relies on an external LLM (GPT-4o-mini) to generate problem-specific strategy hints. Although the authors emphasize that these hints do not contain complete solutions, identifying an appropriate theorem, mathematical concept, or reasoning technique for a particular problem can itself require substantial problem-specific knowledge.

Besides, it remains unclear how much each component in NudgeRL contributes independently to the final performance improvement and whether their combination is necessary. I acknowledge that the paper provides several relevant ablations, including the effects of context dropout, the inter-intra weighting coefficient (λ\lambda), and the policy correction coefficient (λPC\lambda_{\mathrm{PC}}). Nevertheless, these experiments primarily vary individual hyperparameters within the proposed framework rather than systematically isolating the contributions of the three mechanisms.

Additionally, I have concerns about whether the comparisons fully reflect the computational resources required by the proposed method:

  • NudgeRL relies on additional inference from an external LLM to generate strategy hints. While the authors include the offline hint-generation tokens in their token-efficiency analysis, the FLOPs estimates in Table 3 are based on the training-policy computation and do not fully account for the computational cost of the external model. Moreover, comparing raw token counts across different models does not necessarily reflect their actual computational costs.
  • The comparison is primarily conducted across configurations with different rollout budgets rather than strictly matched total compute. According to Table 3, NudgeRL with 8 rollouts requires 0.492 EFLOPs, compared with 0.342 EFLOPs for GRPO with 8 rollouts, representing approximately 44% additional computation. Therefore, it would be important to compare against GRPO trained for additional steps under the same total compute budget, rather than mainly comparing NudgeRL-8 against GRPO-32.
Questions for authors

Please see the weakness part above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewers,

Thank you very much for your thoughtful comments and valuable feedback. Your suggestions have been very helpful in improving our work.

As the discussion period draws to a close, we would like to follow up on our responses and additional clarifications. We have provided further explanations to address the concerns raised during the discussion.

We would greatly appreciate it if you could review our follow-up responses and let us know whether your remaining concerns have been adequately addressed. We hope that our explanations have sufficiently addressed your concerns, and we welcome any further feedback or suggestions.

Thank you again for your time and consideration.

Best regards

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.