- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the learning-cliff problem in reinforcement learning with verifiable rewards (RLVR), where difficult problems can produce uniformly unsuccessful rollouts and provide little learning signal under GRPO. Privileged guidance, such as reference-solution prefixes, helps generate successful trajectories but also creates a mismatch between the guided rollout distribution and the unguided objective used at inference time. This paper formulates this mismatch as the off-context problem and propose OC-GRPO, which uses importance-based correction to learn from guided rollouts while targeting the original unguided objective. The paper presents theoretical analyses of objective alignment and credit assignment, and evaluates the method on mathematical reasoning benchmarks using Qwen2.5 models at three scales. OC-GRPO-Fixed achieves improvements over vanilla GRPO and several guided-training baselines, with positive results also reported for smaller models.
Critical strengths
-
The paper identifies a relevant mismatch between guided exploration and unguided inference. Explicitly formulating this issue provides a useful perspective on training with privileged information.
-
The importance-based correction is conceptually straightforward and can be integrated into existing GRPO training without introducing substantial additional algorithmic complexity.
-
The comparisons with guided-training baselines are relevant, and OC-GRPO-Fixed achieves encouraging results across the reported mathematical reasoning benchmarks. Its performance at smaller model scales provides further evidence of practical potential.
Critical weaknesses
-
The paper identifies a meaningful mismatch between guided rollout generation and unguided optimization. However, the main modification in Eq. (8) follows an established importance-sampling principle by using the guided behavior distribution in the denominator. The contribution therefore appears to lie primarily in identifying this mismatch and applying an established correction to guided GRPO. This is a meaningful application, but its incremental methodological significance would be more convincing with stronger evidence that the specific correction provides benefits beyond existing guided-training approaches.
-
The experiments do not fully isolate the benefit of the proposed correction. In Table 2, the results are encouraging, but the larger gain over vanilla GRPO reflects both privileged guidance and the proposed correction. The comparison with guided baselines is more informative about the incremental contribution.
Questions for authors
-
What evidence distinguishes the benefit of the proposed off-context probability ratio from more general advantage weighting or gradient stabilization? This seems important for assessing the incremental algorithmic contribution.
-
Could the authors clarify the dataset construction and explain which comparisons support correction-specific gains under matched budgets?
4 comments
Dear Reviewer 1kSQ,
The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?
Thank you for serving as a reviewer for ICLR.
Best,
AC
Thanks for the response. I have decided to maintain my original score.
Thank you for your response.
We hope that the results in our submission and clarifications in our rebuttal addressed all your standing concerns. We are happy to clarify anything further
thank you for your review
Thank you for the review. All numbers are from the submitted checkpoints. We have updated the PDF with the new analyses where appropriate (additions in blue: App. K.2 and G.1).
W1. Significance. We agree that the weight is standard importance sampling. The contribution is to show that guided RLVR methods optimize a different objective from the one used at test time, and that targeting it (OC-GRPO-Fixed) gives gains that are significant at every model scale. Average Pass@1 over AIME, Gaokao2023 and OmniMath, 3 seeds (Tables 2 and 5):
Differences with 95% paired-bootstrap CIs:
How the intervals are computed: for each test problem we take the difference in Pass@1 between the two methods, averaged over the 3 seeds. We resample the problems with replacement 10,000 times (within each benchmark), recompute the Average difference, and report the 2.5th to 97.5th percentiles. An interval above zero means the gain is robust to resampling the test problems.
W2. Separating guidance from the correction. We agree that OC-GRPO-Fixed vs vanilla GRPO mixes the two; the informative comparison is with the guided baselines. There are two simpler ways to learn from guided rollouts, and both fall short:
OC-GRPO instead targets the original objective J(θ)=Ey∼πθ(⋅∣x)[r(x,y)] and reweights guided rollouts by the off-context ratio. The trajectory-level estimator is unbiased for ∇θJ(θ) (App. E.1); we implement a token-level version (App. E.3). POPE* and OC-GRPO-Fixed share guided data, steps and clipping and differ only in the objective, so the POPE* row of the second table isolates the effect of targeting J.
Q1. Off-context ratio vs advantage weighting or gradient stabilization.
Q2. Dataset construction and matched budgets.
Correction-specific gains: OC-GRPO-Fixed vs POPE* and PrefixRL* is matched by construction (3 seeds, W1). The correction adds one no-grad forward pass (5.5–5.7% of step time at 7B) and nothing at inference.
Against vanilla GRPO at matched compute (7B, seed 1, Average Pass@1 at intermediate checkpoints):
OC-GRPO-Fixed is ahead of both at every budget; against vanilla GRPO by +2.00 [+1.47, +2.54] at 0.5× and +1.85 [+1.31, +2.41] at 1.0× (paired bootstrap).