acceptodds
Under review as a conference paper at ICLR 2027

Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models. Yet, typical RLVR approaches fail on difficult problems: when a model cannot generate any correct solutions, it receives zero learning signal. Providing privileged guidance during training, such as solution prefixes, overcomes this learning cliff by steering the model towards rewarding generations. We observe that these guided generations are fundamentally off-context, i.e., there is a mismatch between the inference and training distributions. Our key contribution is to address this mismatch, ensuring the model ultimately learns to solve problems without guidance. To this end, we introduce Off-Context GRPO (OC-GRPO): a minimally modified variant of GRPO with an importance-corrected objective. OC-GRPO explicitly adjusts for context mismatch, cleanly separating guidance (where to explore) from the objective (what to optimize). By provably recovering an unbiased gradient estimator for the original unguided objective, OC-GRPO prevents instabilities during training. Empirically, our algorithm achieves a 3.9% absolute improvement (13.8% relative gain) over vanilla GRPO on average across standard mathematical reasoning benchmarks without adding any extra computational complexity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, optimization modeling, importance sampling, privileged information, GRPO, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1kSQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the learning-cliff problem in reinforcement learning with verifiable rewards (RLVR), where difficult problems can produce uniformly unsuccessful rollouts and provide little learning signal under GRPO. Privileged guidance, such as reference-solution prefixes, helps generate successful trajectories but also creates a mismatch between the guided rollout distribution and the unguided objective used at inference time. This paper formulates this mismatch as the off-context problem and propose OC-GRPO, which uses importance-based correction to learn from guided rollouts while targeting the original unguided objective. The paper presents theoretical analyses of objective alignment and credit assignment, and evaluates the method on mathematical reasoning benchmarks using Qwen2.5 models at three scales. OC-GRPO-Fixed achieves improvements over vanilla GRPO and several guided-training baselines, with positive results also reported for smaller models.

Critical strengths
  1. The paper identifies a relevant mismatch between guided exploration and unguided inference. Explicitly formulating this issue provides a useful perspective on training with privileged information.

  2. The importance-based correction is conceptually straightforward and can be integrated into existing GRPO training without introducing substantial additional algorithmic complexity.

  3. The comparisons with guided-training baselines are relevant, and OC-GRPO-Fixed achieves encouraging results across the reported mathematical reasoning benchmarks. Its performance at smaller model scales provides further evidence of practical potential.

Critical weaknesses
  1. The paper identifies a meaningful mismatch between guided rollout generation and unguided optimization. However, the main modification in Eq. (8) follows an established importance-sampling principle by using the guided behavior distribution in the denominator. The contribution therefore appears to lie primarily in identifying this mismatch and applying an established correction to guided GRPO. This is a meaningful application, but its incremental methodological significance would be more convincing with stronger evidence that the specific correction provides benefits beyond existing guided-training approaches.

  2. The experiments do not fully isolate the benefit of the proposed correction. In Table 2, the results are encouraging, but the larger gain over vanilla GRPO reflects both privileged guidance and the proposed correction. The comparison with guided baselines is more informative about the incremental contribution.

Questions for authors
  1. What evidence distinguishes the benefit of the proposed off-context probability ratio from more general advantage weighting or gradient stabilization? This seems important for assessing the incremental algorithmic contribution.

  2. Could the authors clarify the dataset construction and explain which comparisons support correction-specific gains under matched budgets?

4 comments
AuthorsOpenReview ↗

thank you for your review

Thank you for the review. All numbers are from the submitted checkpoints. We have updated the PDF with the new analyses where appropriate (additions in blue: App. K.2 and G.1).

W1. Significance. We agree that the weight is standard importance sampling. The contribution is to show that guided RLVR methods optimize a different objective from the one used at test time, and that targeting it (OC-GRPO-Fixed) gives gains that are significant at every model scale. Average Pass@1 over AIME, Gaokao2023 and OmniMath, 3 seeds (Tables 2 and 5):

Method7B3B1.5B
Vanilla GRPO27.323.917.7
PrefixRL*28.024.416.9
POPE*29.724.518.4
BREAD*28.523.015.9
OC-GRPO-Fixed31.125.819.7

Differences with 95% paired-bootstrap CIs:

OC-GRPO-Fixed minus7B3B1.5B
Vanilla GRPO+3.75 [+3.07, +4.43]+1.90 [+0.88, +2.93]+1.97 [+0.89, +3.07]
POPE* (strongest baseline at each scale)+1.34 [+0.94, +1.76]+1.35 [+0.37, +2.36]+1.27 [+0.25, +2.34]

How the intervals are computed: for each test problem we take the difference in Pass@1 between the two methods, averaged over the 3 seeds. We resample the problems with replacement 10,000 times (within each benchmark), recompute the Average difference, and report the 2.5th to 97.5th percentiles. An interval above zero means the gain is robust to resampling the test problems.

  • OC-GRPO-Fixed is significantly above vanilla GRPO and the strongest guided baseline at every scale. At 7B it is also significantly above PrefixRL* (+3.08 [+2.41, +3.76]) and BREAD* (+2.55 [+1.97, +3.17]).
  • No guided baseline is significantly above vanilla GRPO at 3B or 1.5B, and BREAD* is significantly below it at 1.5B (Table 5).

W2. Separating guidance from the correction. We agree that OC-GRPO-Fixed vs vanilla GRPO mixes the two; the informative comparison is with the guided baselines. There are two simpler ways to learn from guided rollouts, and both fall short:

  1. Optimize the guided objective (POPE*, BREAD*, PrefixRL*): Jguide(θ)=Ey∼πθ(⋅∣g(x))[r(x,y)]J^{\text{guide}}(\theta)=\mathbb E_{y\sim\pi_\theta(\cdot\mid g(x))}[r(x,y)]. This trains the policy that sees the hint and relies on transfer to the prompt without it. The two objectives can disagree: in the toy example of App. F they prefer opposite policies.
  2. Train the unguided policy on guided rollouts without a weight: the update is off-policy and biased, and training collapses (App. D, Fig. 3).

OC-GRPO instead targets the original objective J(θ)=Ey∼πθ(⋅∣x)[r(x,y)]J(\theta)=\mathbb E_{y\sim\pi_\theta(\cdot\mid x)}[r(x,y)] and reweights guided rollouts by the off-context ratio. The trajectory-level estimator is unbiased for ∇θJ(θ)\nabla_\theta J(\theta) (App. E.1); we implement a token-level version (App. E.3). POPE* and OC-GRPO-Fixed share guided data, steps and clipping and differ only in the objective, so the POPE* row of the second table isolates the effect of targeting JJ.

Q1. Off-context ratio vs advantage weighting or gradient stabilization.

  • It changes which policy is trained, not only the size of the update. The gradient goes through πθ(⋅∣x)\pi_\theta(\cdot\mid x), the policy used at test time, and the weight is the change of measure this requires. Reweighting advantages of πθ(⋅∣g(x))\pi_\theta(\cdot\mid g(x)) still optimizes JguideJ^{\text{guide}} (item 1 of W2); training πθ(⋅∣x)\pi_\theta(\cdot\mid x) without the weight collapses (item 2).
  • Stabilization is the same for all methods: gradient clipping at norm 1.0, ratio clipping at ϵ=0.2\epsilon=0.2. OC-GRPO adds only the weight's truncation to [0.01,1][0.01,1] (App. E.3).
  • Evidence: POPE* has the same data, steps and stabilization, and OC-GRPO-Fixed is significantly above it at every scale (W1).

Q2. Dataset construction and matched budgets.

  1. At 7B, 595 MATH training problems (Levels 3–5) are failed by the base model in 64 of 64 samples. A random 90/10 split gives 536 for training and 59 held out.
  2. Each training problem gets the shortest solution prefix (20% to 100%) with at least one success in 8 rollouts. This unlocks 424 of the 536.
  3. Training set: 536 original + 424 guided = 960 rows. POPE*, PrefixRL* and OC-GRPO-Fixed all train on it for 120 steps; vanilla GRPO, BREAD* and OC-GRPO-Adaptive train on the 536 for 64 steps.

Correction-specific gains: OC-GRPO-Fixed vs POPE* and PrefixRL* is matched by construction (3 seeds, W1). The correction adds one no-grad forward pass (5.5–5.7% of step time at 7B) and nothing at inference.

Against vanilla GRPO at matched compute (7B, seed 1, Average Pass@1 at intermediate checkpoints):

Compute (× vanilla's full run)Vanilla GRPOPOPE*OC-GRPO-Fixed
0.5×28.028.630.0
1.0×28.929.430.8
1.9× (full guided run)–29.631.7

OC-GRPO-Fixed is ahead of both at every budget; against vanilla GRPO by +2.00 [+1.47, +2.54] at 0.5× and +1.85 [+1.31, +2.41] at 1.0× (paired bootstrap).

Dear Reviewer 1kSQ,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Thanks for the response. I have decided to maintain my original score.

Thank you for your response.

We hope that the results in our submission and clarifications in our rebuttal addressed all your standing concerns. We are happy to clarify anything further

Reviewer_QcYJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OC-GRPO, which uses hints or solution prefixes to help LLMs generate successful rollouts on difficult math problems. It applies an IS correction to account for the difference between guided training prompts and the original prompts used at test time. Experiments on several math benchmarks show that OC-GRPO outperforms vanilla GRPO.

Critical strengths
  • This paper considers novel off-context correction, which is ignored by closely related work (prefixRL, POPE, BREAD). Also, this paper shows that training without off-context correction will lead to the explosion of grad-norm and log-prob-diff, and thus cause reward collapse. So, this paper has a really good position in the research direction and community of RLVR.
  • OC-GRPO is evaluated against several guided-training baselines under a unified training setup. The results show consistent gains over vanilla GRPO at smaller model scales, where none of the uncorrected guided baselines is significantly better than vanilla GRPO.
  • The paper provides a clear theoretical explanation of IS correction: it aligns guided rollouts with the unguided objective and reduces credit for successes that depend heavily on guidance.
Critical weaknesses
  • The authors didn't demonstrate the case study of rollout trajectories, so reader can't intuitively understand what is "privilege", and intuitively understand how "privilege" can facilitates models to rollout correct trajectories. Although the description of three types of privilege (light hint, successful rollout, etc.) are provided at page 6, It would be much more helpful to give specific example for each type.
  • The authors use MATH dataset. However, MATH is relatively simple. Employing more challenging data would align better with the principle of "targeting hard problem" of this paper, e.g., hard data in [1] (sampled from deepmath-103k).
  • Did the authors try to use the most advanced LLMs (e.g., GPT6-Astra) to sample solutions, and treat them as GT and to be viewed as the privilege in OC-GRPO, on training set without reference solution? If so, how much could it increase the performance?
  • Table 2 suggests that “Training” has the lowest score, and the considered eval sets are relatively simple. The authors are suggested to test performance on harder eval sets, for example HMMT2025, to show that the model trained by OC-GRPO has the generalization capability on harder benchmarks.

[1] Liu, Zihe, et al. "Part i: Tricks or traps? a deep dive into rl for llm reasoning." arXiv preprint arXiv:2508.08221 (2025).

Questions for authors

No.

4 comments
AuthorsOpenReview ↗

Thank you for your review

Thank you for the review. We have updated the PDF with the new analyses where appropriate (additions in blue).

W1. What "privilege" is, with an example for each case. Privileged information is any information available at training time but not at inference time. In our main experiments it is a prefix of the reference solution; Table 4 also uses hints written by the policy itself or by a frontier model, and feedback on a failed attempt. A real training prompt with a 20% prefix (LaTeX simplified):

Problem: A plane intersects a right circular cylinder of radius 1 forming an ellipse. If the major axis of the ellipse of [sic] 50% longer than the minor axis, the length of the major axis is (A) 1 (B) 3/2 (C) 2 (D) 9/4 (E) 3 Partial reference solution: We note that we can draw the minor axis to see that Here is a partial reference solution to the problem above. Complete the rest of the solution step by step and output the final answer within \boxed{}.

Without the prefix, the base model solves this problem in 0 of 64 attempts. With it, a rollout writes "the minor axis of the ellipse is the same as the diameter of the cylinder ... 2a=22a=2 ... 2b=32b=3" and answers 3, which is correct. On hard MATH problems, solution prefixes raise the expected verifier reward from 0.091 with no prefix to 0.790 with the full solution (Figure 2).

The correction gives each token the weight wt=πθold(yt∣x)/πθold(yt∣g(x))w_t=\pi_{\theta_{\text{old}}}(y_t\mid x)/\pi_{\theta_{\text{old}}}(y_t\mid g(x)), clipped to [0.01,1][0.01,1], where g(x)g(x) is the guided prompt. Tokens the model would write anyway keep full credit; tokens that only the hint makes likely are discounted. Three real rollouts, one per case of the example on page 6:

CaseMean weightTokens with the lowest weight
Light hint, correct (the prompt above)0.978the final answer
Light hint, wrong (another problem)0.975its opening words, and "add" (the hint's first step)
Full solution in the prompt, correct0.928"follow the [reference solution]", and a point EE that only the reference solution names

We will add the three rollouts in full, with per-token weights, to the appendix of the final version.

W2. MATH is relatively simple. We train only on problems the base model never solves: MATH problems (Levels 3–5) it fails in 64 of 64 attempts. "Hard" is relative to the model (Sec. 6.1). The method needs only a reference solution or a hint per problem; we have not run it on DeepMath-103k.

W3. Frontier-model solutions as privilege. We have not used frontier-model solutions as ground truth on problems without a reference solution. The closest experiment is Table 4 and App. I (OC-GRPO-Adaptive, 7B, Pass@1):

GuidanceAIMEGaokao2023OmniMath
None (vanilla GRPO)16.846.122.6
Hints written by Claude Sonnet 417.448.523.0
Hints written by the policy itself17.949.923.2

Frontier hints are above vanilla GRPO's mean on each benchmark, and below hints from the policy itself. Unlike the setting you propose, both hint writers see the reference solution at hint levels 2–5.

W4. The "Training" column; harder evaluation sets. The "Training" column of Table 2 is lowest by construction: it is the hard set itself, which the base model fails 64 of 64 times, evaluated without guidance. Harder sets, on the submitted checkpoints with the protocol of Table 2 (7B, Pass@1, 16 samples, 3 seeds); Δ\Delta = OC-GRPO-Fixed − vanilla GRPO, 95% paired-bootstrap CI:

BenchmarknBaseVanilla GRPOOC-GRPO-FixedΔ\Delta [95% CI]
OlympiadBench67428.128.731.1+2.32 [+1.52, +3.12]
OlymMATH2004.24.65.5+0.90 [−0.10, +2.01]
HMMT Feb 2025302.52.12.7+0.62 [−0.07, +1.39]
HMMT Feb 2026333.84.24.9+0.76 [−0.63, +2.40]
AIME 2026302.54.26.5+2.36 [+0.28, +5.28]
  • OC-GRPO-Fixed is the best of the three on every row.
  • Over four olympiad-level sets (the first three rows and IMO-AnswerBench), it is +1.09 [+0.67, +1.52] above vanilla GRPO. Against the base model on these sets, vanilla GRPO gains +0.10 [−0.34, +0.52] and OC-GRPO-Fixed +1.19 [+0.67, +1.74].
  • HMMT is near the floor for a 7B model (2–5%), so its intervals include zero; both years point the same way.

The table, with IMO-AnswerBench, Minerva and AMC23 added, is in the revised PDF (App. K.1).

Dear Reviewer QcYJ,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Thank you for the detailed responses and additional analyses. The concrete examples and harder evaluation benchmarks help clarify the method and strengthen the empirical evidence. However, the questions about harder training data and the setting without reference solutions remain open. On balance, I will maintain my original score.

AuthorsOpenReview ↗

Thank you for your response and sustained engagement

Thank you for the follow-up. Both open points concern one setting: hard problems without a reference solution, with guidance from a stronger model. We want to highlight the following points again:

  • For OC-GRPO No reference solution is needed. The guidance can be any text, and the correction uses only the policy's own probabilities under the prompts with and without guidance. A frontier-model solution simply replaces the reference solution.

  • Model-written guidance is already in the paper. In Table 4, OC-GRPO trained with hints written by Claude Sonnet 4, and with hints written by the policy itself, is above vanilla GRPO on every benchmark.

  • DeepMath-103k fits as is. Each problem has a verifiable final answer and DeepSeek-R1 solutions, but no human solution. The R1 solutions serve as guidance, and the same filter (i.e., the problems the base model fails in all 64 attempts) selects the hard training set.

Reviewer_srcqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the off-context problem in reinforcement learning with verifiable rewards (RLVR), in which privileged information is available during training to help language models solve challenging problems but is unavailable at inference time. The authors argue that existing guidance-based approaches may inadvertently optimize an objective that differs from the original unguided objective, potentially leading to a mismatch between training and inference.

To address this issue, the paper proposes Off-Context GRPO (OC-GRPO), which uses importance sampling to correct the distribution mismatch between guided rollouts and the unguided target policy. The authors provide theoretical analyses of the importance correction mechanism and its implications for credit assignment. They further introduce both fixed and adaptive guidance strategies to incorporate privileged information during training while accounting for its absence at inference time.

Critical strengths
  • Strengths 1: Principled motivation and practical applicability. The paper identifies a relevant distribution mismatch in guidance-based RLVR and provides a principled motivation for correcting it. The proposed method is relatively straightforward to integrate into existing GRPO training frameworks and introduces no additional inference-time overhead, making it potentially useful in practice.

  • Strengths 2: Broad empirical evaluation. The authors evaluate OC-GRPO against multiple baselines across three model sizes. In particular, the improvements of OC-GRPO-Fixed over vanilla GRPO at the 3B and 1.5B scales, supported by paired-bootstrap confidence intervals, provide valuable evidence for the effectiveness of the proposed approach. These results suggest that its benefits may extend beyond a single model configuration.

Critical weaknesses
  1. Gap between the theoretical analysis and practical implementation. The theoretical analysis relies on trajectory-level importance sampling, whereas the implementation applies token-level corrections with importance weights clipped to [0.01,1][0.01, 1]. These approximations may introduce bias, and it is unclear to what extent the theoretical results, particularly the credit-redistribution interpretation in Theorem 1, remain applicable under the practical implementation. The authors should better justify this gap and provide ablations to quantify the effects of token-level correction and weight clipping.

  2. Limitations in computational efficiency and empirical validation. A full OC-GRPO-Fixed run requires approximately 1.92×1.92\times the GPU-hours of vanilla GRPO. Although the paper reports compute-matched results, these results rely on a single random seed, limiting the ability to assess the robustness of the observed gains. Moreover, the improvements over the strongest guided baseline at the 7B scale appear relatively modest, and no statistical significance analysis is provided. Finally, the evaluation is restricted to mathematical reasoning tasks and a single model family, leaving the generalizability of the proposed method to other task domains and model families insufficiently established.

  3. Presentation and readability. The main text contains a large number of equations, which makes the key ideas and methodological contributions harder to follow. Some tables also have formatting and layout issues that reduce readability. The authors should streamline the theoretical presentation, move less essential derivations to the appendix, and improve the table formatting to make the paper more accessible.

Questions for authors
  1. Theoretical analysis versus practical implementation. Could the authors clarify the gap between the trajectory-level theoretical analysis and the token-level implementation, particularly the effects of clipping importance weights to [0.01,1][0.01, 1]? An ablation study that separately examines importance correction, weight clipping, and advantage downweighting would help clarify the contribution of each component.

  2. Data consistency and potential evaluation leakage. Section 6.2 reports 595 hard problems for the 7B model, whereas Appendix G.1 mentions 536. Could the authors clarify this discrepancy and explain how the corresponding datasets and splits were constructed? Additionally, did the authors check for potential overlap between the MATH training data and historical AIME evaluation problems?

  3. Presentation and readability. Could the authors improve the paper's readability by reducing the number of equations in the main text, moving less essential derivations to the appendix, and fixing the formatting issues in the tables?

2 comments
AuthorsOpenReview ↗

Thank you for your review

Thank you for the careful review. Nothing was retrained; all numbers come from the submitted checkpoints. We have updated the PDF with the new analyses where appropriate (additions in blue).

W1, Q1. Theory vs implementation. The theory (Lemma 1, Theorem 1, App. E.1) is trajectory-level; the implementation is its token-level version, derived by the same steps as GRPO's objective (full derivation in the revised PDF, App. E.3):

  1. Policy gradient (exact). ∇θJ(θ)=Ey∼πθ(⋅∣x)[A∑t∇θlog⁡πθ(yt∣x)]\nabla_\theta J(\theta)=\mathbb E_{y\sim\pi_\theta(\cdot\mid x)}\big[A\sum_t\nabla_\theta\log\pi_\theta(y_t\mid x)\big], with AA the advantage (conditioning on y<ty_{\lt t} omitted).

  2. Importance sampling (exact). GRPO samples from πθold(⋅∣x)\pi_{\theta_{\text{old}}}(\cdot\mid x) and weights each rollout by ∏sρs\prod_s\rho_s. OC-GRPO samples from πθold(⋅∣g(x))\pi_{\theta_{\text{old}}}(\cdot\mid g(x)) and weights it by ∏sρsws\prod_s\rho_s w_s, where

    ρs=πθ(ys∣x)πθold(ys∣x),ws=πθold(ys∣x)πθold(ys∣g(x)).\rho_s=\frac{\pi_\theta(y_s\mid x)}{\pi_{\theta_{\text{old}}}(y_s\mid x)},\qquad w_s=\frac{\pi_{\theta_{\text{old}}}(y_s\mid x)}{\pi_{\theta_{\text{old}}}(y_s\mid g(x))}.

  3. Token-level approximation (the only approximate step, as in GRPO). In the tt-th term the product is replaced by the factor of token tt: ∏sρs→ρt\prod_s\rho_s\to\rho_t in GRPO, ∏sρsws→ρtwt\prod_s\rho_s w_s\to\rho_t w_t in OC-GRPO:

    ∇θJ(θ)≈Ey∼πθold(⋅∣g(x))[∑tρtwtA∇θlog⁡πθ(yt∣x)].\nabla_\theta J(\theta)\approx\mathbb E_{y\sim\pi_{\theta_{\text{old}}}(\cdot\mid g(x))}\Big[\sum_t\rho_t w_t A\nabla_\theta\log\pi_\theta(y_t\mid x)\Big].

  4. Clipping, as in GRPO. ρt\rho_t to 1±0.21\pm0.2; wtw_t to [0.01,1][0.01,1].

Why step 3: token-level correction is what makes importance correction work in our setting. The trajectory-level weight is degenerate: its median on our guided rollouts is 6×10−606\times10^{-60}, and by Lemma 1 it averages p/q≈0p/q\approx0 on guided successes, erasing exactly the signal guidance provides. The token-level form keeps the sampled prefixes and corrects each next token to the unguided policy (off-policy actor-critic, Degris et al., 2012). We do not claim that at token level it is unbiased.

Truncation is limited. 6,784 guided rollouts per policy, start and end of training:

TokensShare
Weight below 1, untouched by either bound80–83%
Capped at 115–18% (median raw ratio 1.001; above 1.2 for 2% of all tokens)
At the floor 0.011–3%

The cap removes rare spikes: untruncated ratios reach 40–95 on single generic tokens the model writes more readily without the hint ("Let's start", "we need to calculate"), which would otherwise weigh as much as 40–95 ordinary tokens. Like ratio clipping in GRPO, it bounds these and leaves the rest unchanged.

Does Theorem 1 hold empirically? (Start of training; end-of-training seeds agree.)

Solution prefix shown20%40%60%80%100%
Per-token log-gap, nats0.070.140.190.260.23
Mean weight, correct0.9710.9580.9420.9300.924
Mean weight, failed0.9730.9570.9500.9330.947
  • Guided successes get less credit, more so with heavier guidance: yes (0.97 → 0.92).
  • Failures keep their penalty: tokens average 0.97 before truncation, 0.94–0.95 after (86–88% at ≥0.99). The amplification λ−>1\lambda_->1 (Theorem 1(i)) is trajectory-level and does not carry over to tokens; it moves to the appendix (W3).

Ablation (Q1): POPE* has the same data, steps and clipping but optimizes the guided objective; OC-GRPO-Fixed leads it by about 1.3 points at 7B, 3B and 1.5B, significant at each (3 seeds). This also answers W2.

Q2. 595 vs 536, and AIME overlap. 595 hard MATH problems (Levels 3–5; the base model fails 64 of 64 samples), split 90/10 at random: 536 for training (App. G.1), 59 held out. Sec. 6.2 quotes the total.

Overlap exists: 172 of the 993 AIME test problems are among the 595 (MATH contains past AIME). Without them, and on other hard sets (at most one shared problem each) (7B, Pass@1, 3 seeds):

SetnVanillaOC-GRPO-FixedΔ\Delta
AIME 1983–2024, overlap removed76121.724.3+2.6
AIME 2026304.26.5+2.4
OlympiadBench67428.731.1+2.3
OlymMATH2004.65.5+0.9

W2. Compute. The 1.92× cost comes from the guided data, shared by the other guided baselines (POPE*, PrefixRL*); the correction adds one forward pass (5.5–5.7% of step time). At matched compute OC-GRPO-Fixed is ahead of vanilla GRPO's full run (7B, Pass@1; intermediate checkpoints evaluated for seed 1):

Compute (× vanilla's full run)AIMEGaokao2023OmniMathAverage
Vanilla GRPO (1.0×)17.346.922.628.9
OC-GRPO-Fixed (0.5×)17.949.123.130.0
OC-GRPO-Fixed (1.0×)18.150.523.730.8

W3, Q3. Presentation. We agree. Theorem 1 and the set-up equations of Sec. 5 move to the appendix. In their place: the measured weights above and one worked example on real rollouts. Tables 2–4 will be reformatted.

Dear Reviewer srcq,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the reviewer-author discussion

We thank the reviewers for their time and constructive feedback, and summarize below the strengths they noted and how we addressed their concerns.

Strengths noted by the reviewers

  • A relevant and principled problem formulation (srcq, QcYJ, 1kSQ).
  • A novel correction, missed by closely related work (QcYJ).
  • A simple addition to GRPO with no inference cost (srcq, 1kSQ).
  • A clear theoretical account of credit assignment (QcYJ).
  • Evidence that uncorrected guided training collapses (QcYJ).
  • Significant gains at smaller model scales (srcq, QcYJ, 1kSQ).

Concerns and how we addressed them (additions in the revised PDF are marked in blue)

  • Theory vs token-level implementation (srcq, 1kSQ): we derived the token-level objective (App. E.3) and reported the measured importance weights.
  • Significance against guided baselines (srcq, 1kSQ): OC-GRPO-Fixed is significantly above the strongest guided baseline, POPE*, at all three scales (App. K.2).
  • Evaluation sets too easy (QcYJ): we added six harder benchmarks, including HMMT 2025 and 2026 (App. K.1).
  • AIME train/test overlap (srcq): the gains hold with the overlapping problems removed (App. K.3).
  • Compute cost and matched budgets (srcq, 1kSQ): we added per-benchmark matched-compute tables (App. G.1).
  • What privileged information looks like (QcYJ): we gave a worked example with per-token weights.
  • Settings without reference solutions (QcYJ): OC-GRPO does not need them; datasets such as DeepMath-103k, which come with model-generated solutions, fit the method as is.

Other limitations are beyond the scope of this paper and are noted in its Limitations section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.