acceptodds
Under review as a conference paper at ICLR 2027

When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO

Abstract

Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models. Although GRPO computes advantages from the group mean, it treats each output as an independent sample during optimization and leaves unused the contrast between correct and incorrect solutions within the same group, where successful reasoning traces could be compared explicitly against failed ones. To exploit this contrast, we present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples. Building on this insight, we propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces during the optimization, enabling a direct information flow across samples. We further introduce Reward-Confidence Correction (RCC) to stabilize training by dynamically adjusting the advantage baseline in GRPO using reward-confidence covariance derived from the first-order approximation of the variance-minimizing estimator. Both mechanisms operate on the sampled group alone and apply to GRPO and its variants. Experiments with Qwen3-4B and Phi-4-mini on Math500, AMC 2023, and AIME 2024/2025 show that BICCimproves GRPO and five of its variants in 54 of 56 comparisons, by 0.3 to 1.9 points with larger gains on the weaker base model, while RCC brings gradient variance 31 to 37% below GRPO. The gains persist at 14B scale and on Llama-3.1-8B, and training on mathematics transfers zero-shot to logic and code. The implementation is available at [BiCC Codebase](https://anonymous.4open.science/r/BiCC-77C9)

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_x7FuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Bilateral Context Conditioning (BICC) evaluates correct responses using incorrect responses as context, and vice versa, within GRPO. Reward-Confidence Correction (RCC) adjusts the advantage baseline using the covariance between reward and confidence. The paper reports consistent accuracy gains and lower gradient variance.

Critical strengths

S1. BICC uses other responses within a sampled group in a simple way, with no added inference cost.

S2. The reported gains span several optimizers, model families, and model sizes. Experiments with another training corpus and transfer to logic and code broaden the evaluation.

S3. I found the same-partition context control and larger-rollout baseline useful. The reported training-time and memory overhead also help assess the method's cost.

Critical weaknesses

W1. The equivalence to GRPO claimed in Section 4.1 and Appendix B.2.4 does not hold. Even when clipping is inactive, the sequence-ratio objective differs from token-level GRPO: two token ratios of 1.05 have an average of 1.05 and a product of 1.1025, both within the clipping interval. The partition-based rearrangement is valid, but replacing token averages with sequence products changes the objective.

W2. Appendix B.4 claims uniformly bounded per-sample gradient contributions, but clipping does not establish this bound. With a negative advantage and a conditioned ratio above 1+ε, the surrogate remains Aρ. Its gradient contains Aρ∇θlogπθ, which clipping does not bound. Bounding the conditioning weight alone does not resolve the problem.

W3. The variance argument for RCC relies on an untested approximation. Equation (36) includes squared score gradients in the optimal baseline; Appendix B.3 drops that factor under an independence approximation, without checking the assumption. The derivation is also for an unconditioned estimator, whereas BICC uses a conditioned, clipped objective. There is a separate experimental ambiguity: RCC removes advantage normalization, but Table 5 has no centered, unnormalized control without the covariance correction. The results therefore do not separate the correction's effect from gradient rescaling.

W4. I cannot reconcile the reported standard deviations with the stated evaluation procedure. Appendix C.2.2 describes 32 samples per problem and standard deviations over 32 evaluation seeds. If each seed produces one answer per problem, AMC accuracy changes in increments of 2.5 points. A mean of 74.2% over 32 seeds would then require a standard deviation of roughly 1.2 percentage points, compared with the 0.49 reported in Table 9. There may be additional averaging, but the protocol does not describe it. Variation across evaluation seeds also does not establish robustness across independent training runs.

W5. Tables 1 and 7 define Dr.GRPO using a frozen initial policy in the ratio and a KL penalty. This differs from the cited method, which removes response-length and reward-standard-deviation normalization while retaining old-policy ratios (see the original Dr.GRPO paper, Section 3.2). The discrepancy weakens the evidence for compatibility across established variants.

Questions for authors

Q1. Does the implementation use token-level or sequence-level ratios? Please correct the reformulation to match the objective used in the code.

Q2. What assumptions establish the gradient bound, especially for negative advantages with large ratios?

Q3. How does RCC compare with centered, unnormalized advantages without the covariance correction? Please also compare variance using common rollouts at the same checkpoint.

Q4. Are the 32 samples generated overall or separately for each evaluation seed? Please provide per-seed accuracies, the standard-deviation calculation, and the number of independent training runs.

Q5. What exact Dr.GRPO loss was used, and why does it differ from the cited implementation?

2 comments

Thank you for your response. The explanations cleared most of my questions, especially about the sources of performance gains and the robustness of the method. Therefore, I have raised my score to 3.

AuthorsOpenReview ↗

Thank you for your detailed review and feedback

We sincerely appreciate your careful review and are glad that our explanations have addressed your concerns. Your suggestions have clearly strengthened the paper, and we will incorporate the corresponding results and clarifications into the final version.

Reviewer_XtAvOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Bilateral Context Conditioning for GRPO. During training, each sampled response is evaluated with responses from the opposite reward group as additional context. It also introduces Reward-Confidence Correction, which adjusts the baseline using the covariance between rewards and log-probability changes. The method is motivated by a contrastive reformulation of GRPO. Experiments report gains across several GRPO variants and model families, with additional transfer tests on logic and code.

Critical strengths

Using opposite-outcome responses as training context is a concrete way to reuse information already present in the sampled group. It adds no inference-time steps. The evaluation covers several optimization variants and model families. The same-class conditioning controls and enlarged-rollout baseline are useful attempts to distinguish the proposed mechanism from extra context and training cost.

Critical weaknesses

Appendix B.4 treats one-sided clipping as a bound on gradient magnitude, but the negative-advantage branch does not have the claimed bound. Section 4.1 also replaces a token-level average with a sequence-level product and calls them equivalent without clipping. These are substantive problems in the theoretical justification. The uncertainty estimates in Table 9 also need clarification before the reported significance can support the empirical conclusions.

Questions for authors
  1. How does Proposition 4 handle negative advantages? For A=−1A=-1, ϵ=0.2\epsilon=0.2, and ρ=10\rho=10, the clipped surrogate equals −10-10, even when the conditioning factor is w=1w=1. What additional assumptions are needed for the claimed bound on each sample's gradient contribution?

  2. Which objective is implemented: the token-level objective in Eq. (3), or the sequence-level surrogate in Section 4.1? A product of token ratios differs from their average even without clipping. Which parts of the contrastive reformulation remain exact for the implemented objective?

  3. Eq. (36) includes the squared score-gradient factor, which is dropped under an independence approximation. What supports this approximation for the trained policies? Does retaining this factor materially change the proposed baseline correction?

  4. What happens when σ\sigma-normalization is removed without adding the covariance correction? A comparison with RCC at matched gradient scale would help determine whether the variance reduction comes from the correction or simply smaller updates.

  5. How is the AMC result 74.2±0.4974.2 \pm 0.49 in Table 9 calculated? If each seed produces one answer per question, accuracy on 40 questions changes in steps of 2.52.5 percentage points. This cannot produce the reported mean and standard deviation over 32 seeds. Is 0.490.49 a standard error, or does each seed include additional averaging?

1 comment

Thank you for the thorough response and revisions. My concerns have been largely addressed, and I have raised my rating to 3.

Reviewer_zAZ4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose Bilateral Context Conditioning (BiCC), a method that conditions the numerator of each sample’s importance ratio in GRPO on samples from the opposite reward partition. For a correct output, the ratio becomes πθ(oi∣q,O−)/πθold(oi∣q)\pi\theta(o_i \mid q, \mathcal{O}^-)/\pi{\theta_{old}}(o_i \mid q). For an incorrect output, the context holds the correct outputs instead. The denominator stays on the bare query, and inference uses only qq. The motivation is that GRPO scores each output in isolation. Under binary rewards, its objective can be rewritten as a margin between the average clipped ratios of correct and incorrect outputs (Eq. 9 to 11). The authors also propose Reward-Confidence Correction (RCC). RCC subtracts 2Cov^(R,δ)2\widehat{\mathrm{Cov}}(R,\delta) from the centered reward and drops the σ\sigma normalization. It comes from a first-order expansion of the variance-minimizing baseline under importance sampling (Eq. 14).

The experiments train Qwen3-4B-Instruct-2507 and Phi-4-mini-instruct on DAPO-Math-17k. They report pass@1 averaged over 32 samples on Math500, AMC23, AIME24 and AIME25. BiCC is applied to GRPO and five variants, and it improves 54 of 56 comparisons by 0.3 to 1.9 points (Table 2). Adding RCC gives another 0.4 points on Qwen and 0.7 on Phi on Math500 (Table 13). The authors also report 31 to 37% lower gradient variance than GRPO. The appendix adds Qwen3-14B, Llama-3.1-8B, zero-shot logic and code, and a G = 9 compute-matched baseline.

Critical strengths
  1. The idea is simple and cheap, and the authors test it widely. GRPO already samples correct and incorrect attempts for the same question, and BiCC reuses them as context during training without changing inference. It works with GRPO, Dr.GRPO, ASPO, GMPO, DAPO and GSPO through a one-line change to the ratio (Table 1). The training step is only 5% slower (Table 11). This matters to my assessment because, if the effect is real, many people working on GRPO could use it right away.
  2. The same-class control is a good experiment. At the same context length, filling the context with same-partition samples gives 91.5 on Qwen and 76.4 on Phi. That is close to GRPO at 91.4 and 76.2, while cross-partition context gives 92.2 and 78.1 (Table 3b). The same ordering holds on AIME24 (Table 20). This is the strongest evidence in the paper that the gain does not come from a longer context alone.
Critical weaknesses
  1. BiCC trains a different distribution from the one used at test time, and the paper does not rule out a simpler explanation. The gradient of the BiCC objective is ∇θlog⁡πθ(o∣q,O∓)\nabla_\theta \log \pi_\theta(o \mid q, \mathcal{O}^\mp). It updates the policy conditioned on other attempts, but at test time the model sees only qq. The LUPI framing does not cover this. In LUPI, the privileged information shapes training but is never an input to the learned function. The paper does not say how this surrogate relates to J(θ)J(\theta) under πθ(⋅∣q)\pi_\theta(\cdot \mid q). Section 5.1 itself calls ρ+\rho^+ a context-augmented surrogate and not a likelihood ratio. Any improvement on the bare query has to come through shared weights, and this is the part I most want explained.

The simpler explanation comes from Table 4. Since ρic=wiρi\rho^c_i = w_i \rho_i, BiCC multiplies the ratio of correct samples by w+w^+, which grows from 1.01 to 1.14 during training. It multiplies the ratio of incorrect samples by w−w^-, which falls from 0.99 to 0.84. The 99th percentile of ww spans 0.6 to 1.4 (Appendix B.4.5), which is outside the clip range of ϵ=0.2\epsilon = 0.2. So BiCC upweights positive samples, downweights negative ones, and changes which samples get clipped. Reweighting positive and negative samples can change RLVR results on its own (e.g., Zhu et al., 2025, The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning). I do not expect the authors to prove convergence for this surrogate. My point is that the paper needs a control that applies the same asymmetric scaling to plain GRPO without any context. It also needs the clip fraction for both methods. If BiCC still beats that control, the contrastive explanation becomes convincing.

  1. The contribution of RCC is not isolated, and its derivation does not match the implementation. RCC changes two things at once: it adds the covariance term, and it removes σ\sigma normalization, which is the Dr. GRPO change. No row removes σ\sigma without also adding the covariance term. The 0.4-point gain from RCC on Qwen (Table 13) is measured on Math500, which is close to saturated for Qwen3-4B: 89.2 before RL and 91.4 after GRPO (Table 2).

The definition of δ\delta is also unclear. Eq. 12 and Algorithm 1 use πθold\pi_{\theta_{old}}, but Figure 1 and the Figure 2 caption use πref\pi_{\mathrm{ref}}. If it is πθold\pi_{\theta_{old}}, then δ=0\delta = 0 at the point where Algorithm 1 computes it (line 7, before the update), and RCC does nothing. If it is πref\pi_{\mathrm{ref}}, the trust-region premise behind Eq. 13 does not hold. In that case the 0.27 to 0.56 gap in mean δ\delta between correct and incorrect samples (Figure 2) is also too large for a first-order expansion. The variance numbers are hard to interpret too. Eq. 48 is not scale-normalized, yet the variant that keeps σ\sigma has larger advantages and still shows a larger variance reduction, 51% vs. 34% (Table 5). I would like to see three things: the missing σ\sigma ablation, a clear definition of δ\delta, and a scale-free variance measure such as gradient SNR.

Questions for authors

Questions that could change my score:

  1. Can you run GRPO with A+A^+ and A−A^- scaled to match the w+w^+ and w−w^- statistics in Table 4, and compare it with BiCC-GRPO and GRPO on both models? This is the result that matters most for my score.
  2. Is δ\delta in RCC computed against πθold\pi_{\theta_{old}} or πref\pi_{\mathrm{ref}}? How many gradient updates do you run per rollout batch? If δ\delta uses πθold\pi_{\theta_{old}} and there is one update per batch, RCC is zero everywhere. In that case, please explain where the reported gain of 0.4 to 0.7 points comes from.
  3. What does BiCC-GRPO give with σ\sigma removed and no covariance term?
  4. Over training, how does pass@1 under π(⋅∣q)\pi(\cdot \mid q) compare with pass@1 under π(⋅∣q,O)\pi(\cdot \mid q, \mathcal{O})? This would show how the gain transfers to the bare query.

Clarifications:

  • Is wiw_i sequence-level or token-level? Rollouts average 385 tokens (Section 5.1). Early in training, I would expect 800 tokens of same-question solutions in the context to move sequence log-probabilities much more than w+=1.01±0.03w^+ = 1.01 \pm 0.03 suggests.
  • How are the opposite-partition samples formatted in the prompt? Are they labeled as correct or incorrect?
  • Training allows up to 2048 new tokens (Table 6), and 0.4 of 2048 tokens is reserved for context. How does the augmented sequence “stay within the length the baseline processes” (Section 5.1)? What is the truncation rate?
  • Section 2, Table 1 and Table 7 describe Dr.GRPO as using a frozen reference policy in the denominator. Dr.GRPO (Liu et al., 2025) removes the length and std normalization. What does the “BiCC Dr.GRPO” row implement?
  • The context roughly triples the sequence length, but the backward pass is only 7% slower and peak memory only 12% higher (Table 10). Could you explain these numbers?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading. All reviewers find BiCC simple and cheap, since it reuses the attempts that GRPO already samples without changing inference, and value the same-class control, the larger-rollout baseline, and the breadth of the evaluation (zAZ4, XtAv, x7Fu). Revised parts of the paper are highlighted in blue.

Contrastive form (XtAv, x7Fu). Reviewer XtAv questioned whether Sec. 4.1 replaces the token-level average with a sequence-level product, a point Reviewer x7Fu also raised. The contrastive form is exact for the token-level objective of Eq. (3) that we implement, with and without clipping, and Sec. 4.1 now writes the token mean in place of the compact notation (Lines 179-181 and 828-831).

Update bound (XtAv, x7Fu). Reviewer XtAv asked how Prop. 4 bounds negative advantages with large ratios, a point Reviewer x7Fu also raised. On this branch the conditioned term stays within a bounded factor of the identical GRPO term by Theorem 3, and the dual clip used in both methods caps it on only 0.004% to 0.007% of tokens, as the revised Prop. 4 states (Lines 924-935 and 979-981).

RCC (XtAv, x7Fu). Reviewer XtAv questioned the independence approximation in the RCC baseline and whether its variance reduction comes from smaller updates, as Reviewer x7Fu also did. Retaining the squared score factor yields the same correction, with a weighted covariance 1.08 to 1.12 times the plain one, and at matched scale the covariance term raises the gradient signal-to-noise ratio beyond the removal of std normalization alone (App. B.3 and F.3).

Evaluation protocol (XtAv, x7Fu). Reviewer XtAv found the deviations of Table 9 inconsistent with the protocol, as Reviewer x7Fu also did. Table 9 now reports the deviation across 32 seeds with a paired test, under which the Math500 gains are significant on both models, and three training seeds on Qwen3-4B confirm gains of 0.5 to 1.1 points on all four benchmarks (App. F.5).

Other clarifications (zAZ4, x7Fu). Applying the scaling of BiCC to GRPO without context gains at most 0.5 points against up to 1.9 for BiCC (App. F.1), BiCC improves the original Dr.GRPO by 0.6 to 1.8 points (App. F.4), and App. C.2.3 reports the re-measured cost at 1.27 times the GRPO step.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.