- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Bilateral Context Conditioning (BICC) evaluates correct responses using incorrect responses as context, and vice versa, within GRPO. Reward-Confidence Correction (RCC) adjusts the advantage baseline using the covariance between reward and confidence. The paper reports consistent accuracy gains and lower gradient variance.
Critical strengths
S1. BICC uses other responses within a sampled group in a simple way, with no added inference cost.
S2. The reported gains span several optimizers, model families, and model sizes. Experiments with another training corpus and transfer to logic and code broaden the evaluation.
S3. I found the same-partition context control and larger-rollout baseline useful. The reported training-time and memory overhead also help assess the method's cost.
Critical weaknesses
W1. The equivalence to GRPO claimed in Section 4.1 and Appendix B.2.4 does not hold. Even when clipping is inactive, the sequence-ratio objective differs from token-level GRPO: two token ratios of 1.05 have an average of 1.05 and a product of 1.1025, both within the clipping interval. The partition-based rearrangement is valid, but replacing token averages with sequence products changes the objective.
W2. Appendix B.4 claims uniformly bounded per-sample gradient contributions, but clipping does not establish this bound. With a negative advantage and a conditioned ratio above 1+ε, the surrogate remains Aρ. Its gradient contains Aρ∇θlogπθ, which clipping does not bound. Bounding the conditioning weight alone does not resolve the problem.
W3. The variance argument for RCC relies on an untested approximation. Equation (36) includes squared score gradients in the optimal baseline; Appendix B.3 drops that factor under an independence approximation, without checking the assumption. The derivation is also for an unconditioned estimator, whereas BICC uses a conditioned, clipped objective. There is a separate experimental ambiguity: RCC removes advantage normalization, but Table 5 has no centered, unnormalized control without the covariance correction. The results therefore do not separate the correction's effect from gradient rescaling.
W4. I cannot reconcile the reported standard deviations with the stated evaluation procedure. Appendix C.2.2 describes 32 samples per problem and standard deviations over 32 evaluation seeds. If each seed produces one answer per problem, AMC accuracy changes in increments of 2.5 points. A mean of 74.2% over 32 seeds would then require a standard deviation of roughly 1.2 percentage points, compared with the 0.49 reported in Table 9. There may be additional averaging, but the protocol does not describe it. Variation across evaluation seeds also does not establish robustness across independent training runs.
W5. Tables 1 and 7 define Dr.GRPO using a frozen initial policy in the ratio and a KL penalty. This differs from the cited method, which removes response-length and reward-standard-deviation normalization while retaining old-policy ratios (see the original Dr.GRPO paper, Section 3.2). The discrepancy weakens the evidence for compatibility across established variants.
Questions for authors
Q1. Does the implementation use token-level or sequence-level ratios? Please correct the reformulation to match the objective used in the code.
Q2. What assumptions establish the gradient bound, especially for negative advantages with large ratios?
Q3. How does RCC compare with centered, unnormalized advantages without the covariance correction? Please also compare variance using common rollouts at the same checkpoint.
Q4. Are the 32 samples generated overall or separately for each evaluation seed? Please provide per-seed accuracies, the standard-deviation calculation, and the number of independent training runs.
Q5. What exact Dr.GRPO loss was used, and why does it differ from the cited implementation?
2 comments
Thank you for your detailed review and feedback
We sincerely appreciate your careful review and are glad that our explanations have addressed your concerns. Your suggestions have clearly strengthened the paper, and we will incorporate the corresponding results and clarifications into the final version.
Thank you for your response. The explanations cleared most of my questions, especially about the sources of performance gains and the robustness of the method. Therefore, I have raised my score to 3.