- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Adaptive Contrastive Self-Play (A-CSP), which switches between GRPO and contrastive preference optimization based on the fraction of zero-variance reward groups. A-CSP constructs preference pairs from higher- and lower-reward responses to improve RL fine-tuning. Experiments on GSM8K and MATH-L1-3 show that A-CSP can improve final performance and reduce late-training degradation in some settings. However, the gains are inconsistent across models and tasks, and simpler switching schedules can achieve comparable or better results. The paper also includes ablations and analyses of the optimization and switching mechanisms.
Critical strengths
-
The paper addresses a relevant issue in RLVR. Zero-variance reward groups are a common challenge for group-based policy optimization. Investigating whether the optimization objective should change as training progresses is an interesting direction.
-
The method is relatively simple to implement. A-CSP uses reward statistics already available during GRPO training and does not require an additional reward model or separate verifier. The computational overhead of the switching controller itself is negligible.
-
The experimental analysis goes beyond a basic GRPO comparison. The paper includes experiments on KL masking, informative-group filtering, dynamic sampling, switching schedules, and CSP objective components. I particularly appreciate the comparison with shuffled switching schedules that approximately match the total number of CSP steps. These experiments help distinguish some possible explanations for the observed improvements.
Critical weaknesses
-
The motivation for the adaptive switching mechanism is not fully convincing. The paper argues that zero-variance reward groups provide no useful gradient for GRPO. However, CSP also cannot construct preference pairs from these groups. It is therefore unclear why a higher fraction of zero-variance groups should favor CSP over GRPO. More justification for this switching criterion is needed.
-
The benefit of adaptive switching is not clearly established. In Table 4, random switching outperforms A-CSP on Qwen2.5-1.5B GSM8K while using far fewer CSP steps (31 vs. 283). On Qwen2.5-3B GSM8K, A-CSP spends about 90% of training in CSP mode. These results raise the question of whether adaptive switching is actually necessary or whether a simple fixed schedule would achieve similar performance.
-
The experimental evaluation is somewhat limited. The experiments use only 500 training questions, 100 evaluation questions, and three random seeds. Moreover, the reported peak accuracy is selected from repeated evaluations on the same subset, without a separate final test evaluation. Some results also show very large variance across seeds. These limitations make it difficult to judge how reliable the improvements are.
-
The CSP objective needs further justification. CSP combines a pairwise preference loss, supervised loss, and entropy regularization, but the contribution of each component is not entirely clear. Also, preference pairs are selected using shaped rewards, meaning that both responses in a pair may be incorrect. It would be useful to understand how this affects training and whether the proposed CSP objective offers advantages over existing online preference optimization methods.
-
The improvements are inconsistent across models and tasks. A-CSP performs worse than GRPO on Qwen2.5-1.5B MATH-L1-3, while both methods collapse to zero final accuracy on Llama-3.2-3B MATH-L1-3. Since the switching thresholds are fixed and the zero-variance statistic depends on reward design, it is unclear how well the method generalizes to other settings.
Questions for authors
-
Since CSP also cannot learn from zero-variance groups, why should a high fraction of such groups favor switching from GRPO to CSP? Is there empirical evidence supporting this choice?
-
On Qwen2.5-3B GSM8K, A-CSP spends around 90% of training in CSP mode. How does it compare with a simple GRPO-to-CSP schedule with matched CSP exposure?
-
What fraction of the preference pairs constructed by CSP contain two incorrect responses? Have the authors examined whether these pairs help or hurt training?
-
Could the authors compare CSP with existing online preference optimization methods under the same training setup? This would help clarify the contribution of the proposed objective.
-
Can the authors provide evaluations on larger standard benchmarks or held-out test sets? Given the small evaluation subsets and relatively large variance across seeds, it is difficult to assess the reliability of some reported improvements.
3 comments
...(Continued from the previous post, post 2/3)
W3 / Q5: Evaluation reliability and larger test sets
We have now evaluated final checkpoints on the complete GSM8K test set of 1,319 questions. We also separately evaluated 1,044 questions not used in the original trajectory evaluations.
The expanded evaluation provides additional positive evidence:
| Model and setting | A-CSP minus GRPO, Full Test EM | Reported interval |
|---|---|---|
| Qwen2.5-3B, stabilized GRPO, KL 0.04 | +0.0066 | [-0.0048, +0.0177] |
| Qwen2.5-3B, stabilized GRPO, KL 0.10 | +0.0091 | [-0.0096, +0.0258] |
| Llama-3.2-3B GSM8K, stabilized GRPO, KL 0.04 | +0.0392 | [+0.0263, +0.0528] |
The Qwen results remain positive on average, although the intervals include zero. On the 1,044 example subset, Qwen2.5-3B at KL 0.04 improves by 0.0134, with a reported interval of [+0.0013, +0.0255].
More importantly, A-CSP retains a 3.92 percentage point advantage over stabilized GRPO on Llama-3.2-3B GSM8K, with positive differences across all three seeds. This shows that the benefit is not limited to comparisons against GRPO runs that collapse completely. These additional evaluations substantially expand the original 100 example protocol.
W4 / Q3 / Q4: CSP objective, incorrect pairs, and comparison with preference optimization
CSP combines three complementary components. The pairwise term learns from reward ordering, the supervised term reinforces the preferred completion, and entropy regularization affects the policy distribution. Our existing ablations show that these terms affect peak performance and retention differently. For example, the full A-CSP objective achieves retention of 0.97, compared with 0.91 for the pairwise term alone and 0.95 for pairwise learning with entropy.
We also measured the correctness of CSP preference pairs using additional checkpoint rollouts:
| Dataset | Pairs with two incorrect responses | Interpretation |
|---|---|---|
| GSM8K | None observed across 4,500 diagnostic groups | Incorrect answers usually share the same shaped reward |
| MATH-L1-3, noncollapsed checkpoints | 21% to 35% | Shaped rewards can rank two incorrect answers |
| MATH-L1-3, collapsed Llama checkpoint | 84% | Most eligible pairs contain two incorrect answers |
On MATH-L1-3, the higher reward incorrect answer is generally closer to the reference answer under the implemented edit distance reward. Such pairs can provide relative training information even when neither answer is fully correct.
To address the comparison with existing preference optimization methods, we also evaluated online DPO with a fixed reference policy.
| Comparison | Full GSM8K Final EM difference | Reported interval |
|---|---|---|
| A-CSP minus online DPO | +0.0144 | [+0.0025, +0.0260] |
| A-CSP minus online DPO, 1,044 example subset | +0.0204 | [+0.0083, +0.0322] |
A-CSP outperforms this online DPO baseline in all three seeds. This provides additional evidence that the proposed training strategy is competitive with an established form of preference optimization.
...(Continued in the next post, post 2/3)
...(Continued from the previous post, post 3/3)
W5: Why are the improvements inconsistent across models and tasks?
We agree that the benefits depend on the model and task. This is an important part of our research question.
Our goal is to study settings where the model still has substantial room to improve, but RL training can also undergo severe regression. We do not expect A-CSP to provide the same benefit when a model already solves nearly every problem or when neither objective receives useful reward information.
The experiments show both sides of this tradeoff. A-CSP improves late training behavior in several GSM8K settings, including the new comparison against stabilized Llama GRPO. It also fails to improve some MATH-L1-3 settings. These differences are why we study objective switching rather than proposing CSP as a universal replacement for GRPO.
The central contribution is a practical framework for choosing between optimization objectives during RLVR training, supported by experiments that identify both its benefits and its limits. We believe this is a useful direction because a fixed optimization objective need not be the best choice throughout training, even when no single switching rule works best in every setting.
Another practical advantage is that A-CSP allows users to adjust the balance between performance and stability through its switching thresholds. A higher entry threshold favors more aggressive GRPO training by requiring stronger evidence before switching to CSP, while a lower threshold favors earlier use of the more conservative CSP objective. This flexibility allows the same framework to accommodate different training preferences without changing the optimization objectives. We do not claim that one threshold is optimal across all settings.
Thank you again for the constructive feedback. The additional experiments directly address your concerns about switching schedules, evaluation reliability, incorrect preference pairs, and comparisons with existing preference optimization methods. We believe the main value of A-CSP is not that CSP solves zero variance or that one switching rule is always optimal. Rather, it provides a simple, low overhead framework for balancing two optimization objectives as training conditions change. The new full test results and stronger baselines provide further evidence of its practical value, while our controls make clear where simpler alternatives are competitive. We hope these results clarify the motivation and strengthen the contribution of the paper.
Thank you for the thoughtful review and for recognizing the value of our method and experimental controls. Your questions helped us clarify the motivation for A-CSP and conduct additional experiments. We address each concern below.
TL;DR: Our motivation is to balance GRPO's performance potential with the risk of severe training regression, not to recover supervision from zero variance groups. A-CSP provides an inexpensive way to switch between GRPO and CSP based on observed reward statistics, with adjustable thresholds to favor more aggressive or conservative training (W1/Q1, W5). New fixed switch experiments show comparable performance, confirming that objective switching can be effective even with simple schedules, while A-CSP avoids specifying the switching point in advance (W2/Q2). We also evaluated the full 1,319 question GSM8K test set, finding a 3.92 percentage point improvement over stabilized Llama GRPO (W3/Q5). Additional diagnostics show that 21% to 35% of CSP pairs on MATH contain two incorrect answers, while A-CSP outperforms the tested online DPO baseline by 1.44 points on GSM8K (W4/Q3/Q4). These results strengthen the evidence for objective switching while clarifying when its benefits are limited.
W1 / Q1: Why switch to CSP if both methods cannot learn from zero variance groups?
We agree that neither GRPO nor CSP can obtain useful reward information from a group whose shaped rewards are identical. Our motivation is not to recover this missing information, but to balance performance and training stability.
GRPO can achieve strong performance when training progresses well, but some runs undergo severe regression. CSP uses a different objective that may provide more conservative updates. In our additional experiments, we observed collapse onset in 10 GRPO runs but none of the CSP only runs examined. This supports the motivation for switching objectives.
A-CSP therefore starts with GRPO to pursue reward improvement, switches to CSP when the observed reward regime changes, and can return to GRPO when the controller condition is satisfied. The intention is to use GRPO when its training signal remains useful while limiting exposure to regimes where a different objective may be preferable.
We use
zv_fracbecause it is directly available from the current rollout groups and requires no additional model calls. It measures the prevalence of groups without useful relative reward differences. It provides a simple, inexpensive rule for balancing the two objectives.W2 / Q2: Why favor A-CSP over simpler switching schedules?
The main value of A-CSP is that it chooses the training mode from observed reward statistics, rather than requiring a switching step to be selected in advance. This is useful because different models and tasks can reach very different reward regimes during training. We added the requested fixed switching comparisons:
These results show that a well chosen fixed schedule can match or exceed A-CSP. We therefore do not claim that repeated adaptive switching is always necessary.
However, a fixed schedule requires choosing when to switch, while A-CSP provides a common rule that responds to the observed training state. The method introduces negligible controller overhead and does not require additional model evaluations.
We view this as the practical contribution of the controller: a simple way to adjust the optimization objective during training without manually specifying a schedule for each setting. Our results also show where this flexibility helps and where a simpler schedule is sufficient.
...(Continued in the next post, post 1/3)