acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Contrastive Self-Play for Regime-Aware Reinforcement Learning Fine-Tuning of Language Models

Abstract

Reinforcement learning with verifiable rewards (RLVR) relies on variation among sampled rewards. When every response in a rollout group receives effectively the same shaped verifier reward, Group Relative Policy Optimization (GRPO) receives no policy gradient from reward for that group. We ask whether the optimization objective should change as such groups become common. We introduce Adaptive Contrastive Self-Play (A-CSP), which monitors the fraction of zero variance groups and switches between GRPO and a contrastive objective that learns from higher and lower reward responses when eligible pairs exist. Across 156 training runs on GSM8K and a custom MATH-L1-3 pool, we find that the value of switching is strongly regime dependent. On Qwen2.5-1.5B GSM8K, A-CSP reaches similar peak exact match to GRPO, 0.730 versus 0.737, while retaining more of its peak performance at the end of training, 0.97 versus 0.90. On Qwen2.5-3B, A-CSP improves peak accuracy on both tasks and final accuracy on GSM8K, while final accuracy on MATH-L1-3 is essentially unchanged. Mechanistic controls show that masking KL on zero variance groups does not reproduce the A-CSP behavior, while the decomposition suggests that batch composition affects optimization dynamics more than KL masking alone on the flagship setting. Simple switching schedules can also match or exceed A-CSP in peak accuracy, showing that zero variance alone is not sufficient to determine the best switching policy. These results show that adaptive objective switching can improve RLVR training in some regimes, but its benefit depends on more than the prevalence of zero variance groups.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, GRPO, language model fine tuning, adaptive objective switching, contrastive self play, reward variance, preference optimization, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9d8TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Contrastive Self-Play (A-CSP), which switches between GRPO and contrastive preference optimization based on the fraction of zero-variance reward groups. A-CSP constructs preference pairs from higher- and lower-reward responses to improve RL fine-tuning. Experiments on GSM8K and MATH-L1-3 show that A-CSP can improve final performance and reduce late-training degradation in some settings. However, the gains are inconsistent across models and tasks, and simpler switching schedules can achieve comparable or better results. The paper also includes ablations and analyses of the optimization and switching mechanisms.

Critical strengths
  1. The paper addresses a relevant issue in RLVR. Zero-variance reward groups are a common challenge for group-based policy optimization. Investigating whether the optimization objective should change as training progresses is an interesting direction.

  2. The method is relatively simple to implement. A-CSP uses reward statistics already available during GRPO training and does not require an additional reward model or separate verifier. The computational overhead of the switching controller itself is negligible.

  3. The experimental analysis goes beyond a basic GRPO comparison. The paper includes experiments on KL masking, informative-group filtering, dynamic sampling, switching schedules, and CSP objective components. I particularly appreciate the comparison with shuffled switching schedules that approximately match the total number of CSP steps. These experiments help distinguish some possible explanations for the observed improvements.

Critical weaknesses
  1. The motivation for the adaptive switching mechanism is not fully convincing. The paper argues that zero-variance reward groups provide no useful gradient for GRPO. However, CSP also cannot construct preference pairs from these groups. It is therefore unclear why a higher fraction of zero-variance groups should favor CSP over GRPO. More justification for this switching criterion is needed.

  2. The benefit of adaptive switching is not clearly established. In Table 4, random switching outperforms A-CSP on Qwen2.5-1.5B GSM8K while using far fewer CSP steps (31 vs. 283). On Qwen2.5-3B GSM8K, A-CSP spends about 90% of training in CSP mode. These results raise the question of whether adaptive switching is actually necessary or whether a simple fixed schedule would achieve similar performance.

  3. The experimental evaluation is somewhat limited. The experiments use only 500 training questions, 100 evaluation questions, and three random seeds. Moreover, the reported peak accuracy is selected from repeated evaluations on the same subset, without a separate final test evaluation. Some results also show very large variance across seeds. These limitations make it difficult to judge how reliable the improvements are.

  4. The CSP objective needs further justification. CSP combines a pairwise preference loss, supervised loss, and entropy regularization, but the contribution of each component is not entirely clear. Also, preference pairs are selected using shaped rewards, meaning that both responses in a pair may be incorrect. It would be useful to understand how this affects training and whether the proposed CSP objective offers advantages over existing online preference optimization methods.

  5. The improvements are inconsistent across models and tasks. A-CSP performs worse than GRPO on Qwen2.5-1.5B MATH-L1-3, while both methods collapse to zero final accuracy on Llama-3.2-3B MATH-L1-3. Since the switching thresholds are fixed and the zero-variance statistic depends on reward design, it is unclear how well the method generalizes to other settings.

Questions for authors
  1. Since CSP also cannot learn from zero-variance groups, why should a high fraction of such groups favor switching from GRPO to CSP? Is there empirical evidence supporting this choice?

  2. On Qwen2.5-3B GSM8K, A-CSP spends around 90% of training in CSP mode. How does it compare with a simple GRPO-to-CSP schedule with matched CSP exposure?

  3. What fraction of the preference pairs constructed by CSP contain two incorrect responses? Have the authors examined whether these pairs help or hurt training?

  4. Could the authors compare CSP with existing online preference optimization methods under the same training setup? This would help clarify the contribution of the proposed objective.

  5. Can the authors provide evaluations on larger standard benchmarks or held-out test sets? Given the small evaluation subsets and relatively large variance across seeds, it is difficult to assess the reliability of some reported improvements.

3 comments

Thank you for the thoughtful review and for recognizing the value of our method and experimental controls. Your questions helped us clarify the motivation for A-CSP and conduct additional experiments. We address each concern below.


TL;DR: Our motivation is to balance GRPO's performance potential with the risk of severe training regression, not to recover supervision from zero variance groups. A-CSP provides an inexpensive way to switch between GRPO and CSP based on observed reward statistics, with adjustable thresholds to favor more aggressive or conservative training (W1/Q1, W5). New fixed switch experiments show comparable performance, confirming that objective switching can be effective even with simple schedules, while A-CSP avoids specifying the switching point in advance (W2/Q2). We also evaluated the full 1,319 question GSM8K test set, finding a 3.92 percentage point improvement over stabilized Llama GRPO (W3/Q5). Additional diagnostics show that 21% to 35% of CSP pairs on MATH contain two incorrect answers, while A-CSP outperforms the tested online DPO baseline by 1.44 points on GSM8K (W4/Q3/Q4). These results strengthen the evidence for objective switching while clarifying when its benefits are limited.


W1 / Q1: Why switch to CSP if both methods cannot learn from zero variance groups?

We agree that neither GRPO nor CSP can obtain useful reward information from a group whose shaped rewards are identical. Our motivation is not to recover this missing information, but to balance performance and training stability.

GRPO can achieve strong performance when training progresses well, but some runs undergo severe regression. CSP uses a different objective that may provide more conservative updates. In our additional experiments, we observed collapse onset in 10 GRPO runs but none of the CSP only runs examined. This supports the motivation for switching objectives.

A-CSP therefore starts with GRPO to pursue reward improvement, switches to CSP when the observed reward regime changes, and can return to GRPO when the controller condition is satisfied. The intention is to use GRPO when its training signal remains useful while limiting exposure to regimes where a different objective may be preferable.

We use zv_frac because it is directly available from the current rollout groups and requires no additional model calls. It measures the prevalence of groups without useful relative reward differences. It provides a simple, inexpensive rule for balancing the two objectives.

W2 / Q2: Why favor A-CSP over simpler switching schedules?

The main value of A-CSP is that it chooses the training mode from observed reward statistics, rather than requiring a switching step to be selected in advance. This is useful because different models and tasks can reach very different reward regimes during training. We added the requested fixed switching comparisons:

SettingA-CSP compared with fixed switchingResult
Qwen2.5-1.5B GSM8K, matched exposure, 3 seedsFinal EM difference: +0.0045, interval [-0.0159, +0.0232]Similar performance
Qwen2.5-3B GSM8K, stabilized setting, fixed switch at step 250Final EM difference: 0.000Equal mean performance
Qwen2.5-3B GSM8K, retrospective fixed switch at step 150, seed 7A-CSP: 0.8393; fixed: 0.8544Fixed switch performs better for this seed

These results show that a well chosen fixed schedule can match or exceed A-CSP. We therefore do not claim that repeated adaptive switching is always necessary.

However, a fixed schedule requires choosing when to switch, while A-CSP provides a common rule that responds to the observed training state. The method introduces negligible controller overhead and does not require additional model evaluations.

We view this as the practical contribution of the controller: a simple way to adjust the optimization objective during training without manually specifying a schedule for each setting. Our results also show where this flexibility helps and where a simpler schedule is sufficient.

...(Continued in the next post, post 1/3)

...(Continued from the previous post, post 2/3)

W3 / Q5: Evaluation reliability and larger test sets

We have now evaluated final checkpoints on the complete GSM8K test set of 1,319 questions. We also separately evaluated 1,044 questions not used in the original trajectory evaluations.

The expanded evaluation provides additional positive evidence:

Model and settingA-CSP minus GRPO, Full Test EMReported interval
Qwen2.5-3B, stabilized GRPO, KL 0.04+0.0066[-0.0048, +0.0177]
Qwen2.5-3B, stabilized GRPO, KL 0.10+0.0091[-0.0096, +0.0258]
Llama-3.2-3B GSM8K, stabilized GRPO, KL 0.04+0.0392[+0.0263, +0.0528]

The Qwen results remain positive on average, although the intervals include zero. On the 1,044 example subset, Qwen2.5-3B at KL 0.04 improves by 0.0134, with a reported interval of [+0.0013, +0.0255].

More importantly, A-CSP retains a 3.92 percentage point advantage over stabilized GRPO on Llama-3.2-3B GSM8K, with positive differences across all three seeds. This shows that the benefit is not limited to comparisons against GRPO runs that collapse completely. These additional evaluations substantially expand the original 100 example protocol.

W4 / Q3 / Q4: CSP objective, incorrect pairs, and comparison with preference optimization

CSP combines three complementary components. The pairwise term learns from reward ordering, the supervised term reinforces the preferred completion, and entropy regularization affects the policy distribution. Our existing ablations show that these terms affect peak performance and retention differently. For example, the full A-CSP objective achieves retention of 0.97, compared with 0.91 for the pairwise term alone and 0.95 for pairwise learning with entropy.

We also measured the correctness of CSP preference pairs using additional checkpoint rollouts:

DatasetPairs with two incorrect responsesInterpretation
GSM8KNone observed across 4,500 diagnostic groupsIncorrect answers usually share the same shaped reward
MATH-L1-3, noncollapsed checkpoints21% to 35%Shaped rewards can rank two incorrect answers
MATH-L1-3, collapsed Llama checkpoint84%Most eligible pairs contain two incorrect answers

On MATH-L1-3, the higher reward incorrect answer is generally closer to the reference answer under the implemented edit distance reward. Such pairs can provide relative training information even when neither answer is fully correct.

To address the comparison with existing preference optimization methods, we also evaluated online DPO with a fixed reference policy.

ComparisonFull GSM8K Final EM differenceReported interval
A-CSP minus online DPO+0.0144[+0.0025, +0.0260]
A-CSP minus online DPO, 1,044 example subset+0.0204[+0.0083, +0.0322]

A-CSP outperforms this online DPO baseline in all three seeds. This provides additional evidence that the proposed training strategy is competitive with an established form of preference optimization.

...(Continued in the next post, post 2/3)

...(Continued from the previous post, post 3/3)

W5: Why are the improvements inconsistent across models and tasks?

We agree that the benefits depend on the model and task. This is an important part of our research question.

Our goal is to study settings where the model still has substantial room to improve, but RL training can also undergo severe regression. We do not expect A-CSP to provide the same benefit when a model already solves nearly every problem or when neither objective receives useful reward information.

The experiments show both sides of this tradeoff. A-CSP improves late training behavior in several GSM8K settings, including the new comparison against stabilized Llama GRPO. It also fails to improve some MATH-L1-3 settings. These differences are why we study objective switching rather than proposing CSP as a universal replacement for GRPO.

The central contribution is a practical framework for choosing between optimization objectives during RLVR training, supported by experiments that identify both its benefits and its limits. We believe this is a useful direction because a fixed optimization objective need not be the best choice throughout training, even when no single switching rule works best in every setting.

Another practical advantage is that A-CSP allows users to adjust the balance between performance and stability through its switching thresholds. A higher entry threshold favors more aggressive GRPO training by requiring stronger evidence before switching to CSP, while a lower threshold favors earlier use of the more conservative CSP objective. This flexibility allows the same framework to accommodate different training preferences without changing the optimization objectives. We do not claim that one threshold is optimal across all settings.


Thank you again for the constructive feedback. The additional experiments directly address your concerns about switching schedules, evaluation reliability, incorrect preference pairs, and comparisons with existing preference optimization methods. We believe the main value of A-CSP is not that CSP solves zero variance or that one switching rule is always optimal. Rather, it provides a simple, low overhead framework for balancing two optimization objectives as training conditions change. The new full test results and stronger baselines provide further evidence of its practical value, while our controls make clear where simpler alternatives are competitive. We hope these results clarify the motivation and strengthen the contribution of the paper.

Reviewer_nonhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Contrastive Self-Play (A-CSP), which switches between GRPO and an online contrastive objective according to the prevalence of zero-variance reward groups. The contrastive branch learns from eligible higher- and lower-reward response pairs rather than recovering supervision from zero-variance groups themselves. Experiments show improvements in late-training performance in some settings, but benefits vary across models, tasks, and switching schedules.

Critical strengths
  1. The paper raises a worthwhile question: whether RLVR should use the same optimization objective throughout training. The proposed controller is simple and reuses existing rollout rewards without additional model calls.
  2. The study includes useful controls covering KL masking, informative-group filtering, dynamic sampling, switching schedules, and CSP objective components.
  3. The presentation is appropriately candid about negative results and boundary conditions, including settings where simple schedules outperform A-CSP or switching fails to prevent regression.
Critical weaknesses
  1. The paper does not report difficulty-based data pre-screening. Repeatedly training on prompts that are already mastered or consistently unsolved may amplify the prevalence and practical impact of zero-variance groups. It remains unclear whether A-CSP retains its benefits on a better-curated training set.

  2. The evaluation is limited to a few relatively older, small-model families, with models up to 3B parameters and small subsets of GSM8K and MATH. This narrow scope makes it difficult to establish the method’s reliability on larger, more recent models and broader reasoning tasks.

  3. The necessity of adaptive switching is not established. Random switching outperforms A-CSP on Qwen2.5-1.5B GSM8K, while A-CSP spends approximately 90% of training in CSP on Qwen2.5-3B GSM8K. A matched single-switch baseline—GRPO warmup followed by CSP—is missing, leaving unclear whether continual zero-variance monitoring is needed.

  4. Switching to CSP simultaneously changes pair selection, reference-policy regularization, supervised learning, entropy regularization, and loss aggregation. Although component ablations are provided, they do not sufficiently isolate the benefit of objective switching from these accompanying changes.

  5. Some large gains arise in settings where GRPO suffers severe late-stage failure, including zero final accuracy or very high cross-seed variance. The causes of these failures remain unclear. Failure diagnostics and comparisons against appropriately tuned or stabilized GRPO are needed to determine whether A-CSP addresses a general optimization problem rather than instability specific to the chosen configuration.

Questions for authors

Could the authors report separate trajectories for all-correct, all-incorrect, and mixed-correctness groups, alongside the distribution of shaped-reward variance? This would clarify which training conditions drive the controller’s decisions.

4 comments

Thank you for the thoughtful review and for appreciating our experimental controls and discussion of negative results. Your comments raise important questions about when objective switching is useful and whether the benefits remain under stronger baselines. We conducted additional experiments to investigate these questions.


TL;DR: We intentionally study challenging tasks with smaller models to understand severe GRPO training regression; difficulty based curriculum learning is a complementary approach. New experiments show that fixed switching can match A-CSP, but A-CSP automatically selects the training mode without a predefined switch point. Our CSP ablations clarify the role of its components, and A-CSP outperforms an online DPO baseline by 1.44 percentage points. On the full 1,319 question GSM8K test set, A-CSP retains a 3.92 point advantage over stabilized Llama GRPO, although gains on Qwen are smaller. New correctness diagnostics also show that 97%–98% of zero variance groups are all correct in the sampled Qwen3B checkpoints, clarifying what the controller measures and why its benefits depend on the training regime.


W1: Difficulty based data selection

Thank you for this suggestion. We agree that curriculum learning could help reduce uninformative rollout groups, particularly when questions are already mastered or remain too difficult.

Our experiments use the same fixed training pool and sampling procedure across methods so that we can focus on the effect of changing the optimization objective. Difficulty based sampling addresses a different part of the training process and could naturally be combined with A-CSP.

We see this as a valuable direction rather than a competing explanation that invalidates objective switching. We do not claim that A-CSP is preferable to a carefully designed curriculum, which has not been directly tested here.

W2: Why focus on relatively small models and challenging reasoning tasks?

This choice is intentional and closely related to our research goal. We want to study settings where the model has substantial room to improve through RL, but where training can also become unstable. Relatively small models on challenging mathematical reasoning tasks allow us to observe both meaningful learning progress and severe training regression.

For a model that already solves nearly every problem, there may be little room for either GRPO or CSP to improve performance. At the other extreme, if almost no responses provide useful reward differences, neither method has sufficient supervision. The setting of interest lies between these extremes, where learning is possible but training outcomes can differ substantially.

The purpose of our experiments is therefore not to demonstrate the strongest possible mathematical reasoning model, but to understand how optimization behavior changes during challenging RL training. We agree that larger and more recent models are important for establishing broader generalization. The current results establish the behavior and limitations of A-CSP in the settings studied.

W3: Is adaptive switching necessary compared with a single fixed switch?

The motivation for adaptive switching is that a suitable transition point may differ across models, tasks, and training runs. If that point were known in advance, a fixed schedule could certainly be used. A-CSP instead starts with GRPO and uses the user preferred zv_frac to determine when to change objectives. It also allows a return to GRPO when the controller condition is met. We added the requested single switch comparisons:

SettingA-CSP versus fixed switch, Final EMFinding
Qwen2.5-1.5B GSM8K, matched CSP exposure, 3 seeds+0.0045, interval [-0.0159, +0.0232]Comparable performance
Qwen2.5-3B GSM8K, fixed switch at step 2500.000 mean differenceEqual performance
Qwen2.5-3B GSM8K, fixed switch at step 150, one seed-0.0151Fixed switch performs better

These experiments show that fixed switching can achieve similar results. This is useful evidence: the benefit of changing objectives does not require a complicated controller. The value of A-CSP is that it provides an automatic way to select the optimization mode without specifying a transition step for each training setting. We do not claim that repeated adaptive switching always outperforms a well chosen fixed schedule. Rather, A-CSP offers a practical way to make this decision from the observed training state, while allowing the balance between GRPO and CSP to be adjusted through its thresholds.

...(Continued in the next post, post 1/3)

...(Continued from the previous post, post 2/3)

W4: Does switching simply combine several changes to the optimization objective?

We agree that switching from GRPO to CSP changes more than one part of the update. This is intentional: our goal is to study switching between complete optimization objectives rather than modifying a single GRPO term.

We included ablations to understand the contribution of the CSP components. On Qwen2.5-1.5B GSM8K, using the same adaptive controller:

CSP objectivePeak EMFinal EMRetention
Pairwise only0.7500.6830.91
SFT only0.7400.4570.62
Pairwise + SFT0.7570.6930.92
Pairwise + entropy0.7530.7170.95
Full A-CSP0.7300.7070.97

The full objective achieves the highest retention among these adaptive configurations, although several partial objectives have higher peak or final accuracy. This illustrates why peak performance and late training behavior should be considered separately.

Our GRPO decomposition further shows that KL masking alone does not reproduce A-CSP's behavior. We also added a comparison with online DPO using a fixed reference policy:

ComparisonA-CSP advantage in Final EMReported interval
Full GSM8K test, 1,319 questions+1.44 points[+0.25, +2.60]
Subset unused in trajectory evaluation, 1,044 questions+2.04 points[+0.83, +3.22]

A-CSP performs better in all three seeds of this comparison. These results provide evidence that A-CSP is competitive with another preference optimization approach.

W5: Does A-CSP only help because the GRPO baseline is unstable?

This is an important question because severe training regression is exactly the behavior that motivated this work.

Our aim is not simply to obtain a higher peak accuracy. It is to explore whether changing the optimization objective can help preserve performance during training, particularly when continued GRPO updates lead to large regression. We therefore added experiments with a lower learning rate and different KL coefficients to stabilize GRPO.

Model and settingA-CSP minus stabilized GRPO, Final EMReported interval
Qwen2.5-3B GSM8K, KL 0.04+0.66 points[-0.48, +1.77]
Qwen2.5-3B GSM8K, KL 0.10+0.91 points[-0.96, +2.58]
Llama-3.2-3B GSM8K, KL 0.04+3.92 points[+2.63, +5.28]

With the stabilized GRPO settings, the severe regression previously observed on Qwen2.5-3B disappears, and the performance difference becomes small. This confirms that baseline stability matters.

However, A-CSP still improves Final EM by 3.92 percentage points on Llama-3.2-3B GSM8K, with positive differences across all three seeds, even against stabilized GRPO. This is important because it shows that the observed benefit is not limited to comparisons against GRPO runs that collapse completely.

Our results do not establish that A-CSP always outperforms a tuned GRPO baseline. Instead, they show that the benefits of objective switching depend on the model and training conditions. We believe identifying and addressing this variation is a meaningful research direction.

...(Continued in the next post, post 2/3)

...(Continued from the previous post, post 3/3)

Q1: What do zero variance groups contain?

Thank you for suggesting this analysis. We added checkpoint rollout diagnostics that separately measure all correct, all incorrect, and mixed correctness groups.

The following results are from GSM8K:

Model and methodAll correctAll incorrectMixed
Qwen2.5-3B GRPO62.2%1.7%36.1%
Qwen2.5-3B A-CSP57.0%4.9%38.1%
Llama-3.2-3B52.1%1.4%46.4%
Qwen2.5-1.5B21.6%5.8%72.6%

For Qwen2.5-3B, approximately 97% to 98% of the zero variance groups are all correct at the sampled checkpoints. This clarifies an important distinction: zero shaped reward variance does not necessarily mean the model is failing. It can also mean that the model already solves the sampled questions.

The distinction is different on MATH-L1-3, where incorrect answers can receive different shaped rewards. In our additional diagnostics, 21% to 35% of eligible CSP pairs at noncollapsed checkpoints contain two incorrect responses. These pairs can still be ordered by the shaped reward, although their independent contribution to training has not been isolated.

The measurements come from sampled checkpoints rather than complete historical rollout records. They describe the reward regimes encountered by the models without assuming that zero variance is equivalent to incorrectness or impending collapse.


We appreciate the reviewer's focus on understanding when objective switching is useful. The additional experiments make the conclusions more precise while providing new evidence beyond the original evaluation.

Our main contribution is to investigate a practical alternative to using one optimization objective throughout RLVR training. A-CSP starts with GRPO, uses observed reward statistics to adjust the training objective, and adds negligible controller overhead. The results show that this approach can improve late training behavior in some challenging settings, even against a stabilized GRPO baseline.

We believe the combination of a simple method, extensive controls, additional full test evaluations, and clear evidence about its operating limits makes this a useful contribution to understanding RLVR training.

Thank you for your detailed response. Your response answered some of my questions. However, one of my main concerns—the choice of models and datasets—remains unaddressed. Although the authors state that these models and datasets were chosen to better observe the experimental phenomena, (1) experimental conclusions may change as models are updated and scaled, so using more practical experimental settings would increase confidence in the findings; and (2) in the ablation provided for W4 and the performance in Table 2, some differences are very small and the ablation results are not consistently positive, which may also be related to the experimental setup.

In my experience, using Qwen2.5 or Qwen3 series models trained on DAPO-Math-17k and evaluated on datasets such as AIME, Minerva, and AMC would provide clearer differentiation and more convincing evidence. However, this would involve restructuring the entire experimental setting, which is not feasible to resolve during the rebuttal period.

Overall, while the authors' rebuttal addresses several points, the substantive improvements are limited, so I have decided to maintain my score.

Reviewer_uw4vOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether the objective in reinforcement learning with verifiable rewards should change when many GRPO rollout groups have zero reward variance and contribute no reward-derived policy gradient. It introduces Contrastive Self-Play, which applies a DPO-style pairwise update between the highest- and lowest-reward completions of groups with a sufficient reward gap, plus a supervised term on the preferred completion and an entropy term, with no frozen reference policy. Adaptive Contrastive Self-Play switches between GRPO and this contrastive objective based on the fraction of zero-variance groups, checked periodically with hysteresis. On grade-school math problems and a custom pool of competition-style problems with language models, the benefit is regime-dependent: switching mainly improves late-training retention in some settings but does not dominate across models, tasks, or endpoints, and non-adaptive or random schedules can match or exceed it, suggesting that zero-variance prevalence alone is not a sufficient signal for choosing the objective.

Critical strengths
  • The question is well-motivated and practically relevant. Zero-variance groups are a common and wasteful feature of GRPO training

  • The control experiments are more thorough than is typical for an empirical method paper of this size

Critical weaknesses
  • The method is heuristic rather than principled. The CSP objective combines a detached-weight pairwise term, a supervised term on the preferred completion, and an entropy term (Eq. (10)) with no derivation showing that this combination optimizes a stated underlying objective or addresses the zero-variance gradient problem directly, and the controller switches on a fixed empirical statistic with hand-set hysteresis thresholds rather than a derived criterion.

  • The remaining evidence is too weak to separate the method from noise, baseline instability, and simpler remedies for the same problem.

Questions for authors
  • Given that random switching with 31 CSP steps beats A-CSP on the flagship setting (Table 4), what exactly is the claimed value of the $\mathrm{zv_frac}$ controller? Can you add the missing control — one initial GRPO phase followed by a single CSP block of matched total exposure — on Qwen2.5-3B GSM8K, so that repeated adaptive switching is separated from total CSP exposure?

  • Can the main comparisons be re-evaluated on the full GSM8K test set (1,319 examples) with more seeds and a fixed, locked evaluation split? In particular, which of the Final EM differences in Tables 2–4 survive at that precision?

  • Why does GRPO collapse to zero on the Llama-3.2-3B cells, and were the GRPO hyperparameters (learning rate, KL coefficient) tuned for each model family? Does the A-CSP advantage persist in cells where the GRPO baseline is itself stable?

3 comments

Thank you for the thoughtful review and for recognizing the importance of the problem and our experimental controls. We appreciate your concerns about the motivation for switching, the need for stronger baselines, and the reliability of our results. We have conducted several additional experiments to address these points.


TL;DR: A-CSP studies how to balance GRPO's performance potential with the risk of severe training regression. Its goal is not to recover supervision from zero variance groups, but to provide a simple way to switch optimization objectives as training conditions change (W1). We added single switch baselines, full GSM8K test evaluations, and stabilized GRPO comparisons (W2/Q1/Q2/Q3). Fixed switching can match A-CSP, but A-CSP selects the training mode automatically without a predefined transition step (Q1). On the full 1,319 question GSM8K test set, A-CSP retains a 3.92 percentage point advantage over stabilized Llama GRPO, with positive results across all three seeds (Q2/Q3). These results provide stronger evidence for objective switching in challenging RLVR settings, while clarifying when its benefits are smaller.


W1: Motivation for the CSP objective and the switching criterion

We agree that neither GRPO nor CSP can learn useful reward information from groups where all shaped rewards are identical. Our motivation is not to recover this missing information, but to balance performance and training stability.

GRPO can achieve strong performance, but some training runs experience severe regression. CSP uses a different optimization objective that may provide more conservative updates. In our additional experiments, we observed collapse onset in 10 GRPO runs but none of the CSP only runs examined. While this does not establish that CSP is always more stable, it supports our motivation for studying objective switching.

A-CSP therefore starts with GRPO to pursue reward improvement, switches to CSP when the observed reward regime changes, and can return to GRPO when the controller condition is met.

The CSP objective combines three components with different purposes. The pairwise term learns from the ordering of sampled responses, the supervised term reinforces the preferred completion, and entropy regularization influences the policy distribution. Our existing ablations show that these components affect peak performance and retention differently. The full objective achieves retention of 0.97, compared with 0.91 for the pairwise term alone.

We use zv_frac because it directly measures how often rollout groups lack relative reward differences and is already available during GRPO training. It requires no additional verifier, reward model, or model calls. We agree that this is an empirical switching criterion rather than a theoretically optimal one.

The thresholds also allow different training preferences. A higher entry threshold favors continued GRPO training, while a lower threshold favors earlier switching to CSP. This provides a simple way to adjust the balance between aggressive and conservative optimization without changing the objectives.

W2: Stronger evidence beyond noise, unstable baselines, and simpler alternatives

We appreciate this concern and have conducted several additional experiments specifically to address it.

We extended evaluation from the original 100 question subsets to the full 1,319 question GSM8K test set. We also evaluated stabilized GRPO with different learning rates and KL coefficients, added fixed single switch controls, and compared against online DPO with a fixed reference policy.

The results provide a clearer picture. A-CSP retains a positive advantage against stabilized Llama GRPO, improves over the tested online DPO baseline by 1.44 percentage points, and remains competitive with fixed switching schedules.

We also find that stabilizing GRPO reduces the advantage of A-CSP in some Qwen settings. This is consistent with our research goal: objective switching is most relevant when training regression is a practical concern, rather than when GRPO already trains reliably.

The following sections provide the detailed results.

...(Continued in the next post, post 1/3)

...(Continued from the previous post, post 2/3)

Q1: Why use the zv_frac controller instead of random or fixed switching?

We agree that random switching achieves better results on Qwen2.5-1.5B GSM8K in the original Table 4. This shows that A-CSP does not always find the best schedule.

The reason for using a controller is that a suitable switching point may differ across models, tasks, and training runs. A fixed schedule requires this point to be selected in advance, while A-CSP uses observed reward statistics to decide when to change objectives. We added the requested single switch comparisons:

SettingA-CSP versus fixed switch, Final EMFinding
Qwen2.5-1.5B GSM8K, matched exposure, 3 seeds+0.45 points, interval [-1.59, +2.32]Comparable performance
Qwen2.5-3B GSM8K, fixed switch at step 2500.00 pointsEqual mean performance
Qwen2.5-3B GSM8K, fixed switch at step 150, one seed-1.51 pointsFixed switch performs better for this seed

These results show that a suitable fixed schedule can match A-CSP. The matched exposure comparisons are retrospective, so they also rely on information from the observed A-CSP training schedules.

The value of A-CSP is that it makes the switching decision automatically, without requiring a separate transition point to be chosen for every training setting. Its controller uses existing rollout statistics, adds negligible computational overhead, and allows the balance between GRPO and CSP to be adjusted through the thresholds.

Our claim is not that adaptive switching always outperforms fixed switching. Rather, we introduce a practical and inexpensive way to adjust the optimization objective during training. The experiments show that objective switching can be effective, even when a simpler schedule is sufficient.

Q2: Full GSM8K evaluation and reliability of the improvements

We have now evaluated final checkpoints on the complete GSM8K test set of 1,319 questions. We also report results on 1,044 questions not used in the original repeated trajectory evaluations.

The expanded evaluations provide the following results against stabilized GRPO:

Model and settingA-CSP minus GRPO, Full Test Final EMReported interval
Qwen2.5-3B, KL 0.04+0.66 points[-0.48, +1.77]
Qwen2.5-3B, KL 0.10+0.91 points[-0.96, +2.58]
Llama-3.2-3B, KL 0.04+3.92 points[+2.63, +5.28]

For Qwen2.5-3B with KL 0.04, the improvement on the 1,044 question subset is +1.34 points, with a reported interval of [+0.13, +2.55].

The Qwen results are positive on average. The Llama result is stronger: A-CSP improves Final EM by 3.92 percentage points, with positive differences across all three seeds, even against stabilized GRPO.

These evaluations address the concern that our original findings relied only on 100 question subsets. The final checkpoints are now evaluated on a much larger test set, and the separate 1,044 question subset helps assess whether the improvements extend beyond the examples used for repeated evaluation.

...(Continued in the next post, post 2/3)

...(Continued from the previous post, post 3/3)

Q3: Why does GRPO collapse, and does A-CSP still help when GRPO is stabilized?

Severe training regression is the main problem motivating this work. We intentionally study relatively small models on challenging reasoning tasks because these settings allow us to observe both meaningful RL improvement and unstable training behavior.

Our goal is not to replace GRPO when it already trains reliably. Instead, we want to explore whether changing optimization objectives can help in settings where continued GRPO training leads to substantial regression.

We agree that the original GRPO hyperparameters were not necessarily optimal for every model family. We therefore added experiments with a lower learning rate and different KL coefficients.

Model and settingStabilized GRPO behaviorA-CSP advantage, Full Test Final EM
Qwen2.5-3B, KL 0.04Severe collapse no longer observed+0.66 points
Qwen2.5-3B, KL 0.10Severe collapse no longer observed+0.91 points
Llama-3.2-3B, KL 0.04Stabilized GRPO comparison+3.92 points

These experiments lead to two important findings. First, stabilizing GRPO can substantially reduce the advantage of A-CSP. This is expected: when GRPO already trains reliably, there is less opportunity for objective switching to improve late training performance. Second, A-CSP still shows a meaningful improvement on Llama-3.2-3B GSM8K after GRPO stabilization, with positive differences across all three seeds. This shows that the benefit is not limited to comparing against GRPO runs that collapse completely.

We also inspected the evaluation behavior of the Llama checkpoints. Repeated full test evaluations show some numerical sensitivity, typically around one percentage point. This is smaller than the observed 3.92 point difference, although it remains a source of uncertainty. We have not isolated a single cause for the original GRPO collapse.

The important distinction is that A-CSP is intended for challenging training regimes where stability is difficult to maintain, not as a universal replacement for a well tuned GRPO optimizer.

The Qwen results show that stabilization can be sufficient in some settings. The Llama results show that objective switching can still provide additional value in others. Understanding this difference is an important part of the contribution.


Thank you again for the constructive feedback. Your questions motivated several additional experiments that strengthen the evaluation and clarify the contribution of A-CSP.

The new results show that objective switching can improve late training performance, including against a stabilized GRPO baseline, while also identifying settings where fixed schedules or simpler stabilization methods are sufficient.

We believe the main contribution is a practical framework for using different optimization objectives during RLVR training, rather than assuming one objective is always the best choice. A-CSP provides an automatic, low overhead switching mechanism, and our expanded experiments help establish both its value and its limits.

We hope these results address your concerns and clarify why objective switching is a useful direction for challenging RLVR training.

Comments on the paper

2 comments

We would like to sincerely thank all reviewers and the Area Chair for their time, effort, and thoughtful feedback.

We apologize for taking some extra time to respond, as we wanted to carefully address the concerns with additional experiments rather than explanations alone. We have now completed several new experiments and shared the results in our individual responses.

We truly appreciate the constructive feedback. The questions and suggestions have helped us better understand the strengths and limitations of our work, and we have learned a lot through this process.

Thank you again for your time and consideration. We are happy to continue the discussion and welcome any further questions or suggestions.

Dear Reviewers,

We would like to kindly follow up on our rebuttal and additional experimental results. We hope our responses have helped address your questions and concerns.

We truly appreciate your time and thoughtful feedback. If you feel that the new evidence and explanations have sufficiently addressed your concerns, we would be very grateful if you could reconsider your score.

We are also happy to clarify anything further or continue the discussion. Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.