acceptodds
Under review as a conference paper at ICLR 2027

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

11 views

Abstract

On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with that under privileged context, typically a verified solution. However, we show that the resulting distributional gap concentrates on style tokens rather than task-bearing ones, as the hinted model tends to produce shorter, more direct outputs. We term this pathology privilege-induced style drift, which can destabilize training and shorten responses. To address this, we propose RLCSD (Reinforcement Learning with Contrastive on-policy Self-Distillation), which mitigates this drift by contrasting the teacher-student gap under a correct hint against that under a wrong hint, suppressing style shifts induced by hints regardless of correctness and yielding a signal more concentrated on task-bearing tokens. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think across mathematical and logical reasoning show that RLCSD consistently outperforms GRPO and prior OPSD methods, with additional results on agentic tasks supporting broader applicability. We further show that the contrastive principle is general: it plugs into existing OPSD methods to improve them, and its underlying insight extends to broader cross-model on-policy distillation. Code and data are available in anonymized repositories.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, on-policy distillation, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LN3nOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses style drift introduced by privileged reference solutions in on-policy self-distillation. RLCSD contrasts teacher predictions conditioned on correct and incorrect references, using multiple negative references to construct a token-level training signal. It combines this signal with verifier-based GRPO advantages through bounded, sign-preserving modulation and separately normalizes the selected and remaining token losses. Experiments on four backbones, mathematical reasoning, and Knights-and-Knaves tasks support improvements over GRPO and one-sided self-distillation, with additional evidence from component ablations and harder logic instances.

Critical strengths
  1. The paper connects an observable training problem to a targeted intervention and tests that intervention within a controlled recipe. Replacing the one-sided signal with the contrastive signal improves the 4B model's mathematical average from 74.8 to 78.4 and its logic average from 61.5 to 68.2. Within the shared training recipe, this comparison supports a contribution from the contrastive signal.

  2. Results are reported across four backbones, with ablations examining several design choices. Verifier-based outcome anchoring constrains the direction of token-level updates. Together, the cross-backbone results and component controls support the empirical contribution across several tested settings.

Critical weaknesses
  1. The evidence does not fully establish that the contrast isolates reasoning quality from style. Correct and incorrect references can differ in length, structure, and solution strategy, while the heuristic style categories include words that also carry logical meaning. This limits how strongly the observed improvements can be attributed to removing stylistic interference or improving reasoning credit assignment.

  2. The size and consistency of the advantage over GRPO are less firmly established than the comparison with one-sided self-distillation. The method filters rollout groups and has a reported per-step overhead of roughly 17.5%, but equivalent filtering and total training resources are not clear across comparisons; the main results also lack independent training-seed uncertainty. These details affect the assessment of small gains, such as the mathematical averages of 79.1 versus 79.0 on Olmo, and the reliability of the reported advantage.

Questions for authors

See Weaknesses.

Reviewer_HHSdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RLCSD contrasts a teacher's token likelihood under correct and incorrect references to reduce correctness-independent stylistic effects of privileged conditioning. It marginalizes multiple negative references (also studied one negative reference), and uses bounded contrastive modulation with a sign-preserving verifier anchor. Independently normalized selected and unselected token objectives focus training on informative positions. Experiments on four reasoning backbones cover math and in-/out-of-distribution logic, with component ablations, plug-in variants and smaller agentic evaluations.

Critical strengths
  1. A useful diagnosis tied to a concrete intervention. Correct-versus-incorrect conditioning improves the one-sided variant within the same RLCSD framework and helps other distillation methods. This makes the contribution more persuasive
  2. Negative marginalization, self-exclusion, verifier anchoring and independent normalization each have supporting ablations. Logic gains extend to a second model family.
Critical weaknesses

Asymmetry between offline ground truth positive hints and on-policy negative rollouts. The central theoretical premise of RLCSD is that positive and negative hints share the same stylistic distribution so that privilege-induced style drift cancels under subtraction. However, as noted in Section 4.1, the main experiments use dataset-provided ground-truth solutions as the positive hint yᴄ*, while drawing negative hints yw,k* from on-policy student rollouts.

Questions for authors
  1. RLCSD discards query groups with only one incorrect rollout (7 correct, 1 incorrect out of G = 8) because the single incorrect rollout would have no negative hint after self-exclusion. What fraction of training groups falls into this category over the course of training, and what happens if you instead retain those groups (e.g., by applying unmodulated GRPO updates to the 1 incorrect rollout while still applying contrastive modulation to the 7 correct rollouts, which do have 1 negative hint available)?
  2. Regarding the training objective, how would performance be affected if the standard group-normalized GRPO advantage for unmodulated tokens were replaced with RLSD's one-sided privileged advantage? This would allow for a more direct comparison with the RLSD baseline.
Reviewer_HHSROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work presents RLCSD to address privilege-induced style drift in on-policy self-distillation, where a model conditioned on a correct solution shifts toward shorter and more stylistic outputs rather than toward task-relevant reasoning. RLCSD contrasts teacher logits conditioned on correct and incorrect hints, marginalizes over multiple incorrect hints, and uses this bounded contrastive signal only to modulate the verifier-derived GRPO advantage through a sign-preserving two-path objective. Experiments on four reasoning models show the best average mathematical and logical-reasoning performance among GRPO and several self-distillation baselines, with more stable response length, entropy, and reward curves; ablations support contrastive conditioning, negative-hint marginalization, verifier anchoring, and two-path aggregation.

Critical strengths
  • The diagnosis is concrete, and the intervention directly targets it. The paper shows that one-sided privileged supervision concentrates on discourse and formatting tokens, then cancels hint-shared style effects by contrasting the same teacher under correct and incorrect references. The verifier anchoring and sign-preserving clamp keep this dense signal from overriding outcome correctness.

  • The empirical support is broad. RLCSD is evaluated on four models spanning two families, mathematical, logical, and agentic tasks, multiple OPSD baselines, plug-in contrasts, controlled ablations, computational cost, and hyperparameter sensitivity. The gains are strongest on out-of-distribution logical reasoning, and the training-dynamics evidence is useful beyond the final benchmark scores.

Critical weaknesses
  • The central task-versus-style interpretation relies on a heuristic token taxonomy. Task tokens are identified mainly by digits, operators, LaTeX markers, and a math word list, while discourse markers are treated as style. This partition is plausible but not externally validated, and important mathematical content can occur in “neutral” words while formatting tokens can be task-bearing. The claim that contrast removes style drift would be stronger with manual token annotations or alternative independent lexicons and robustness checks over multiple taxonomies.

  • The main experimental protocol lacks uncertainty and uses validation-selected checkpoints. Table 1 reports the checkpoint with the best validation performance for every method and model, but the paper does not clarify the validation/test split, report multiple training seeds, or provide confidence intervals. Several contrasts are small or tied—for example, Olmo-3-7B has the same math average as GRPO—while plug-in gains for OPSD and RLSD are modest. Fixed-checkpoint and repeated-seed comparisons are needed to establish which rankings are stable.

Questions for authors
  • Can you validate the task/style/neutral token taxonomy with human annotations or an alternative independent lexicon, then report Figures 3-4 and the main gains under all partitions?

  • What happens when a rollout group contains no incorrect siblings, few incorrect siblings, or highly homogeneous errors? Does the method gracefully reduce to GRPO, and how sensitive is it to the group size and K?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.