- Rating
- 3
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses style drift introduced by privileged reference solutions in on-policy self-distillation. RLCSD contrasts teacher predictions conditioned on correct and incorrect references, using multiple negative references to construct a token-level training signal. It combines this signal with verifier-based GRPO advantages through bounded, sign-preserving modulation and separately normalizes the selected and remaining token losses. Experiments on four backbones, mathematical reasoning, and Knights-and-Knaves tasks support improvements over GRPO and one-sided self-distillation, with additional evidence from component ablations and harder logic instances.
Critical strengths
-
The paper connects an observable training problem to a targeted intervention and tests that intervention within a controlled recipe. Replacing the one-sided signal with the contrastive signal improves the 4B model's mathematical average from 74.8 to 78.4 and its logic average from 61.5 to 68.2. Within the shared training recipe, this comparison supports a contribution from the contrastive signal.
-
Results are reported across four backbones, with ablations examining several design choices. Verifier-based outcome anchoring constrains the direction of token-level updates. Together, the cross-backbone results and component controls support the empirical contribution across several tested settings.
Critical weaknesses
-
The evidence does not fully establish that the contrast isolates reasoning quality from style. Correct and incorrect references can differ in length, structure, and solution strategy, while the heuristic style categories include words that also carry logical meaning. This limits how strongly the observed improvements can be attributed to removing stylistic interference or improving reasoning credit assignment.
-
The size and consistency of the advantage over GRPO are less firmly established than the comparison with one-sided self-distillation. The method filters rollout groups and has a reported per-step overhead of roughly 17.5%, but equivalent filtering and total training resources are not clear across comparisons; the main results also lack independent training-seed uncertainty. These details affect the assessment of small gains, such as the mathematical averages of 79.1 versus 79.0 on Olmo, and the reliability of the reported advantage.
Questions for authors
See Weaknesses.