- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper combines reinforcement learning from verifiable rewards with on-policy self-distillation through a single token-level advantage. Each rollout is re-scored under two privileged contexts, one containing the reference answer and one a mistake drawn from the same prompt group, and the clipped difference between the two teacher log-probabilities is added to the group-relative advantage. The intent is that biases shared by both contexts cancel, so the reward still sets the direction of each update while the teacher redistributes credit across tokens. Because the contrast needs no successful rollout, groups in which every rollout fails still receive a signal. Experiments cover math, science question answering, tool use, and code generation.
Critical strengths
-
The diagnosis of why self-distillation fails on long reasoning is the most valuable part of the paper. Figure 4 shows reward, entropy, and response length all falling while the gradient norm grows, and the proposed explanation, that an answer-aware teacher favors continuations heading straight to the answer, is both plausible and testable. Table 4 supports it from the other direction, since adding an analytic reverse KL toward a single teacher costs 14.7 points on the math average while remaining competitive on short-response tasks.
-
The contrastive construction follows from that diagnosis, and the mechanism claim is checked rather than asserted. Single-teacher evidence has a near-zero median but a consistently negative mean, while the contrastive evidence fluctuates around zero without a consistent sign. The design is also cheap in the right way, adding two teacher forward passes over existing rollouts rather than new generation.
Critical weaknesses
-
The reported gains are small relative to the uncertainty in the evaluation, and no uncertainty is reported. Every table appears to come from one training run, and checkpoints are selected by accuracy on the evaluation benchmarks themselves. The margins carrying the conclusions are 2.1 points on the math average over group-relative RL and 1.3 points on the science average over self-distillation. On AIME24 and AIME25, which have thirty problems each, the problem-level standard error alone is roughly 5 to 7 points, so the reported 1.9 and 1.1 point gaps are not resolvable, and the 0.1 point gap on AIME24 between the full method and its contrastive ablation certainly is not. The ablation chain in Table 4 moves by 1.6 and then 1.1 points, so the attribution to the contrast and then to failed groups rests on differences of about this size. Self-distillation alone also wins 4 of the 10 science and tool-use cells, and at one hour the proposed method trails group-relative RL on biology by 3.0 points.
-
The negative context is constructed asymmetrically and the consequence is not analyzed. For an incorrect rollout the negative answer is its own final answer, whereas for a correct rollout it is the most common group mistake. A teacher conditioned on a rollout's own wrong answer will favor precisely the tokens that produced it, so the evidence partly encodes which rollout it is rather than which token deserves credit. In a uniformly failed group the group-relative advantage is zero and this term carries the whole update, so what the method supplies there is a hindsight negative update away from the rollout's own mistake rather than dense credit. Given that negative-only updates are known to destabilize training, I would expect entropy, length, and pass@k curves on math. None are reported.
Questions for authors
-
What is the group size? With the clip bound at 0.2 and the teacher weight at 0.5, the stated sign-preservation condition requires a group size below 10, and the group size never appears in the main text.
-
Can the math and science results be repeated with at least three seeds, with checkpoints chosen on a held-out split rather than on the reported benchmarks?
-
How do entropy, response length, and pass@k evolve on math relative to group-relative RL, including when the contrastive term is applied only to mixed groups?
-
Does the asymmetry in choosing the negative answer matter? Drawing it from the group for incorrect rollouts too would show whether the evidence measures token quality or rollout identity.
-
Differences such as 2.1, 7.7, and 14.7 are percentage points, not percentages, and the text writes them with a percent sign. The relative changes are 3.7, 12.3, and 25.7 percent.
3 comments
Response to Reviewer SzEh (cont.)
Q3. Entropy, response length and pass@k.
Entropy and response length are in the training logs of the math runs, and SIPO tracks GRPO on both:
| Method | Entropy (step 1 / 50 / 125) | Response length in tokens (step 1 / 50 / 125) |
|---|---|---|
| GRPO | 0.339 / 0.339 / 0.353 | 2,191 / 3,712 / 5,203 |
| SIPO | 0.353 / 0.357 / 0.369 | 2,047 / 3,902 / 5,194 |
| SIPO, mixed groups only | 0.330 / 0.357 / 0.363 | 2,170 / 4,192 / 4,876 |
Entropy stays between 0.33 and 0.37 for all three runs, and SIPO's is above GRPO's at each of the three steps. Response length grows from about 2,100 to about 5,000 tokens for all three. None shows the collapse of SDPO in Fig. 4, where reward, entropy and response length all fall. The mixed-groups-only variant is the ablation in G2, which Reviewer qg5m also asks for in Q1. We do not have pass@k on math; entropy, which measures sampling diversity more directly, shows no decline there. On science QA and tool use we do have pass@16. It peaks early in training for every method (within the first hour in all but one case), so we report its best value within 5h:
| Method | Chemistry | Physics | Biology | Materials | Tool use | Avg. |
|---|---|---|---|---|---|---|
| GRPO | 88.1 | 93.3 | 84.1 | 84.7 | 69.2 | 83.9 |
| SDPO | 90.2 | 92.7 | 85.9 | 83.1 | 71.0 | 84.6 |
| SIPO | 90.9 | 94.8 | 84.7 | 84.4 | 70.3 | 85.0 |
SIPO's best pass@16 is at or above GRPO's on four of the five tasks and on average. On the concern behind this request, G4 shows that the update on failed groups is not negative on net.
Q5. Percentage points.
Thank you. We will write absolute differences in points, for example 2.1 points (3.7% relative).
Discussion Reminder
We thank you again for your thoughtful review. With the discussion period ending soon, we wanted to check if our individual response and the General Response (G1–G4) have addressed your concerns. Please let us know if anything remains unclear, we would be happy to provide further clarification before the deadline.
Response to Reviewer SzEh
Thank you for the careful review.
W1 / Q2. Uncertainty, seeds and checkpoint selection.
We keep the checkpoint rule of Hübotter et al. (2026), which keeps every method directly comparable with the baselines and with prior work (G1), and the math and science comparisons are now repeated with 3 seeds (G1). On math SIPO averages 58.7 ± 0.9 against 57.7 ± 0.4 for GRPO, and on Chemistry and Biology 66.4 ± 0.8 against 65.3 ± 0.5 for SDPO. On the specifics:
AIME. We respectfully note that these numbers are avg@32 (960 samples each), so the decoding noise is at most 1.6 points. The 5 to 7 point figure is the uncertainty over the choice of problems, which is shared when methods are compared on the same 30 problems. Over 3 seeds, SIPO averages 55.4 and 40.8 on the two sets against 54.5 and 40.4 for GRPO (G1), so the direction holds on average. What AIME does resolve are the large effects: SDPO and the analytic-KL variant fall 15 to 29 points below GRPO, and the two contrastive variants are the only self-teacher methods that stay at GRPO's level on both sets.
Table 4. We read this chain together with the new results in G1 and G2. For the first step, the case for the contrast rests on large effects: single-teacher variants either collapse (14.7 points below GRPO for the analytic KL, 12.4 for SDPO) or stall while the teacher is active (RLSD, Fig. 5). Note that this step replaces the single-teacher evidence with the contrast and, with it, keeps the teacher weight at 0.5 instead of decaying it to zero. For the last step, the ablation in G2 isolates the effect of uniformly failed groups: including them raises the math average from 58.3 to 58.7 over 3 seeds, with a higher mean on five of the six benchmarks.
Science and tool use. The seeds in G1 show that single cells move by several points across seeds, which also applies to the cells you point to. On Biology, SIPO averages 49.6 ± 3.5 at 1h, level with GRPO's 49.8, and 60.6 ± 3.4 at 5h. Against SDPO, SIPO is ahead on the four-cell average (66.4 ± 0.8 vs. 65.3 ± 0.5) and, unlike SDPO, does not collapse on math (58.7 vs. 44.9).
More generally, the margins you single out are against the best baseline in each domain, and that baseline changes. Each baseline trails SIPO somewhere by 7 points or more: SDPO by about 14 on math, GRPO by 7.7 on science/tool use and 15.1 on code, and RLSD by 7.0 on science/tool use.
W2 / Q4. The asymmetric negative context.
We respectfully disagree that failed groups receive only a negative update. The evidence is a signed, per-token contrast: a token gains credit when the teacher finds it more likely given the reference answer than given the rollout's own wrong answer, and loses credit in the opposite case. A partially correct trajectory is therefore not pushed down as a whole. Tokens that agree with the reference are reinforced, tokens specific to the mistake are penalized, and tokens on which the two contexts agree receive nothing. The training logs bear this out (G4). In uniformly failed groups, 30% to 40% of tokens receive a non-zero teacher term and the rest receive essentially none, and the signed mean of the teacher term is within a few percent of its RMS and positive at most steps, so positive and negative credit are of comparable size. This is what we mean by dense credit. The update is also small: at most 0.1 per token, below the 1/8 that GRPO already assigns to every token of a failed rollout in a mixed group with one success. The construction is hindsight relabeling, as Sec. 3.2 already states.
The asymmetry follows one rule: contrast each rollout with the most relevant mistake. An incorrect rollout is contrasted with the mistake it actually made (hindsight relabeling, Sec. 3.2); a correct rollout has none, so it takes the group's most common one. We arrived at this pairing empirically in early experiments on science QA and tool use, before the math experiments. Contrasting an incorrect rollout with a mistake it did not make breaks the link to its own outcome. When we substituted the group's most common wrong answer for rollouts without a judged answer, the advantage on uniformly failed groups grew steadily positive as truncated responses became more frequent (App. A.1). With the current pairing it stays near zero (G4). We will add this discussion to Sec. 3.2.
Q1. Group size.
G=8 in all experiments (App. B, A.2), so κδ≤0.1<1/8, consistent with your bound G<10. The projection in Eq. (8) enforces sign preservation regardless. We will add such details in the main text.