acceptodds
Under review as a conference paper at ICLR 2027

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for improving large language models (LLMs) on various tasks, yet its sparse outcome rewards lack token-level credit assignment for intermediate steps. To address this, on-policy self-distillation (OPSD) leverages a self-teacher with privileged context to provide additional dense learning signals. However, because the self-teacher is often overconfident and imposes excessive penalties on long reasoning trajectories, OPSD frequently struggles in practice. To mitigate this, we propose self-instructing policy optimization () with a contrastive self-teacher to provide dense credit. At each iteration, samples multiple rollouts per prompt from the current policy, scores them with environment rewards, and constructs two teacher contexts for each rollout by pairing the reference answer with mistakes made within the group. The model then re-evaluates its own responses under both contexts, using the difference between the two teacher log-probabilities as token-level feedback, so that biases shared by both contexts are expected to largely cancel. The resulting objective yields a token-level advantage for every rollout: the reward still sets the main direction of each update while the self-teacher redistributes credit across tokens. Even in groups where every rollout fails and group-relative advantages vanish, still provides a learning signal. By preserving direct optimization of the task reward while providing dense, token-level feedback, this approach bridges reinforcement learning and on-policy self-distillation. Extensive experiments across multiple reasoning and code-generation benchmarks demonstrate that outperforms both RLVR and OPSD baselines without an external teacher or additional generation.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SzEhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines reinforcement learning from verifiable rewards with on-policy self-distillation through a single token-level advantage. Each rollout is re-scored under two privileged contexts, one containing the reference answer and one a mistake drawn from the same prompt group, and the clipped difference between the two teacher log-probabilities is added to the group-relative advantage. The intent is that biases shared by both contexts cancel, so the reward still sets the direction of each update while the teacher redistributes credit across tokens. Because the contrast needs no successful rollout, groups in which every rollout fails still receive a signal. Experiments cover math, science question answering, tool use, and code generation.

Critical strengths
  1. The diagnosis of why self-distillation fails on long reasoning is the most valuable part of the paper. Figure 4 shows reward, entropy, and response length all falling while the gradient norm grows, and the proposed explanation, that an answer-aware teacher favors continuations heading straight to the answer, is both plausible and testable. Table 4 supports it from the other direction, since adding an analytic reverse KL toward a single teacher costs 14.7 points on the math average while remaining competitive on short-response tasks.

  2. The contrastive construction follows from that diagnosis, and the mechanism claim is checked rather than asserted. Single-teacher evidence has a near-zero median but a consistently negative mean, while the contrastive evidence fluctuates around zero without a consistent sign. The design is also cheap in the right way, adding two teacher forward passes over existing rollouts rather than new generation.

Critical weaknesses
  1. The reported gains are small relative to the uncertainty in the evaluation, and no uncertainty is reported. Every table appears to come from one training run, and checkpoints are selected by accuracy on the evaluation benchmarks themselves. The margins carrying the conclusions are 2.1 points on the math average over group-relative RL and 1.3 points on the science average over self-distillation. On AIME24 and AIME25, which have thirty problems each, the problem-level standard error alone is roughly 5 to 7 points, so the reported 1.9 and 1.1 point gaps are not resolvable, and the 0.1 point gap on AIME24 between the full method and its contrastive ablation certainly is not. The ablation chain in Table 4 moves by 1.6 and then 1.1 points, so the attribution to the contrast and then to failed groups rests on differences of about this size. Self-distillation alone also wins 4 of the 10 science and tool-use cells, and at one hour the proposed method trails group-relative RL on biology by 3.0 points.

  2. The negative context is constructed asymmetrically and the consequence is not analyzed. For an incorrect rollout the negative answer is its own final answer, whereas for a correct rollout it is the most common group mistake. A teacher conditioned on a rollout's own wrong answer will favor precisely the tokens that produced it, so the evidence partly encodes which rollout it is rather than which token deserves credit. In a uniformly failed group the group-relative advantage is zero and this term carries the whole update, so what the method supplies there is a hindsight negative update away from the rollout's own mistake rather than dense credit. Given that negative-only updates are known to destabilize training, I would expect entropy, length, and pass@k curves on math. None are reported.

Questions for authors
  1. What is the group size? With the clip bound at 0.2 and the teacher weight at 0.5, the stated sign-preservation condition requires a group size below 10, and the group size never appears in the main text.

  2. Can the math and science results be repeated with at least three seeds, with checkpoints chosen on a held-out split rather than on the reported benchmarks?

  3. How do entropy, response length, and pass@k evolve on math relative to group-relative RL, including when the contrastive term is applied only to mixed groups?

  4. Does the asymmetry in choosing the negative answer matter? Drawing it from the group for incorrect rollouts too would show whether the evidence measures token quality or rollout identity.

  5. Differences such as 2.1, 7.7, and 14.7 are percentage points, not percentages, and the text writes them with a percent sign. The relative changes are 3.7, 12.3, and 25.7 percent.

3 comments
AuthorsOpenReview ↗

Response to Reviewer SzEh

Thank you for the careful review.

W1 / Q2. Uncertainty, seeds and checkpoint selection.

We keep the checkpoint rule of Hübotter et al. (2026), which keeps every method directly comparable with the baselines and with prior work (G1), and the math and science comparisons are now repeated with 3 seeds (G1). On math SIPO averages 58.7 ± 0.9 against 57.7 ± 0.4 for GRPO, and on Chemistry and Biology 66.4 ± 0.8 against 65.3 ± 0.5 for SDPO. On the specifics:

  • AIME. We respectfully note that these numbers are avg@32 (960 samples each), so the decoding noise is at most 1.6 points. The 5 to 7 point figure is the uncertainty over the choice of problems, which is shared when methods are compared on the same 30 problems. Over 3 seeds, SIPO averages 55.4 and 40.8 on the two sets against 54.5 and 40.4 for GRPO (G1), so the direction holds on average. What AIME does resolve are the large effects: SDPO and the analytic-KL variant fall 15 to 29 points below GRPO, and the two contrastive variants are the only self-teacher methods that stay at GRPO's level on both sets.

  • Table 4. We read this chain together with the new results in G1 and G2. For the first step, the case for the contrast rests on large effects: single-teacher variants either collapse (14.7 points below GRPO for the analytic KL, 12.4 for SDPO) or stall while the teacher is active (RLSD, Fig. 5). Note that this step replaces the single-teacher evidence with the contrast and, with it, keeps the teacher weight at 0.5 instead of decaying it to zero. For the last step, the ablation in G2 isolates the effect of uniformly failed groups: including them raises the math average from 58.3 to 58.7 over 3 seeds, with a higher mean on five of the six benchmarks.

  • Science and tool use. The seeds in G1 show that single cells move by several points across seeds, which also applies to the cells you point to. On Biology, SIPO averages 49.6 ± 3.5 at 1h, level with GRPO's 49.8, and 60.6 ± 3.4 at 5h. Against SDPO, SIPO is ahead on the four-cell average (66.4 ± 0.8 vs. 65.3 ± 0.5) and, unlike SDPO, does not collapse on math (58.7 vs. 44.9).

More generally, the margins you single out are against the best baseline in each domain, and that baseline changes. Each baseline trails SIPO somewhere by 7 points or more: SDPO by about 14 on math, GRPO by 7.7 on science/tool use and 15.1 on code, and RLSD by 7.0 on science/tool use.

W2 / Q4. The asymmetric negative context.

We respectfully disagree that failed groups receive only a negative update. The evidence is a signed, per-token contrast: a token gains credit when the teacher finds it more likely given the reference answer than given the rollout's own wrong answer, and loses credit in the opposite case. A partially correct trajectory is therefore not pushed down as a whole. Tokens that agree with the reference are reinforced, tokens specific to the mistake are penalized, and tokens on which the two contexts agree receive nothing. The training logs bear this out (G4). In uniformly failed groups, 30% to 40% of tokens receive a non-zero teacher term and the rest receive essentially none, and the signed mean of the teacher term is within a few percent of its RMS and positive at most steps, so positive and negative credit are of comparable size. This is what we mean by dense credit. The update is also small: at most 0.1 per token, below the 1/8 that GRPO already assigns to every token of a failed rollout in a mixed group with one success. The construction is hindsight relabeling, as Sec. 3.2 already states.

The asymmetry follows one rule: contrast each rollout with the most relevant mistake. An incorrect rollout is contrasted with the mistake it actually made (hindsight relabeling, Sec. 3.2); a correct rollout has none, so it takes the group's most common one. We arrived at this pairing empirically in early experiments on science QA and tool use, before the math experiments. Contrasting an incorrect rollout with a mistake it did not make breaks the link to its own outcome. When we substituted the group's most common wrong answer for rollouts without a judged answer, the advantage on uniformly failed groups grew steadily positive as truncated responses became more frequent (App. A.1). With the current pairing it stays near zero (G4). We will add this discussion to Sec. 3.2.

Q1. Group size.

G=8G = 8 in all experiments (App. B, A.2), so κδ≤0.1<1/8\kappa\delta \le 0.1 < 1/8, consistent with your bound G<10G < 10. The projection in Eq. (8) enforces sign preservation regardless. We will add such details in the main text.

AuthorsOpenReview ↗

Response to Reviewer SzEh (cont.)

Q3. Entropy, response length and pass@k.

Entropy and response length are in the training logs of the math runs, and SIPO tracks GRPO on both:

MethodEntropy (step 1 / 50 / 125)Response length in tokens (step 1 / 50 / 125)
GRPO0.339 / 0.339 / 0.3532,191 / 3,712 / 5,203
SIPO0.353 / 0.357 / 0.3692,047 / 3,902 / 5,194
SIPO, mixed groups only0.330 / 0.357 / 0.3632,170 / 4,192 / 4,876

Entropy stays between 0.33 and 0.37 for all three runs, and SIPO's is above GRPO's at each of the three steps. Response length grows from about 2,100 to about 5,000 tokens for all three. None shows the collapse of SDPO in Fig. 4, where reward, entropy and response length all fall. The mixed-groups-only variant is the ablation in G2, which Reviewer qg5m also asks for in Q1. We do not have pass@k on math; entropy, which measures sampling diversity more directly, shows no decline there. On science QA and tool use we do have pass@16. It peaks early in training for every method (within the first hour in all but one case), so we report its best value within 5h:

MethodChemistryPhysicsBiologyMaterialsTool useAvg.
GRPO88.193.384.184.769.283.9
SDPO90.292.785.983.171.084.6
SIPO90.994.884.784.470.385.0

SIPO's best pass@16 is at or above GRPO's on four of the five tasks and on average. On the concern behind this request, G4 shows that the update on failed groups is not negative on net.

Q5. Percentage points.

Thank you. We will write absolute differences in points, for example 2.1 points (3.7% relative).

AuthorsOpenReview ↗

Discussion Reminder

We thank you again for your thoughtful review. With the discussion period ending soon, we wanted to check if our individual response and the General Response (G1–G4) have addressed your concerns. Please let us know if anything remains unclear, we would be happy to provide further clarification before the deadline.

Reviewer_qg5mOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SIPO combines group-relative rewards with contrasting teacher scores under correct and incorrect answer contexts. It reuses policy rollouts and can update eligible all-failure groups. Experiments cover math, science, tools, and code, with the strongest evidence in long mathematical reasoning.

Critical strengths
  • All six math benchmarks improve over GRPO, raising the average from 57.3 to 59.4.
  • The method needs no stronger external teacher or additional generation.
  • Off-policy controls, a second science/tool backbone, holdouts, and analytic-hybrid comparisons add useful evidence.
Critical weaknesses
  1. Attribution: The main contrastive baseline changes both advantage construction and group eligibility. It does not isolate SIPO's distinctive learning signal on zero-advantage groups.
  2. Reliability: Best-checkpoint/time-window reporting and task-specific configuration selection leave validation separation and training variability unclear, particularly for the 0.9-point code gain over RLSD and 1.3-point science/tool gain over SDPO.
  3. Interpretation and scope: A near-zero mean contrast does not establish reasoning-step credit. The analytic hybrid is stronger on both science/tool backbones; its stronger code result also uses extra execution feedback. All-failure updates require valid reference/wrong-answer contexts, and code still requires successful and failed siblings.
Questions for authors
  1. Can you disable only the teacher term on zero-advantage groups, keeping the remaining final-method settings fixed? What fraction of failed groups is eligible?
  2. Which validation data selected configurations and checkpoints? Please repeat the principal comparisons with independent seeds.
  3. Does final-answer-only feedback retain the gain? Please interpret the long-math advantage alongside the hybrid results and teacher-scoring cost.
3 comments
AuthorsOpenReview ↗

Response to Reviewer qg5m

We thank you for the detailed and constructive review.

W1 / Q1. Attribution on zero-advantage groups; eligibility.

G2 reports the ablation you describe, which Reviewer SzEh also asks for in Q3. It reaches 58.3 on the math average, the same as RLSD w/ contrastive, so the additive construction loses nothing on mixed groups. The full method, which also learns from uniformly failed groups, reaches 58.7 ± 0.9 over 3 seeds, a further gain of 0.4 points on average.

On eligibility: uniformly failed groups are 20-30% of all groups early in training on DAPO-Math-17k and about 10% at the end. By design, all of them are eligible on math, science QA and tool use: every rollout whose final answer the verifier can extract receives the teacher term, and only rollouts without a parsable answer, such as truncated ones, are skipped (App. A.1). For scale, 16.7% of all rollouts are truncated at the final math step (14.3% for GRPO); we did not log this share within uniformly failed groups. G4 reports what these groups receive in practice. On code generation no failed group is eligible due to the lack of reference answer.

W2 / Q2. Selection and seeds.

Every method received a hyperparameter sweep. For every method, the configuration was selected per task on the test metric with no separate validation split. For checkpoints, we reported the best one for math, and the best avg@16 within 1h and 5h for science QA and tool use, following the protocol of [1] so that all methods stay directly comparable. SIPO's own search was two values of κ\kappa with δ\delta fixed. The math and science seeds are reported in G1, and we will list the details of our configuration in our revision.

Over 3 seeds, SIPO stays above GRPO on math (58.7 ± 0.9 vs. 57.7 ± 0.4) and above SDPO on the two science tasks (66.4 ± 0.8 vs. 65.3 ± 0.5) (G1). On code, SIPO and RLSD are close (56.3 vs. 55.4), and both are well above GRPO (41.2) and SDPO (48.8). Beyond these margins, our claim rests on larger gaps. The strongest baseline differs by domain, and each baseline trails SIPO in at least one domain by 7 points or more: GRPO by 7.7 on science/tool use and 15.1 on code, SDPO by about 14 on math and 7.5 on code, and RLSD by 7.0 on science/tool use.

W3a / Q3. Near-zero mean and reasoning-step credit; final-answer-only feedback.

The mean alone does not establish step-level credit, and we would like to clarify that we do not use it for that. In Sec. 4 it supports a narrower claim: the contrast removes the systematic negative offset of the single teacher, which is the failure mode we link to the collapse on long reasoning (Fig. 4, Table 4). We believe that claim stands.

On whether the signal reduces to final-answer feedback: in mixed groups it cannot dominate. The evidence is clipped at δ\delta per token, so an answer span of a few tokens receives at most κδ≤0.1\kappa\delta \le 0.1 per token, while every one of the thousands of tokens in a math response already carries a reward advantage of at least 1/81/8. In uniformly failed groups this comparison does not apply, because the reward advantage is zero and the teacher term is the whole update, so we measured it directly (G4). There, at least 30% of tokens carry non-zero evidence (∣Ai,t∣>10−3| A_{i,t} | > 10^{-3}) throughout training, and the answer span can account for under 30% of the total teacher term, so the signal is not confined to the final answer.

W3b / Q3. The analytic hybrid and teacher-scoring cost.

The hybrid is the baseline described in G3. On short responses it is comparable or somewhat stronger: 71.2 vs. 70.4 (Qwen3-8B) and 69.7 vs. 66.5 (Olmo3-7B) on science/tool use, and 58.5 vs. 56.3 on LCBv6, where it additionally uses execution feedback. On long reasoning it collapses: 42.6 on math, against 57.3 for GRPO and 59.4 for SIPO (Table 4). SIPO remains strong in both regimes, which is why we adopt it.

On cost, SIPO adds two gradient-free teacher forward passes over existing rollouts and no generation. The science/tool use results are at equal wall-clock time, so that cost is already charged. The math results are at equal steps: on 4 NVIDIA H200 GPUs, with 128 prompts and 8 rollouts per step, a step takes 605.5s for SIPO against 598.8s for GRPO, an overhead of about 1% (rollout generation remains the bottleneck). Teacher scoring is a batched forward pass over existing tokens, which is cheap next to generating 1,024 rollouts autoregressively. We will add these timings to App. B.

W3c. Scope of all-failure updates.

These conditions are stated in Sec. 3.2, Sec. 4 and App. A.1: learning from uniformly failed groups needs a reference answer and a parsable wrong answer, and on code SIPO learns from mixed groups only. We will also state them in the introduction.

[1] Hübotter, Jonas, et al. "Reinforcement learning via self-distillation." arXiv preprint arXiv:2601.20802 (2026).

Thank you for the detailed response; G1–G4 address much of my review.

G2 provides the mixed-groups-only control I requested: disabling the teacher term whenever A^i=0\hat A_i=0, with the other settings fixed, isolates the contribution of uniformly failed groups. The eligibility, scope, analytic-hybrid, and timing clarifications are also useful. I appreciate the explicit acknowledgement that a near-zero mean contrast supports offset cancellation rather than reasoning-step credit by itself.

My principal remaining concern is evaluation robustness.

The mixed-groups-only result is one run (58.3), whereas the full-method result is a three-seed mean (58.7 ±\pm 0.9), so their 0.4-point difference is not a paired comparison. The science repetitions cover only Chemistry and Biology, and the 0.9-point code margin over RLSD remains without independent seeds. The response also confirms that configurations and checkpoints were selected on the test metric without a separate validation split. Applying this protocol equally across methods aids comparability, but does not remove adaptive test-set selection. Finally, G4 provides useful token-level diagnostics, but not a final-answer-only performance control. Could you provide paired per-seed results for full SIPO versus mixed-groups-only SIPO and, if available, independent code repetitions? For the revision, please clarify whether model and checkpoint selection can be separated from the reported evaluation; otherwise, please retain test-based selection as an explicit limitation.

Nevertheless, I will raise my score to 3 as the authors addressed most of my comments.

Thank you for the follow-up and for raising your score. We cannot complete further training runs within the discussion period, so below we report what the existing runs support and state what the revision will add.

Paired comparison for the mixed-groups-only ablation. The per-seed math averages of the full method are 59.4, 59.0 and 57.8. The ablation shares its seed with the first of these, which gives one paired comparison:

Same seedAIME24AIME25AMC23MATH500MinervaOlympiadAvg.
SIPO56.841.487.885.832.053.059.4
SIPO, mixed groups only54.741.386.384.830.552.258.3

On this seed the full method is at or above the ablation on all six benchmarks, by 1.1 points on average. The revision will add the ablation on the other two seeds, so that the comparison is paired on all three, and will report a paired significance test over problems.

Code repetitions. We do not have independent repetitions on LCBv6 yet, and the revision will add them for SIPO and RLSD. The conclusion we draw from the code experiment does not depend on the 0.9-point margin between these two. It is that SIPO improves over GRPO by 15.1 points (56.3 vs. 41.2) and over SDPO by 7.5 points (56.3 vs. 48.8). These gaps are several times larger than the seed-to-seed variation we measured on math and science, where the standard deviation of the averages is below 1 point.

Science repetitions. We chose the two tasks so that the repetition would not be biased toward SIPO, and fixed them before running the seeds. Chemistry is the task used throughout our analysis figures, and Biology is the task where the single-run result was least favorable to SIPO, the only one where it trails GRPO at 1h.

Selection protocol. The reported runs do not allow us to separate selection from evaluation, since no validation split was held out. As you ask, the revision will state the protocol explicitly in the experimental setup and retain test-based selection of configurations and checkpoints as an explicit limitation. For SIPO, this selection is over two values of κ\kappa per task.

Final-answer-only control. G4 is a diagnostic and not a performance control, as you say. What it tells us is how much of the teacher signal a final-answer-only variant would keep. In uniformly failed groups, where the teacher signal is the whole update, the bound in G4 puts more than 70% of it on tokens outside the final answer, so such a variant would discard most of it. In mixed groups the answer is a handful of tokens out of thousands, so restricting the teacher signal to them would leave the update almost identical to GRPO's. The revision will add the performance control itself.

Thank you again for the careful reading and the constructive suggestions, which have helped us make the evaluation more rigorous.

Reviewer_tYBgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SIPO, a straightforward approach to combine GRPO with OPSD by merging sequence level advantage with token level feedback from a contrastive teacher. This densifies the reward and enables learning even when all generated answers within the GRPO group fail. Experiments on reasoning benchmarks show that it improves on both RLVR and OPSD baslines.

Critical strengths

The paper has the merit to present good empirical results across a variety of reasoning benchmarks, improving over GRPO and staying competitive with self distillation baselines.

Critical weaknesses
  • The paper does a poor job justifying the choice of the construction, with a motivation that is purely heuristic: let us combine OPSD with RLVR to get the best of both worlds.

  • After reading the paper, I found limited insight explaining why the method works. In my view, I see the contribution as an engineering contribution around reward densification or shaping, rather than an advance in our theoretical understanding/intuition.

  • With these contributions, I find it easier to improve over a baseline method when the proposed method includes that baseline as an instantiation, for example here putting κ=0\kappa = 0 gets us the RLVR loss.

Questions for authors
  • Why don't we just purely combine OPSD loss and an RLVR loss? What justifies the contrastive formulation?
  • Maybe this depends on a subjective research taste but what motivates really the approach from a theoretical point of view? Isn't this just a form of reward shaping heuristic?

Minor error:

  • All losses need to decouple the parameters of the generating policies from the parameter used on the score function. In general, we use y∼πθold(⋅∣x)y \sim \pi_{\theta_{old}}(\cdot \mid x) to make that difference.
3 comments
AuthorsOpenReview ↗

Response to Reviewer tYBg

We thank you for the constructive review.

W1 / Q1. Why not simply add an OPSD loss to the RLVR loss? What justifies the contrast?

This is exactly the baseline you describe, and the paper evaluates it as “GRPO w/ analytic KL” (Table 4, Tables 7-8). The contrast follows from how it fails:

  • The plain sum works on short responses and collapses on long reasoning. It reaches 71.2 on science/tool use with Qwen3-8B and 69.7 with Olmo3-7B, against 70.4 and 66.5 for SIPO. On math it falls to 42.6, which is 14.7 points below GRPO (57.3) and near the base model (40.8).

  • The failure comes with the single answer-aware teacher, however it is combined with the reward. Pure self-distillation, with no reward term, collapses in the same way (Fig. 4). RLSD, which rescales the reward advantage, stalls while its teacher is active (Fig. 5). On math the plain sum differs from SIPO in two further respects, sibling rollouts as teacher context and a full-distribution KL (App. C.4-C.5), but RLSD shares neither and still stalls: its teacher sees the reference answer, as SIPO's positive teacher does, and it uses token-level evidence. This is likely due to the self-teacher model's overconfidence when given privileged context, which leads to shorter yet more certain incorrect responses, as discussed in our paper and [1].

  • The contrast removes this bias. Both teacher contexts carry an answer, so their shared preference cancels and only the answer-specific evidence remains (W2 makes this exact).

  • With the contrast, the failure disappears. Within RLSD, replacing the single-teacher evidence with the contrast raises the math average from 56.7 to 58.3 (Table 4). It also allows the teacher weight to stay at 0.5 instead of decaying to zero, which the single teacher does not, since it stalls until its weight reaches zero. SIPO reaches 58.7 ± 0.9 on math over 3 seeds (G1) and stays strong on short responses.

To make this chain more visible, we will move it to the start of Sec. 3, and rename the row “GRPO + OPSD loss” with its teacher configuration stated in the caption.

W2 / Q2. Theoretical motivation; is this just a reward-shaping heuristic?

The teacher term can be viewed as shaping, but it is not an arbitrary one: it optimizes a well-defined objective, which we can state exactly. Let π+\pi^{+} and π−\pi^{-} be the teacher's next-token distributions under the two contexts. Ignoring clipping, the expected gradient of the teacher term at each position is

  • Ey∼πθ[e(y)∇θlog⁡πθ(y)]=−∇θ[DKL(πθ∥π+)−DKL(πθ∥π−)]\mathbb E_{y \sim \pi_\theta}\left[ e(y) \nabla_\theta \log \pi_\theta(y) \right] = - \nabla_\theta \left[ D_{\mathrm{KL}}(\pi_\theta \Vert \pi^{+}) - D_{\mathrm{KL}}(\pi_\theta \Vert \pi^{-}) \right]

So SIPO descends a difference of two KL terms, where OPSD descends DKL(πθ∥π+)D_{\mathrm{KL}}(\pi_\theta \Vert \pi^{+}) alone. The OPSD gradient is non-zero wherever the student differs from the answer-aware teacher, including shifts that the privileged context induces whichever answer it holds, such as heading to the answer sooner. The SIPO gradient is exactly zero wherever the two teachers agree. This is the precise sense in which shared biases cancel. It also matches the statistic in Sec. 4. The single-teacher evidence has mean −DKL(πθ∥π+)≤0-D_{\mathrm{KL}}(\pi_\theta \Vert \pi^{+}) \le 0 (measured: −0.002 to −0.004 per token), while the contrast has no fixed sign (measured: within ±0.001). In addition, App. A.2 guarantees that with κδ<1/G\kappa\delta < 1/G the teacher term does not flip the sign of a non-zero reward advantage. Together with the diagnosis of why a single answer-aware teacher fails on long reasoning (W1), this objective is the insight we add beyond the empirical results. We will add this derivation to Sec. 3.

W3. κ=0\kappa = 0 recovers GRPO, which makes improving on it easier.

We agree that SIPO contains GRPO as the κ=0\kappa = 0 special case, as do RLSD and GRPO w/ analytic KL. Two facts limit how much this helps. First, the added search space is small: δ\delta is fixed at 0.2 everywhere and κ\kappa takes one of two values, while every baseline has its own sweep (App. B). Second, containing GRPO does not guarantee a gain: RLSD and GRPO w/ analytic KL score 56.7 and 42.6 on math, against 57.3 for GRPO (Table 4).

Minor. Decoupling the sampling parameters

We thank you for the comment. In Eqs. (1), (2) and (6) we will sample from πθold\pi_{\theta_{\mathrm{old}}} and state the clipped importance-weighted surrogate with ratio ρi,t=πθ(yi,t∣x,yi,<t)/πθold(yi,t∣x,yi,<t)\rho_{i,t} = \pi_\theta(y_{i,t} \mid x, y_{i,<t}) / \pi_{\theta_{\mathrm{old}}}(y_{i,t} \mid x, y_{i,<t}). This keeps θ\theta (optimized), θold\theta_{\mathrm{old}} (behavior policy) and θˉ\bar\theta (teacher snapshot) distinct.

[1] Yang, Chenxu, et al. "Self-distilled rlvr." arXiv preprint arXiv:2604.03128 (2026).

AuthorsOpenReview ↗

Discussion Reminder

Thank you again for your review. As the discussion period is drawing to a close, we would like to check whether our response and the General Response (G1-G4) address your concerns. If any point remains unclear, we would be glad to clarify it in the remaining time.

Thank you for your efforts on the rebuttal and really sorry for the late reply.

I appreciate the new results and the additional theoretical discussion. However, I'm not fully convinced with the theoretical argument, for instance, gradients of function evaluated at zero does not mean that the gradient itself is zero, and cancellation where the teachers agree does not establish that a useful signal comes from when they disagree.

The empirical improvements are welcome, but the proposed explanation remains largely heuristic, and my main concern therefore remains unresolved. I cannot recommend acceptance and will maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.