acceptodds
Under review as a conference paper at ICLR 2027

Where to Go, What to Avoid: State-Relative Self-Distillation

Abstract

Existing on-policy self-distillation learns from the student's own rollouts, but its teacher signal remains conditioned on a fixed reference solution. This can entangle useful local reasoning with answer-dependent confidence and preference for a particular solution path. We argue that supervision should instead reflect the student's current rollout state, encouraging viable continuations while discouraging directions that have already led to failure. We therefore propose **State-Relative Self-Distillation (SRSD)**, which incorporates state-relative rollout information into supervision through two complementary teacher roles. A State Tracker constructs a state representation from the student's complete on-policy rollout, providing a positive *Where to go* signal for subsequent reasoning. A State Guard leverages a failed trajectory sampled from the same policy to provide a negative *What to avoid* signal, discouraging the student from revisiting failure-inducing directions. Together, these signals encourage valid alternative reasoning paths while reducing unsupported confidence and rigid path preference. We evaluate SRSD on mathematical reasoning, code generation, and agentic tool use across multiple model scales and benchmarks. SRSD consistently delivers strong performance improvements across these domains, with the largest gains in code generation, where it outperforms the strongest compared method by 8 points. Code will be released upon publication to facilitate reproducibility.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy self-distillation, LLM post-training, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WiruOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

SRSD uses a diagnosis of the student's rollout and a separate failed rollout to construct positive and negative teachers. It trains the student to follow the positive teacher while moving away from the negative one.

Critical strengths
  • The experiments cover mathematics, coding, and tool use across multiple model sizes and families, with substantial reported gains on coding.
  • The paper includes useful ablations on the teacher components, diagnosis model, and conditioning information.
Critical weaknesses
  • I find the main problem this paper aims to address unclear. The "static reference" issue seems to apply to OPSD, whereas SDPO uses students' own outputs. Is this a general problem in self-distillation? Why is state-relative supervision needed, and what exactly does "state-relative" mean? How does it relate to the static-reference problem and the proposed method?
  • All baseline results in the main tables are quoted from prior work. How is comparability guaranteed? This is particularly concerning in Table 2, which combines results from multiple papers without reproducing the key baselines under a common protocol.
  • The objective in Eq. (10) needs further justification. Without clipping, L+−αL−L^+-\alpha L^- can be unbounded below when q+(a)<αq−(a)q^+(a)<\alpha q^-(a). How do the authors ensure stable optimization? In Table 3, the smallest positive α\alpha tested (0.3) performs best, while α=1.0\alpha=1.0 performs worse at every scale. The results seem sensitive to the choice of α\alpha, and it is unclear how they support the proposed principle.
Questions for authors

See weakness section.

Reviewer_AksyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that on-policy self-distillation still conditions the teacher on a fixed reference solution, so the teacher can favor one solution path and miss the student's current rollout state. SRSD keeps that reference, adds a rollout-level diagnosis from a State Tracker as the positive condition, and uses another failed rollout as the negative condition. The student is trained with forward KL toward the positive teacher and away from the negative teacher. On math, code, and tool use, this improves over the reported baselines, with the largest gain on LiveCodeBench.

Critical strengths
  1. The positive and negative conditions are separated cleanly, and the ablations show that they do different jobs: the positive term carries most of the gain, while the negative term helps only as a correction and is harmful alone.

  2. The code experiments separate a stronger static answer from rollout-level diagnosis. A DeepSeek answer used only as a fixed reference does not explain the gain, and the training dynamics show the two teachers becoming more separable while the student moves toward the positive one.

Critical weaknesses
  1. The reported +8 code gain depends on an external diagnoser. Default SRSD calls DeepSeek-V4-Flash on every supervised rollout. On Qwen3-4B LiveCodeBench, this reaches 55.72, versus 47.79 for LOPD. Replacing the diagnoser with the student itself drops the score to 48.73. The abstract and main tables use the external-diagnoser setting.

  2. The method is much more expensive than OPSD, but the cost is not measured. Each step samples up to eight rollouts to find a failure, calls an external model to write the diagnosis, and then runs three full-vocabulary forwards. The positive and negative prompts also contain long auxiliary context. The limitations mention the extra analysis only qualitatively.

  3. The supervision is not the state at the current prefix. The diagnosis is produced once from the completed rollout and reused at every token. On a wrong trajectory, that diagnosis already locates the later error and states the fix, so early-token targets are shaped by how the rollout ends.

  4. The "outperforms all baselines" claim is not supported by matched reruns. Math results are taken from the source papers, and the RLCSD row is incomplete. Code and tool-use numbers come from Zhang et al. (2026). Those runs can differ in data, steps, and compute from SRSD's 200-step LoRA setup. The math margin over the strongest reported baseline is about one point, and the tool-use margin over LOPD is under one point.

Questions for authors
  1. What is SRSD's wall-clock time per step, including rollout generation, the DeepSeek diagnosis, and the three teacher forwards, relative to OPSD and LOPD under the same hardware? Please also report the main code and tool-use tables with the student itself as the State Tracker.

  2. Can the authors rerun LOPD, OPSD, and the strongest math baselines for the same 200 steps, data, and LoRA budget? The current Table 2 mixes numbers from other papers, including an incomplete RLCSD row.

  3. Because z+ is computed from the finished rollout, how much of the positive-teacher signal remains if the diagnosis can use only the prefix available at the scored position? Please also state the exact rule for choosing the supervised rollout y among the sampled trajectories.

  4. Appendix B.3 corrupts the tool contract for the negative teacher, while Section 4 says the State Guard passes the failed rollout through without extra analysis. Is that corruption necessary for the BFCL gains?

Reviewer_85pFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that on-policy self-distillation (OPSD) has on-policy rollouts but a static teacher, because the teacher branch is conditioned on a fixed reference solution. The proposed fix, SRSD, uses two teacher branches of the same frozen base model. The positive teacher sees the reference solution r plus a rollout-level diagnosis z+ of the student's rollout. The diagnosis is written by a "State Tracker" (DeepSeek-V4-Flash by default) using a template chosen by whether the rollout was correct, wrong, or truncated. The negative teacher sees an independently sampled failed rollout z-, placed where the reference solution would normally go. The student (a LoRA adapter) is trained with L+ - alpha L-, where both terms are forward KL from teacher to student over the full vocabulary. Section 4.3 shows that the logit gradient is (q+ - s) - alpha(q- - s), and argues that any component shared by the two teacher residuals is attenuated by a factor of (1 - alpha). The experiments cover math (Qwen3-1.7B/4B/8B trained on an OpenThoughts subset, evaluated on AIME24/25 and HMMT25), code (Qwen3-4B and Olmo3-7B trained on TACO, evaluated on LiveCodeBench v5/v6), and multi-turn tool use (Qwen3-4B/8B, BFCL v3). Baseline numbers are copied from the source papers. The headline result is a 55.72 vs 47.79 LiveCodeBench average over LOPD on Qwen3-4B. The appendix ablates the diagnosis model, the positive-teacher content, alpha/kappa, and a few training hyperparameters, and adds training-dynamics and "pivot token" analyses.

Critical strengths
  1. The appendix ablations are more informative than the headline tables, and they point to a finding worth keeping. If the comparison is apples-to-apples, conditioning a same-model teacher on a failed self-rollout and pushing away from it gives a large gain on code, without needing a stronger external model.

    • On code, "Only r" in Table 5 is OPSD plus the negative teacher. It reaches 52.34 (Qwen3-4B) and 52.07 (Olmo-7B).
    • For comparison, the "OPSD + Static Answer" control in Table 4 gets 39.61 and 45.38, and the copied OPSD numbers are 38.88 and 44.57.
    • This is the part of the paper most likely to be of lasting interest, but the paper currently frames it as secondary.
  2. The method is cheap to try for anyone who already runs OPSD. That lowers the bar for reproduction once code is out.

    • It needs one extra forward pass on a re-prompted context and a sign-flipped KL term.
    • The prompt templates are given in full (Appendix D).
  3. The method and prompts are described concretely; I found Appendix D especially useful. However, several things a reader needs are missing or scattered. It is not clear how the supervised rollout y is chosen among the K samples, how many samples go into LiveCodeBench pass@1, or whether evaluation uses thinking mode. The notation pi_theta is used for the student, both teachers and the rollout distribution at once (Section 3), which made Eqs. (3)-(7) harder to parse than necessary; separate symbols for the student and the two teacher branches would help. The State Tracker also needs more explanation up front: Appendix D.2 says the rollout "is graded" before the diagnosis prompt is chosen, so the Tracker relies on a correctness check against ground truth. This should be stated in Section 4.1, together with whether the Tracker is also what localizes the error (the LOCATE step in its prompt) or whether that comes from somewhere else. Some statements in the text do not match the tables (details below), and Figure 1 contains a marketing-style "+8 points over SOTA in code" label.

Critical weaknesses
  1. The headline comparisons are not controlled, and the parts that are controlled do not clearly support the "state-relative" story. Baselines are copied, a stronger external model supplies the diagnoses, and hyperparameters are tuned on the test benchmarks, so I can't tell how much of the gain comes from the proposed idea.

    • Copied baselines. All baseline numbers in Tables 1 and 2 come from the source papers (Section 5.1; Table 1 caption). These may differ from SRSD's recipe (200 steps, batch 32, LoRA, 1024-token math rollouts with thinking disabled, eval temperature 1.1 with a 38,912-token budget) in data, steps, fine-tuning type and evaluation details.
    • Small margins with no variance. In math SRSD beats the strongest copied baseline by 45.65 vs 44.35 (1.7B), 66.00 vs 65.65 (4B) and 68.61 vs 67.41 (8B), on 30-problem benchmarks with no seeds reported. On BFCL it beats LOPD by 0.87 (4B) and 0.50 (8B); 0.50 points of the 800-item average is 4 tasks. SRSD is also below Base on the 8B Long Context subset (25.00 vs 25.50).
    • External diagnosis model. z+ is written by DeepSeek-V4-Flash, whose prompt asks it to "CORRECT: Describe the right move at that exact point", so a stronger model supplies hints none of the baselines get. The intro's "deviations ... generally within 5%" holds only for Self-Model + Self-Correct (4.1%, 3.2%). Plain Self-Model gets 48.73 and 50.87 vs 55.72 and 55.35 (Table 4), drops of about 12.5% and 8.1% and only 0.94 and 1.26 points above LOPD. This ablation exists only for code.
    • "Where to go" adds little in math. Tracker only (alpha = 0, essentially OPSD plus z+) gets 42.98 at 1.7B, below the copied OPSD's 43.40, and 63.89 vs 63.60 at 4B, below four copied baselines. On code, the biggest jump comes from the negative teacher (Only r vs OPSD: +13.46 and +7.50), with the Tracker adding 3.38 and 3.28 (Table 5). A code version of Table 3 and an in-house OPSD run are missing.
    • Test-set tuning. Appendix A.5: "we use the corresponding best configuration for each reported setting", chosen over six alpha/kappa combinations per model on LiveCodeBench and BFCL. Alpha in math is likewise picked on AIME/HMMT, and the fixed 200 training steps (Table 9) are never justified on a validation split.
    • The "state-relative" framing. z+ is computed once from the finished rollout and "shared across all scored token positions of y" (Section 4.1). So at an early token the teacher already conditions on a diagnosis of an error that happens later. That is hindsight feedback about the whole trajectory, closer to OPSD with a richer privileged context, and it should be positioned against the "conditioning-oriented" OPSD variants cited in Section 2.
  2. The objective and its analysis have gaps the paper doesn't acknowledge. As used by default, the loss is unbounded below, and the theoretical argument for why the contrast helps is true by definition.

    • With beta = 0, L+ - alpha L- equals a constant minus sum_a w(a) log s(a), where w = q+ - alpha q-. Whenever alpha q-(a) > q+(a), w(a) is negative, and the loss goes to minus infinity as s(a) goes to 0.
    • Such tokens do occur: the paper itself clips negative entries of Q*_alpha = (q+ - alpha q-)/(1 - alpha) in Appendix A.2.2. Yet every domain defaults to kappa = 0, i.e. no clipping (Table 9). Presumably only the 200-step schedule, small learning rate and gradient-norm cap of 0.1 keep training stable. "Guard only" falling below Base at every scale (Table 3) fits this picture.
    • The g/b/h decomposition in Eqs. (13)-(16) is definitional. Any pair of residuals can be split into a "shared" part and two specific parts, so "the shared component cancels at alpha = 1" adds no information. The best math alpha is 0.3, which by the paper's own reading keeps 70% of the shared component.
    • Figure 3(b)-(d) measures how much the student agrees with q+ vs q- at disagreement positions, which the loss directly optimizes. Seeing these rise confirms the optimizer works, not that state-relativity causes the accuracy gains.
Questions for authors

Clarifications that could change my assessment:

  1. Did you run OPSD (and ideally LOPD) yourselves under exactly the SRSD recipe, with the same data, LoRA, 200 steps, rollout length, and evaluation settings, in each domain? Was "OPSD + Static Answer" in Table 4 run in your own pipeline? If it was, it is effectively an in-house OPSD, and the jump to "Only r" (52.34) would be the cleanest evidence in the paper. Please state this explicitly and report in-house OPSD for math and tool use too.
  2. Please report Tracker only / Guard only for code and tool use, and Self-Model diagnosis for math and tool use. In particular, how does SRSD with Self-Model diagnosis compare with the best math baselines in Table 2?
  3. How many seeds stand behind each number? Can you give standard deviations or bootstrap intervals, at least for Table 2, Table 3, and the BFCL results? How many samples per task underlie LiveCodeBench pass@1?
  4. How was alpha chosen without looking at test scores? Would the conclusions survive selecting alpha/kappa on a held-out split, for example a slice of TACO or EnvScaler?
  5. In Algorithm 1, how is the supervised rollout y chosen among the K samples: at random, or preferentially failed? What fraction of training rollouts land in each of correct/wrong/trunc? With 1024-token math rollouts and thinking disabled, I would expect many to be truncated. How many questions are skipped because all eight rollouts are correct?
  6. Is evaluation done with thinking enabled or disabled? Training uses thinking disabled with 1024-token rollouts, while evaluation allows 38,912 tokens at temperature 1.1. Do the copied baseline numbers use the same evaluation mode and temperature?
  7. Given that the objective is unbounded below at kappa = 0, did you see any instability, entropy collapse, or growth of L- in longer runs? What happens beyond 200 steps?
  8. For BFCL, the State Guard also corrupts the task contract and tool schema (Appendix B.3), which is a different negative signal from the other domains. How much of the BFCL result depends on the corruption as opposed to the failed rollout?
  9. Appendix A.5 reports Qwen3-8B coding at 60.81/53.43, which averages to 57.12. The best entry for Qwen3-8B in Table 7 is 57.07. Which is correct?
  10. How often is feedback refreshed? As I read it, z+ is produced once per rollout and z- once per question per step. Is the rollout batch regenerated from the current student at every step, or can the scored prefixes come from an earlier policy? Did you try refreshing z+ at intermediate points of the rollout (e.g., per reasoning segment), so the teacher's context reflects the state actually reached at position t? That seems like the natural test of the "state-relative" claim.
  11. In math, Tracker only (alpha = 0) already recovers most of the gain over Base (Table 3). How much does the negative teacher add on top of a well-tuned alpha = 0 run in math, once seeds are accounted for? In other words, is "what to avoid" needed there, or is the state-relative positive teacher enough?
  12. What is the total training cost of SRSD vs OPSD, counting the external diagnosis calls (up to 16,384 new tokens each), up to eight rollouts to find a failure, and three scoring passes?

Minor and textual: 13. Section 5.3 says that increasing alpha beyond 0.3 "consistently degrades performance". At 4B, however, alpha = 1.0 (65.37) beats alpha = 0.6 (65.00). 14. The positive-teacher prompt says the teacher does "NOT see their attempt", but y is then appended as the teacher's own assistant continuation, so the teacher does condition on it as a prefix. Please reword. 15. Related work should discuss contrastive decoding, DExperts-style expert/anti-expert combination, and classifier-free guidance for LMs. The target (q+ - alpha q-)/(1 - alpha) is close to these. Unlikelihood and negative-sample training objectives are also relevant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.