acceptodds
Under review as a conference paper at ICLR 2027

From Privilege to Parameters: Learning Dynamics of On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) has recently emerged as a promising post-training paradigm for improving language models. A teacher conditioned on privileged information, additional context available during training but unavailable at inference, supervises an unprivileged student's predictions at student-generated prefixes. However, how privileged information becomes internalized during training remains poorly understood. We revisit OPSD from the perspective of privilege transmission, combining a local learning-dynamics analysis with controlled random key–value recall experiments. We find that OPSD internalizes information through behavioral learning: privileged information is acquired only through the supervision it induces on the behavior the student is asked to produce. Internalization further exhibits a prefix-first progression aligned with the structure of autoregressive generation. These findings motivate **Recontextualizing Privileged Information**, a simple yet effective strategy that reuses privileged examples across task contexts while retaining task-local reference guidance, allowing information associated with one task to contribute to supervision on others. Experiments with Qwen3 models ranging from 1.7B to 8B parameters demonstrate improvements over vanilla OPSD, with gains of up to **17.61 percentage points** in best-checkpoint mean Avg@12 across eight mathematical reasoning benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Self-Distillation, Learning Dynamics, Knowledge Distillation, Language Model Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4snkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces RePI a variand of self-distillation that uses other candidates of privileged information to improve the efficacy of self-distillation.

Critical strengths

The paper's motivation and idea seems good I liked the principle of studying how/where privielged distilattion is digested by the model. The results look good I like the pass@k results. Overall I don't have a problem with the paper's results or claims. I think I like the synthetic experiment if it means what I think it does.

Critical weaknesses

The paper is basically unreadable, I have no way to confirm but the paper reads as if it was written with heavy ai assistance. I found myself confused and while the sentences make sense the effort to digest what the authors are trying to communicate are significant. Section 3 for instance is trying to motivate why we need all this terms and theory but that portion is really hard to understand. I don't have gripes with this method or idea I think its neat, but the presentation needs significant improvement.

My rating is fairly dependant on the presentation if you guys can convince me that you will make substantial improvements and show me a better written draft I am confident I understand I am happy to move the rating. Again I like the idea behind the paper and makes sense.

Questions for authors

I don't have many gripes but I also don't quite know if I understood properly what was being presented. I think I did, I could nitpick a few things like having a single seed or that the experiements where only ran for 200 steps, but I think these are minor for a proof of concept. I'd like to see a multi turn setting for instance, just run this on textworld or alfworld, its cheap to run anyways.

Given the heavy importance given to privileged information this citation seems very relevant and should be added

Privileged Information distillation (https://arxiv.org/pdf/2602.04942)

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4snk

We sincerely thank you for the candid review, and for your positive view of the motivation, idea and results. We respond to each point below, and we would welcome further discussion.

W1: Presentation and Section 3

The paper is basically unreadable [...] Section 3 for instance is trying to motivate why we need all this terms and theory but that portion is really hard to understand.

We agree that readability was the main problem with the submission. We have revised the writing throughout the main text, with the largest changes in Section 3. Section 3 now opens with the question it answers, names the student and teacher roles explicitly when defining OPSD, states in plain words what Proposition 1 implies, and ends by explaining how the finding leads to RePI. As disclosed in our AI Use Statement, we used AI tools when drafting and editing the paper. The submitted text was hard to read, and that is our responsibility.

So that you can check your reading against ours, here is what Section 3 and the method say:

  • The question. In OPSD, one model plays two roles. As the teacher, it sees privileged information (e.g., a reference solution) and provides next-token targets on the response the student itself generated. As the student, it never sees that information. Section 3 asks which information in the teacher's context the student actually learns.
  • The synthetic experiment. At every step the teacher sees the same table of 15 random keys, each mapped to a six-digit value. Each training query asks the student to list the values of 3 keys and add them, and we control how often each key is queried. Keys that are never queried are recalled at 0%, although the teacher sees them at every step, whereas the most frequently queried keys reach 81.7%. Being present in the teacher's context is not enough. Information is learned through the supervision it induces on the outputs the student is asked to produce, and the more often it is used, the better it is learned. Proposition 1 gives the corresponding one-step explanation.
  • The method. In OPSD, each reference solution supervises the student only on its own problem. RePI adds nine randomly drawn training problems with their solutions to the teacher's context and retains the current reference solution. The student input and the training objective are unchanged.

If this matches your reading of the synthetic experiment, it is what the revised Section 3 is written to convey.

Q1: Multi-turn settings

I'd like to see a multi turn setting for instance, just run this on textworld or alfworld

Thank you for the suggestion. We ran Base, OPSD and RePI on the Coin Collector task of TextWorld (multi-room navigation to find and pick up a coin) with Qwen3-4B. At each turn the student chooses among the admissible actions. The teacher additionally sees the expert walkthrough for the current game, and RePI adds the walkthroughs of nine other training games. We train on 256 games and report success rate on 64 held-out games.

MethodSuccess rate (%)Gain over Base (points)Relative gain
Base48.4––
OPSD71.9+23.5+48.6%
RePI75.0+26.6+55.0%

Both methods improve over Base, and RePI again outperforms OPSD (75.0% vs. 71.9%).

Q2: Related work

Given the heavy importance given to privileged information this citation seems very relevant and should be added

Thank you for the pointer. We have added this work to Related Work. Penaloza et al. [1] distill privileged information in multi-turn agentic tasks (τ-Bench, TravelPlanner). They introduce π-Distill and an on-policy self-distillation objective, and find that for the latter the information content of the privileged information is the strongest predictor of transfer. The two works are complementary: [1] examines which privileged information helps transfer, while we use a controlled experiment to examine at which student states privileged information induces supervision, and use this to choose the teacher's context.

We have uploaded a revised version of the paper, which includes the revised text, the multi-turn experiment (Appendix C.8) and the citation above (Section 2), with added and revised text shown in blue.

Reference

[1] Penaloza, E. et al. Privileged Information Distillation for Language Models. arXiv:2602.04942, 2026.

Reviewer_4snkOpenReview ↗

response

Thanks for the authors for addressing some of my points.

There are a substantial amount of edits to the manuscript and the paper reads completely differently. I still have concerns on the writing to be honest. The text reads heavily AI assisted and is often over verbose making it really hard to parse. I do think it is better tho.

Could the authors add the training curves as well to the appendix for everything (some are already there) including the newe text-world experiments?

Also some appendices e.g. (C.1 LEARNING DYNAMICS WITH GRPO are empty...).

Again I'm not opposed to this but I am not extremely happy about the presentation. I find the synthetic experiment still really hard to parse and don't think I understand it fully. Also figure 2 which is supposed to explain the method really does not do a good job at it and is really confusing.

I will maintain my rating for now that while the authors did improve the quality of the submission I think there remain substantial clarity issues with the manuscript.

If other reviewers are happy about the submission I am not against acceptance but am wary as the current version of the paper still heavily suffers from presentation issues.

AuthorsOpenReview ↗

Reply to Reviewer 4snk

Thank you for the candid feedback. We have focused this revision on the specific presentation issues you identified.

Training curves and appendices. We added the missing mathematics learning curves (Appendix C.9, Figure 10), which compare OPSD, RePI, and OP²SD at all three model scales and add the non-thinking model comparison. The TextWorld experiment was added during this discussion period as a smaller multi-turn check, and Appendix C.8 reports held-out success at the final checkpoint; we will extend it with the intermediate evaluations so that it matches the other experiments. We also added explanatory text to C.1 and C.2, shortened their figure captions, and corrected the appendix pagination.

Synthetic experiment. Section 3.3 now states the question and intervention directly: the teacher always sees all mappings, while we vary how often each key is requested in training queries. We clarified that evaluation uses held-out combinations of the same keys without privileged context, and scores the individual key–value assignments rather than the final sum. We also shortened the transition to RePI, distinguishing what the experiment establishes from the method it motivates.

Figure 2. We revised the accompanying explanation in Section 3.1 to guide readers through panel (a): student generation, teacher supervision at the same prefixes, and updates to shared parameters. We separately explain that panel (b) illustrates an observed acquisition pattern, not an additional training step.

Thank you again for reading the revision carefully. Your comments helped us identify where the explanation needed improvement, and we will continue to make the writing clearer and more concise.

Dear Reviewer 4snk,

Thank you again for your time and effort in reviewing our submission.

As the discussion period will end in 24 hours, we would like to kindly remind you that we have posted our responses to your latest comments. If you have any further questions or concerns, we would be happy to discuss them.

We sincerely appreciate your constructive feedback and thank you again for your time and consideration.

Reviewer_Vm28OpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

They look at how on-policy self-distillation transfers information from a teacher’s context into a student’s parameters. Their central claim is that the student acquires privileged information through the supervision it induces on student-generated responses. They show that making information available to the teacher does not guarantee that the student learns it. In a controlled experiment, the teacher always sees fifteen random key–value mappings, but the student learns frequently queried mappings more accurately and fails to recall mappings never queried during training.

They introduce RePI, which is a normal on-policy self-distilation method except the fact that the teacher sees current problem’s reference solution plus "several other problems’ worked solutions". Across eight mathematics benchmarks, RePI improves best-checkpoint mean Avg@12 over vanilla OPSD by 3.07, 17.61, and 10.10 percentage points for Qwen3-1.7B, 4B, and 8B, respectively.

Critical strengths
  1. The theoretical results and the intuition that putting text in the teacher's context does not necessarily transfer it to the model are interesting.
  2. Interesting task design for the key–value mapping task. It removed the effect of possible data contamination.
Critical weaknesses
  1. The paper is hard to understand. Like explaining the fact that the method is adding some random questions with referense answer to the teacher context compared to normal OPSD make it easier to understand.
  2. The reason the method improves performance so much is unclear. The toy key–value experiment is about factual knowledge, but the math experiments are about reasoning. So how the findings transfer is unclear to me.
Questions for authors
  1. One baseline you compared against is OP^2SD, which provides the teacher with one question–reference pair but not the reference for the current problem. However, it differs from your method in two ways: you include nine other samples and also the current problem's reference. Can you do an ablation on these two factors, such as including nine other samples but not the current problem's reference?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Vm28

We sincerely thank you for a careful review, and for finding the theoretical results and the key–value task design interesting. We respond to each of your concerns below, and we would welcome further discussion.

W1: Clarity of the presentation

The paper is hard to understand. Like explaining the fact that the method is adding some random questions with reference answers to the teacher context compared to normal OPSD make it easier to understand.

Thank you for the concrete suggestion. We agree that this simple description should be more prominent. The abstract, the introduction and the opening of Section 4 now state it directly: RePI adds random question–solution pairs to the teacher's context while retaining the current reference solution. The student input and training objective remain unchanged. Section 3 motivates this change and Section 4 evaluates it.

W2: Why the method helps, and how the findings transfer to reasoning

The reason the method improves performance so much is unclear. The toy key–value experiment is about factual knowledge, but the math experiments are about reasoning. So how the findings transfer is unclear to me.

In both tasks, privileged information affects the student only through the teacher's feedback on the student's own responses. The key–value experiment isolates this process, and RePI applies it to mathematics.

  • (1) The supervision path is the same. By Proposition 1, privileged information changes the student only through the difference it induces in the teacher's prediction on the student's own responses, whether the information is a stored value or a reasoning step.
  • (2) Why RePI helps. In OPSD, each reference solution is provided only when its own problem is sampled. RePI also places it in the teacher's context for other problems, giving each reference a wider range of problem contexts in which it can shape supervision.
  • (3) The math results are consistent with this picture. On Qwen3-4B, mean Avg@12 rises from 24.91% with no other examples to 51.76% with eight and 51.57% with nine (Table 2). The student's solutions also move toward the structure of the reference solutions (headings, step markers, notation), with alignment rising from −0.26 under OPSD to 0.58–0.67 with eight to sixteen examples (Section 4.2.4).

Q1: Disentangling the other examples from the current reference

One baseline you compared against is OP^2SD [...] Can you do an ablation on these two factors, such as including nine other samples but not the current problem's reference?

We added two conditions without the current reference solution, with one and with nine other examples. All runs use Qwen3-4B, the RePI teacher template, and the same training hyperparameters and evaluation protocol. We report Avg@12 (Appendix C.6, Table 8).

MethodCurrent ref.Other examplesAIME 2024AIME 2025HMMT Feb. 25Mean
OPSD✓035.0023.0616.6724.91
RePI (m=1m=1)✓137.2233.3318.0629.54
w/o current ref. (new)✗116.9413.616.1112.22
RePI (m=9m=9)✓963.6155.8335.2851.57
w/o current ref. (new)✗959.4453.8933.0648.80

Both factors matter, and the role of the current reference depends on the number of other examples.

  • One other example. This is the OP²SD condition in both factors you named. With the RePI template, removing the current reference lowers Avg@12 from 29.54 to 12.22. The OP²SD baseline in Table 1 scores much higher because we ran it with its own teacher prompt, adapted from the original paper, which reminds the teacher to use the example's style rather than its content, and with a longer generation limit (Appendix B.2). Even against this stronger configuration, RePI achieves a higher 8-benchmark mean Avg@12 at all three scales (Table 1).
  • Nine other examples. Removing the current reference gives 48.80, within the range of RePI across three training seeds (44.91–51.57, Appendix C.3). The other examples then account for almost all of the gain over OPSD (24.91 → 51.57), consistent with our analysis that each reference supervises the student in more problem contexts.

We have uploaded a revised version of the paper, which includes this ablation and the revised writing, with added and revised text shown in blue.

Thanks for your response. Your response addressed my concerns with the paper; hence, I'm raising my score.

Dear Reviewer Vm28,

Thank you for your follow-up and for raising your score. We appreciate your careful reading of our work. The ablation you suggested has clarified the source of the gain, and your advice on presentation has made the method easier to follow. Both are now reflected in the revised paper.

Best regards, The Authors

Reviewer_ZZB3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the self-distillation setting and operates under the hypothesis that giving a teacher a reference solution does not guarantee that the corresponding student learns whatever information it contains; a question of interest is the science behind what and how information gets internalized into the student, which the paper examines. The paper proposes Recontextualizing Privileged Information (RePI), which uses in-context examples as the privileged information and trains via full-vocabulary forward KL. Experiments use Qwen3-1.7B, 4B, and 8B as the models of testing for math and Qwen3-4B for the recall test. Notably, thinking is disabled throughout. Shows RePI gains a few percentage points over standard OPSD.

Critical strengths
  • The paper's experimental setup is comprehensive and ablations are done properly.
  • One surprising result is that using random in-context examples for the teacher's PI produces large gains during training even when the teacher already can solve the question. The paper states this properly.
Critical weaknesses
  • (Major) The method is not novel enough to warrant a new name for it; it simply uses in-context examples for similar problems as privileged information.
  • The paper uses thinking models but disables thinking throughout. Is the method truly improving the model, or is it merely teaching the model to think outside of its thinking block (<think> ... </think>)?
Questions for authors
  • See W1. Before and after training, does the model's response become longer? I suspect the training is merely teaching the model to generate longer outputs to "simulate" thinking in its final response.
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZZB3

We sincerely thank you for a careful and constructive review. We respond to both of your concerns below, and we would welcome further discussion.

W1: Novelty and the naming of the method

The method is not novel enough to warrant a new name for it.

Our main contribution is the analysis and empirical characterization of how privileged information becomes expressed in student behavior. RePI is a deliberately simple intervention that translates these findings into a practical training strategy. We agree that the method itself is simple. The novelty we claim lies in the finding that motivates it.

One detail of the method bears on this. The additional problems are sampled at random from the training set, with no similarity selection or retrieval (Section 4.1). This connects to the result you noted as surprising: random examples in the teacher's context produce large gains even when the teacher can already solve the question. RePI is the simplest way to act on this finding.

  • What the name stands for.
    • (1) Mechanism: Privileged information is acquired where it induces supervision, not where it is merely available: in our key–value experiment, mappings the teacher always sees but the student is never queried on are recalled at 0%. "Recontextualizing" names the resulting design dimension, namely reusing privileged information across task contexts.
    • (2) Practice: A one-line change to the teacher's context yields the best overall results among all compared methods, at only 5.8% extra training time. RePI achieves the best mean Pass@12 at all three scales and the best mean Avg@12 at 4B and 8B.
  • Reasons for minimal intervention. We keep the design simple so that the gain can be attributed to the change in the teacher's context alone. It can be extended into a more elaborate method with likely stronger performance.

W2 & Q1: Thinking outside the thinking block

Before and after training, does the model's response become longer? Is the method truly improving the model, or is it merely teaching the model to think outside of its thinking block?

We indeed find that the output becomes substantially longer in our original setting (Qwen3-4B with thinking disabled). To test whether the gain depends on a suppressed thinking mode, we repeat the comparison of Base, OPSD and RePI on Qwen3-4B-Instruct-2507, which supports only non-thinking mode and never generates <think>...</think> blocks [1]. Training data, hyperparameters and evaluation are identical to the Qwen3-4B runs. Length statistics use all 1,080 responses (90 problems × 12 samples).

  • Response length.

    ModelMethodMean tokensCorrectIncorrect
    Qwen3-4BBase3,5471,8083,933
    OPSD7,3472,0749,096
    RePI11,5826,59916,889
    Qwen3-4B-Instruct-2507Base9,4824,63813,673
    OPSD7,3234,38910,108
    RePI8,9575,51712,788

    On Qwen3-4B, mean length rises from 3,547 tokens (Base) to 7,347 (OPSD) and 11,582 (RePI), so in this setting the gain comes with substantially longer outputs. On Qwen3-4B-Instruct-2507, by contrast, RePI improves over the base model without producing longer outputs (8,957 vs. 9,482 tokens).

  • The gain persists on a non-thinking model.

    MethodAIME 2024AIME 2025HMMT Feb. 25Mean Avg@12Pass@12
    Base60.8346.6731.6746.3954.44
    OPSD63.0652.5030.5648.7054.44
    RePI66.3957.2234.4452.6964.44

    RePI improves over OPSD on every benchmark, by 3.99 points in mean Avg@12 and 10.00 points in Pass@12. Since this model has no suppressed thinking mode to recover, the gain cannot come from re-enabling one.

Together, these results show that RePI's gain does not depend on recovering a suppressed thinking mode. It persists on a model that has none, and there RePI is shorter than the base model.

We have uploaded a revised version of the paper, which includes the experiments and length statistics above (Appendix C.7), with added and revised text shown in blue.

Reference

[1] Qwen Team. Qwen3-4B-Instruct-2507 model card. https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507

Thank you for your response to the review.

  • Regarding W1: Thank you for the clarification. If I understand correctly, then, the main finding of the paper is that a random sample as the privileged information still works. However, instead of proposing RePI, a result that would be more worth of a conference publication would be to better understand exactly why this is the case: Is it a property of the task, or the model not being able to properly use privileged information, or something else? I don't believe I see any explanation for why of this sort.
  • W2 & Q1: RePI working on non-thinking models is not surprising, as OPSD was previously demonstrated to mainly for non-thinking models also. Separately, if training does make the generations substantially longer, I don't see how your current experiment in the rebuttal refutes that this is simply teaching the model to use more inference-time compute.

For the above reasons, I do not believe this submission offers much scientific insight beyond what is already known about self-distilation, and thus I retain my score.

Thank you for taking the time to read our response and for the further comments. We appreciate the opportunity to clarify these points.

W1. We would like to clarify that our main finding is not only that random examples work as privileged information. The paper makes three contributions:

  1. A privilege transmission framework that characterizes how supervision induced by privileged information on student-generated responses propagates through shared parameters.
  2. Controlled evidence that OPSD internalizes information through behavioral learning and in a prefix-first progression.
  3. RePI, which achieves consistent improvements over OPSD across three model scales and mathematical reasoning benchmarks.

Adding to our previous reply, these findings also explain the route through which random examples help.

  • In OPSD, each reference solution appears only when its own problem is sampled, so it provides direct supervision only on the student's responses to that problem.
  • RePI places each reference solution in the teacher's context for m+1 problems on average, so it provides direct supervision on more problems.

Two results are consistent with this account:

  • Ablation (Qwen3-4B, mean Avg@12 over the three competition benchmarks). Removing the current reference and keeping nine other examples gives 48.80, within the range of RePI across three seeds (44.91–51.57). The other examples account for almost all of the gain over OPSD.
  • Behavior (step 200, eight to sixteen other examples). The alignment of student solutions with the structure of the reference solutions rises from −0.26 under OPSD to 0.58–0.67.

W2. We appreciate this question. Longer responses on Qwen3-4B do not by themselves show that the gain is simply due to more inference-time computation. On Qwen3-4B-Instruct-2507, accuracy and average output length move in opposite directions (Appendix C.7): RePI improves mean Avg@12 over Base from 46.39 to 52.69 while reducing average response length from 9,482 to 8,957 tokens. On this model, the gain over Base is not explained by an increase in average output length.

We hope this addresses your concerns. If you have any further questions, we would be glad to discuss them.

Dear Reviewer ZZB3,

Thank you again for your time and effort in reviewing our submission.

As the discussion period will end in 24 hours, we would like to kindly remind you that we have posted our responses to your latest comments. If you have any further questions or concerns, we would be happy to discuss them.

We sincerely appreciate your constructive feedback and thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.