acceptodds
Under review as a conference paper at ICLR 2027

Predicting Privileged Information Effectiveness for On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) enables large language models (LLMs) to learn from privileged information (PI) without a separate teacher model. However, prior work largely treats PI as a fixed design choice, leaving the effects of its content and construction poorly understood. This challenge is particularly important in multi-turn agentic settings, where trajectories contain actions, observations, and reasoning that can be transformed into PI in many ways. We systematically evaluate 14 PI configurations across three multi-turn agentic benchmarks and three student models that vary in scale and reasoning mode. We find that PI effectiveness depends on its content, generation source, and the student model. Information-dense trajectories and PI produced by more capable generators generally yield larger gains, whereas thinking models benefit less consistently. To assess PI before training, we compare PI utility with two KL-based metrics: sampled-token reverse KL and error-step KL contrast. The KL-based metrics track post-training performance more consistently than PI utility, which can be distorted by information leakage. Finally, effective PI remains beneficial when combined with GRPO, allowing the combined objective to outperform a strong GRPO baseline. Overall, our results provide practical guidance for effective PI construction and evaluation within OPSD training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
OPSD, Post Training, Privileged information

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DcqyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies privileged information (PI) for on-policy self-distillation (OPSD), evaluating 14 PI configurations across three agentic benchmarks and three student models. It compares PI formats and generation sources, proposes offline metrics for predicting post-training performance, and evaluates combining OPSD with GRPO.

Critical strengths
  1. PI selection is practically relevant, and assessing PI before training could reduce the cost of choosing training configurations.
  2. The experiments cover multiple PI formats, generation sources, benchmarks, and student models.
Critical weaknesses
  1. The paper reads as a record of combinatorial experiments and does not establish a research contribution sufficient for a top-tier conference. Its main approach is to enumerate combinations of PI formats, generation sources, and student models, report their performance, and provide post-hoc explanations for the observed outcomes. The PI recommendations and proposed metrics lack a convincing theoretical foundation or a clear a priori framework capable of predicting these outcomes. The correlations establish empirical associations, but the paper does not develop them into a substantive, generalizable understanding of PI effectiveness. Extensive experimental coverage alone does not establish the conceptual depth or scientific insight expected of an ICLR research paper.
  2. The presentation is disorganized and difficult to follow. The transitions between OPSD, OPD, and SDPO are insufficiently explained, making the terminology and training formulation hard to track. Several figures are densely packed with small labels, numerous annotations, and repeated legends, obscuring the key comparisons. The text frequently moves from experimental observations to explanatory claims without clearly establishing the evidential link. Together, these issues make it difficult to identify the central argument and assess how strongly the experiments support it.
Questions for authors

What theoretical foundation or a priori framework supports the PI recommendations and proposed metrics? Please explain what predictions this framework makes before observing training outcomes, and what general understanding the paper establishes beyond configuration rankings and correlations.

1 comment

We thank the reviewer for the feedback. We respectfully disagree that the work does not establish a research contribution, and we address each concern below. In response, we added three controlled studies to the revision (changes are marked in blue): ALFWorld-Passcode (Section 5; Appendix E, Figure 6), a corrupted-PI ablation (Section 5; Appendix F, Figure 7), and a per-task PI selection experiment (Section 5, Figure 5; Appendix G).

W1 + Q1 (a priori framework and predictions). Our metrics follow from the OPSD objective and were not fitted to the results. In OPSD, the student and the teacher share parameters and differ only in access to PI, so the training signal at each token is the difference between the two policies on the student's own rollouts (Section 2.1). This gives a framework with three predictions that can be stated before any training:

  1. High PI utility does not imply a training gain. PI that hands the student an answer to copy helps while it is in context, but OPSD cannot transfer it, because the student cannot reproduce that answer once the PI is removed.
  2. The size of the shift is not enough. PI that shifts the student strongly but in the wrong places should have a large D and still train poorly.
  3. Effective PI concentrates its shift on the student's mistakes. Choosing PI by error-step contrast should therefore improve training, including in cases where choosing by D fails.

The revision tests each prediction by intervention, and not by correlation:

  1. In ALFWorld-Passcode, a PI containing only the correct key gives the student 98.1 pass@1 in context, yet training on it brings no improvement. Replacing the key with a decoy changes utility by 25 points and leaves training performance and both KL metrics essentially unchanged.
  2. The student's own failed rollouts, used as PI, shift its distribution as much as its successful rollouts, but they train poorly and have a much lower error-step contrast.
  3. Selecting PI for each task by error-step contrast outperforms random selection by 1.5 ± 0.3 points with eight PI sources and by 4.6 ± 1.0 points in a pool that mixes successful and failed rollouts, where selecting by D is 19.6 points worse than random.

This is an empirical framework and not a formal theory, and we do not claim otherwise. The general understanding goes beyond a ranking of configurations: PI is effective when it corrects the student at the steps where the student makes mistakes. By contrast, PI that only helps the student succeed while it is in context, or that only shifts the student's distribution by a large amount, does not lead to a training gain. The 126 training runs over 14 PI configurations, three benchmarks, and three students are the evidence base that makes these conclusions testable across settings (Section 3), and Section 5 now opens by stating how the controlled studies test them.

W2 (presentation). We thank the reviewer for these points. The terminology was inconsistent: Section 4 used SDPO for what the rest of the paper calls OPSD. We removed SDPO, so the method has one name throughout, and Section 2.1 defines how OPSD relates to OPD. On the figures and the link between observations and claims, we would like to improve both and would appreciate more specific pointers: which figures obscure a key comparison, and which claims the reviewer finds are not supported by the evidence that precedes them. We will revise these directly once we know where the problems are.

Reviewer_5LYuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical study of Privileged Information (PI) construction and offline selection for On-Policy Self-Distillation (OPSD) in multi-turn agentic tasks. Specifically, the authors evaluate 14 PI configurations—spanning three trajectory formats (Action, Action+Obs., Action+Obs.+Thought) and three hint formats (Summary, Skill, Checklist) generated by GLM-5.2, Qwen3.7-Plus, or the student model—across three benchmarks (ALFWorld, AppWorld, and τ2\tau^2-retail) using three Qwen3 student models (Qwen3-4B-Instruct, Qwen3-4B-Thinking, and Qwen3-30B-A3B-Instruct). To predict PI effectiveness before training, the paper compares PI Utility against two offline KL-based diagnostics: sampled-token reverse KL (DD) and error-step KL contrast (Δderr−\Delta d^-_{\text{err}}). Finally, the authors investigate combining OPSD with GRPO via a weighted advantage objective on Qwen3-4B-Instruct and observe that static PI accelerates early learning but can plateau or hinder performance later in training.

Critical strengths
  1. The main strength is the breadth of the empirical study. The authors compare 14 PI configurations across three multi-turn agentic benchmarks and three Qwen3 student models. Even if some of the conclusions remain limited, this gives a useful picture of how PI format and generation source affect OPSD performance in the settings they study.
Critical weaknesses
  1. The practical value of the offline metrics is not convincingly shown. The paper motivates them as a way to choose effective PI before expensive OPSD training, but the evaluation mostly reports correlations rather than the outcome of an actual selection procedure. Positive rank correlation does not mean the top-scoring PI is a good choice: on Qwen3-4B-Thinking/AppWorld, taking the highest error-step contrast yields 5.85% pass@1, while the best candidate reaches 13.24%. Similar gaps appear in other settings. The paper also does not report the total screening cost—rollout collection, candidate scoring, and external error annotation—nor does it show what error-step contrast adds over the simpler metric DD. Without a direct evaluation of selection quality and cost, the central practical claim remains weakly supported.

  2. The contribution needs clearer positioning, and the experimental scope is too narrow to support broad conclusions. Closely related work has already examined training-free diagnostics for comparing distillation contexts [1], targeted privileged supervision and its decay during training [2], adaptive exposure to privileged reasoning [3], and probability shifts that favor reference imitation rather than correctness [4]. These studies are not discussed, making it difficult to assess what new understanding the proposed diagnostic provides beyond existing analyses. For an empirical analysis paper, the student-model coverage is also limited: all three students are post-trained Qwen3 variants, and the observation about thinking models rests on a single 4B checkpoint. The Qwen3 technical report explicitly describes strong-to-weak distillation for its lightweight models [5], making validation across different model families and post-training histories particularly relevant. Experiments on another student family would help determine whether the PI rankings and diagnostic relationships extend beyond the shared Qwen3 background. Furthermore, the Self-PI conditions only evaluate student-written summaries or skills derived from successful GLM-5.2 trajectories, with the PI fixed throughout training. They do not establish whether the findings extend to PI generated from the student's own experience or refreshed as the student improves.

References

[1] Mohammadreza Armandpour et al. Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why. arXiv:2605.10889, 2026.

[2] Jiaxuan Wang et al. TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment. arXiv:2605.10194, 2026.

[3] Zihao Han et al. Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning. arXiv:2605.11458, 2026.

[4] Sarthak Harne et al. Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation. arXiv:2608.04794, 2026.

[5] Qwen Team. Qwen3 Technical Report. arXiv:2505.09388, 2025.

Questions for authors
  1. How much does the full screening process cost, including rollout collection, scoring each PI candidate, and error annotation? Please account for reuse across candidates and compare this with short pilot OPSD runs under the same budget.

  2. How should a practitioner choose PI when DD and error-step contrast disagree? It would help to report the performance of the selected PI and its gap to the best candidate in each setting. Does error annotation lead to better choices than using DD alone or simply choosing a fixed PI format?

  3. How does this work differ from [1]–[4], particularly the training-free diagnostic framework in [1]? What does the proposed metric add in terms of predicting which PI will help? A small-scale comparison with a relevant prior diagnostic would make this difference clearer.

  4. The Self hints still start from successful GLM-5.2 trajectories. Have you tried constructing PI from the student’s own successful or failed rollouts? Updating these hints as training progresses would also be an interesting direction worth exploring. For the current results, how much do the findings depend on having fixed expert references?

  5. Do the PI rankings and metric–performance relationships hold for students outside the Qwen3 family, with different post-training backgrounds? The observation about thinking models currently comes from a single 4B checkpoint. Have you checked whether it holds for another thinking model, or is it specific to this checkpoint?

3 comments

We thank the reviewer for the detailed and constructive feedback. In response, we added three controlled studies to the revision (changes are marked in blue): a per-task PI selection experiment (Section 5, Figure 5; Appendix G), a corrupted-PI ablation (Section 5; Appendix F, Figure 7), and ALFWorld-Passcode (Section 5; Appendix E, Figure 6). We address each point below.

W1 (practical value of the offline metrics). We agree that correlation alone does not establish selection quality. We added a per-task PI selection experiment on ALFWorld with Qwen3-4B-Instruct. For each training task, we build a pool of candidate PIs, score each candidate with reverse KL (D) and with error-step KL contrast, and train with a PI drawn from the top three candidates by either metric. We compare against a PI drawn uniformly at random from the same pool, over three seeds. The pools vary in candidate strength and source diversity: winning trajectories from eight different strong models, eight winning trajectories from a single strong model, eight successful rollouts of the student itself, and eight successful plus eight failed student rollouts.

In Figure 5, selecting by error-step contrast outperforms random selection whenever the candidates differ in source or strength: by 1.5 ± 0.3 points with eight PI sources, and by 4.6 ± 1.0 points when the pool mixes successful and failed rollouts. In that mixed pool, selecting by D is 19.6 points worse than random, because failed rollouts produce the largest shift. When all candidates are successful rollouts of the student, selection by error-step contrast also outperforms random selection, leading by 5.1 points at step 40. When all candidates come from a single strong model, selection by D helps and the contrast matches random. These results show what error annotation adds: D only measures how much the PI shifts the student, whereas error-step contrast measures whether that shift falls on the student's mistakes.

Q1 (screening cost). Screening has a shared cost and a per-candidate cost. The shared cost is paid once: we sample rollouts of the student without PI and label error steps in the failed rollouts with the judge. These rollouts and labels do not depend on the PI, so every candidate reuses them. The per-candidate cost is a single forward pass over those rollouts conditioned on the candidate PI, with no generation and no gradient computation. On ALFWorld, the error-step contrast uses 32 rollouts for each of 200 tasks. On ALFWorld, a single OPSD run takes about 5–10 hours on 8 H100 GPUs for Qwen3-4B-Instruct, so training on all 14 PI candidates takes about 70–140 hours. Screening all 14 candidates takes about 5–10 hours in total, roughly the cost of one training run. The metrics therefore serve as an inexpensive proxy for PI effectiveness when training on every candidate is impractical.

We did not compare against short pilot OPSD runs under a matched budget. We agree this is an interesting experiment, as it offers another empirical way to check the effectiveness of a PI. However, it does not scale well with the number of PI candidates: each candidate requires its own training run and evaluation, whereas our offline metrics are computed on a shared set of rollouts and error labels that all candidates reuse. We leave this comparison to future work.

Q2 (choosing PI when D and error-step contrast disagree). To answer this, we consider the task of selecting one PI configuration out of the 14 we study (the combinations of PI format and source) for each student and benchmark. The selection is made at the benchmark level, with one configuration used for all tasks, and not for each individual task as in the per-task selection experiment of Section 5. We select the configuration with the highest D or the highest error-step contrast, and report the gap in pass@1 between the selected configuration and the best of the 14 (lower is better; 0 means the best configuration was selected). A random choice among the 14 gives the expected gap without any selection.

StudentRandomTop DTop contrast
Qwen3-4B-Thinking8.46.84.8
Qwen3-4B-Instruct14.22.32.8
Qwen3-30B-A3B-Instruct11.40.42.9
Average of nine settings11.33.23.5

Gap in pass@1 to the best of the 14 PI configurations, selected at the benchmark level. Per-student rows are averaged over ALFWorld, AppWorld, and τ²-retail.

From the table, both KL metrics select a better PI configuration than random selection for all three students, reducing the gap to the best configuration from 11.3 points to about 3 points on average. Averaged over the nine settings, selecting by D is slightly better than selecting by error-step contrast (3.2 against 3.5 points). However, error-step contrast is more consistent when the PI quality is low or when the candidates are mixed with incorrect trajectories, as shown by the per-task selection experiment and the corrupted-PI ablation (Section 5, Figure 5; Appendix F, Figure 7).

Our recommendation is therefore as follows. As a default, use the most consistent PI format: the Action + Obs. trajectory from a strong model, which is within one point of the best configuration in seven of the nine settings. When several PI candidates are available, use the two KL metrics to select among them: D when all candidates come from a single strong source, and error-step contrast when candidates differ in source or strength or when the pool may contain misleading PI. The per-task selection experiment supports this rule in training (Section 5, Figure 5). We do not claim that the metrics can identify the single best configuration.

Self-PI conditions (W2) + Q4 (PI from the student's own rollouts). When the student itself summarizes the reference trajectories into hints, we observe positive performance gains for Qwen3-4B-Instruct on all three benchmarks (Section 3.1, Figure 2). We also tested PI built from the student's own experience. In preliminary experiments, we asked the student to analyze its own successful and failed rollouts and write the hint. Performance improved early in training and then collapsed, because the small student cannot reliably produce correct feedback. Using the student's own successful rollouts directly as PI works better: in the new selection experiment, a randomly chosen successful self-rollout raises Qwen3-4B-Instruct on ALFWorld from 10.4 to 69.9 pass@1, against 86.9 with trajectories from strong models (Section 5, Figure 5; Appendix G). Its failed rollouts are harmful as PI (Section 5; Appendix F, Figure 7). Refreshing PI as the student improves is certainly relevant, and we leave it to future work, as our focus is on understanding how static PI affects post-training performance.

Student-model coverage (W2) + Q5 (other model families). We agree with the reviewer that results on more model families would be helpful. All three students are Qwen3 models, and the observation about thinking models rests on one checkpoint. Our metrics are defined on the student's own token probabilities and make no assumptions about the model family, but we have not verified the PI rankings outside Qwen3, and we will discuss this limitation in the final version of the paper. That said, we chose Qwen3 because it is widely used in prior work on on-policy self-distillation, including the references the reviewer provided: all four of [1]–[4] use Qwen3 students. We also believe our current experiments support our research question of predicting PI effectiveness for post-training: we train Qwen3-4B-Instruct, Qwen3-4B-Thinking, and Qwen3-30B-A3B-Instruct on three benchmarks with 14 PI configurations, for a total of 126 OPSD runs, and we also have further studies on combining OPSD with GRPO, per-task PI selection, corrupted PI, and ALFWorld-Passcode.

W2 + Q3 (positioning against [1]–[4]). We thank the reviewer for these references, which we will discuss in the related work section. Our work has a different focus from all four. We study the effectiveness of privileged information itself: which PI makes OPSD work, and whether this can be predicted before training. We study this on multi-turn agentic tasks, where the PI is a long interaction trajectory or a hint derived from one, and where its format and source can vary widely. To this end, we compare 14 PI configurations across three agentic benchmarks and three students, and we validate every offline metric against the results of actual OPSD training. In contrast, [1]–[4] take the PI as given, usually a reference solution, and study how to distill from it, mostly on single-turn reasoning tasks.

  • [1] (training-free gradient alignment). [1] scores a distillation context by the cosine similarity between its gradient and an estimated ideal gradient. This is the closest work to ours, and it differs in three ways. First, [1] studies single-turn question answering and math (BoolQ, MMLU, and AIME case studies) and leaves long-horizon agentic tasks to future work, whereas we study multi-turn agentic benchmarks. Second, [1] compares a small set of contexts (five self-distillation contexts and three external teachers), whereas we vary PI format and source over 14 configurations. Third, and most importantly, the analysis in [1] is entirely offline: it runs no distillation training and reports no post-training accuracy, so it does not test whether its alignment score predicts which context trains better. We validate our metrics against downstream training results, over 126 OPSD runs, and by intervention: selecting PI by error-step contrast improves held-out performance over random selection, and corrupted PI lowers the contrast (Section 5, Figure 5; Appendix F, Figure 7). Our metrics also need no gradients, only one forward pass per candidate over a shared set of rollouts. The two analyses agree where they overlap: [1] finds that distillation signals align better with the ideal update on incorrect rollouts, and we find that effective PI concentrates its shift on the student's error steps.

  • [2] (span-level targeting). [2] proposes a training method for math reasoning that restricts distillation to selected spans and hands off to GRPO after a short distillation phase. It does not ask which PI to use. We use error steps to evaluate PI, not to mask the loss, so the two can be combined. Their short distillation phase is consistent with our observation that fixed PI accelerates early learning but can plateau later when combined with GRPO (Section 4).

  • [3] (adaptive exposure). [3] learns how much of a reference solution to reveal to the teacher on competition math. It adapts the amount of one kind of PI, whereas we compare different kinds of PI. The conclusions are compatible: richer PI is not always better, and adding observations or thinking tokens hurts the already capable thinking model (Section 3.1).

  • [4] (PI bias). [4] shows that a PI-conditioned teacher can pull the student toward the reference trajectory rather than toward correctness. It diagnoses why self-distillation fails, whereas we ask which PI avoids this failure and how to tell before training. PI utility is inflated when the student can copy from the PI (ALFWorld-Passcode; Appendix E, Figure 6), and the student's failed rollouts produce a shift as large as its successful ones (Appendix F, Figure 7). Error-step contrast separates these cases, because it measures whether the shift falls on the student's mistakes and not only how large it is.

[1] Mohammadreza Armandpour et al. Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why. arXiv:2605.10889, 2026.

[2] Jiaxuan Wang et al. TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment. arXiv:2605.10194, 2026.

[3] Zihao Han et al. Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning. arXiv:2605.11458, 2026.

[4] Sarthak Harne et al. Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation. arXiv:2608.04794, 2026.

Reviewer_5Ln7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the construction of Privileged Information (PI) and the prediction of its effectiveness in On-Policy Self-Distillation (OPSD). The authors systematically evaluate 14 PI configurations across three agent benchmarks and three student models. These configurations include reference trajectories and compressed hints. The study examines how PI content, generation source, and student model characteristics affect distillation performance. The experimental results show that reference trajectories enriched with environmental observations are generally more effective, while Skill tends to perform well among hint-based PI formats. However, richer PI does not consistently benefit thinking models. Overall, this paper is primarily an empirical study, with its main contributions lying in the systematic comparison of PI construction strategies and the exploration of offline metrics for predicting self-distillation gains.

Critical strengths

Novel and meaningful research perspective. This paper investigates how the construction and selection of privileged information (PI) affect OPSD training. To the best of my knowledge, it is among the first works to systematically examine the effectiveness of different PI configurations in OPSD. This is an interesting and important research direction, as it provides valuable insights into the role of different types of PI in self-distillation.

Well-designed and comprehensive experiments. The authors consider two categories of PI, trajectory-based PI and hint-based PI, covering 14 configurations. They conduct extensive experiments across three established agent benchmarks and three student models. The experiments systematically examine multiple factors, including PI content, generation source, and student model characteristics. The results are clearly presented and provide substantial empirical support for the main findings.

Critical weaknesses

Lack of a practical PI selection pipeline. One important motivation for studying PI effectiveness is to identify suitable PI before expensive OPSD training. However, the paper mainly compares different PI configurations and evaluates three offline metrics through correlation analysis, without demonstrating how these metrics can reliably guide PI selection. I suggest adding PI selection experiments with comparisons against random and oracle selection strategies to better validate the practical value of the proposed metrics. Limited discriminative ability of Error-step KL Contrast for thinking models. Although Error-step KL Contrast shows positive correlations with post-training performance across all nine settings, the results for thinking models in the third row of Figure 3 are largely concentrated near zero, with relatively small differences across PI configurations. This raises concerns about the stability and practical discriminative ability of the metric, particularly under sampling noise. The authors should provide further analysis to support their claims about its effectiveness for thinking models.

Insufficient justification for Error-step KL Contrast. The proposed metric assumes that suppressing tokens at erroneous steps provides useful supervision. However, reducing the probability of incorrect tokens does not necessarily imply that the teacher assigns higher probability to correct actions. Moreover, the baseline includes both erroneous and non-erroneous steps, rather than strictly comparing errors against non-errors. While the metric correlates with downstream performance, this does not fully validate its underlying interpretation. Further justification or empirical analysis would strengthen this contribution.

Lack of multi-seed experiments. The paper does not clearly report the variability of evaluation results. Given the stochastic nature of agent rollouts, the reported performance differences may be sensitive to sampling randomness, particularly when the gains are small. I suggest evaluating each model with multiple rollouts per task and reporting confidence intervals or standard deviations to better assess the robustness of the results.

Questions for authors

PI Selection: Can the proposed offline metrics reliably select the best PI configuration before training? How would metric-based selection compare with random selection and oracle selection? Effectiveness for Thinking Models: In Figure 3(g)–(i), Error-step KL Contrast values for thinking models are largely concentrated near zero. Could the authors provide a quantitative comparison of the metric distributions between thinking and instruct models and investigate why the contrast values are consistently small for thinking models? Error-step KL Contrast Design: Could the authors compare the proposed metric with a variant that uses only non-error steps as the baseline? Additionally, could they investigate whether stronger token suppression at error steps corresponds to increased probabilities of correct actions? Evaluation Robustness: Could the authors evaluate each trained model with multiple rollouts per task and report confidence intervals or standard deviations for pass@1, particularly for PI configurations with small performance differences?

2 comments

We thank the reviewer for the careful reading and constructive feedback. In response, we added three controlled studies to the revision (changes are marked in blue): a per-task PI selection experiment (Section 5, Figure 5), a corrupted-PI ablation (Appendix F, Figure 7), and ALFWorld-Passcode, a modified benchmark showing that PI utility can be inflated by leaked information the student can copy but not learn (Appendix E, Figure 6). We address each point below.

W1 / Q1: Lack of a practical PI selection pipeline

We added a per-task PI selection experiment on ALFWorld with Qwen3-4B-Instruct (Section 5, Figure 5; Appendix G). For each training task, we build a pool of candidate PIs, score each candidate with reverse KL (D) and with error-step KL contrast, and train with a PI drawn from the top three candidates by either metric. We compare against a PI drawn uniformly at random from the same pool, over three or four seeds. The pools vary in candidate strength and source diversity: winning trajectories from eight different strong models, eight winning trajectories from a single strong model, eight successful rollouts of the student itself, and eight successful plus eight failed student rollouts.

In Figure 5, selecting by error-step contrast outperforms random selection whenever the candidates differ in source or strength: by 1.5 ± 0.3 points with eight PI sources, and by 4.6 ± 1.0 points when the pool mixes successful and failed rollouts. In that mixed pool, selecting by D is 19.6 points worse than random, because failed rollouts produce the largest shift. When all candidates are successful rollouts of the student, selection by error-step contrast also outperforms random selection, leading by 5.1 points at step 40. When all candidates come from a single strong model, selection by D helps and the contrast matches random. We do not report an oracle, because the best PI for each task is unknown without training on every candidate separately, which is infeasible for eight candidates on each of 1,600 tasks.

W2 / Q2: Metric effectiveness for the thinking model

The contrast values are much smaller for the thinking model (Table 4, Appendix H.2). On ALFWorld, they range from 0.007 to 0.018 for Qwen3-4B-Thinking, against 0.07 to 0.22 for Qwen3-4B-Instruct and 0.04 to 0.09 for Qwen3-30B-A3B-Instruct. On AppWorld and τ²-retail, they stay below 0.003 for the thinking model. The main reason is dilution: each step of the thinking model contains a long reasoning trace, so the per-token shift induced by PI is averaged over many tokens that PI barely affects. The overall shift D is also about three times smaller than for Qwen3-4B-Instruct. Because we use rank correlation, the small scale does not by itself weaken the result: the contrast still ranks PI configurations in line with post-training performance on ALFWorld (ρ = 0.83) and τ²-retail (ρ = 0.55) (Figure 3).

The correlation is weaker on AppWorld (ρ = 0.38) because OPSD itself is less effective for the thinking model there. Its pass@1 stays low under every PI, between 4.2 and 13.2 from a baseline of 4.7, whereas Qwen3-4B-Instruct improves from 7.8 to as much as 29.9 on the same benchmark (Table 3, Appendix H.1). With such small gains, the performance gaps among PI configurations are compressed, and most hint configurations fall within two points of each other. Small gaps make the ranking harder to recover from a metric whose values are themselves close to zero. We discuss the thinking model's small contrast values in Section 3.2.

W3 / Q3: Further justification or empirical analysis of the KL contrast

We added an ablation with corrupted PI to justify why error-step KL contrast is preferred (Section 5; Appendix F, Figure 7). It shows that incorrect PI can shift the model just as much as correct PI: the student's own failed rollouts, used as PI, produce about the same reverse KL (D) as its successful rollouts, yet they train poorly. D measures only how far PI moves the model; it does not tell us whether PI moves the model toward the correct distribution. We expect a metric that relies on non-error steps to behave similarly to D, since the shift at non-error steps reflects the overall magnitude of the change and not whether mistakes are corrected. Error-step KL contrast, in contrast, distinguishes incorrect PI from correct PI, because correct PI shifts the model away from its mistakes: the contrast is much lower for the failed rollouts. The selection experiment confirms this in training. In a pool that mixes successful and failed rollouts, selecting PI by D is far worse than random selection, while selecting by error-step contrast is better than random.

On whether suppression corresponds to higher probability of correct actions: OPSD trains on the student's sampled tokens, weighting each by the difference between the teacher's and the student's log-probability. Tokens the PI-conditioned teacher finds less likely are suppressed, and the freed probability moves to the alternatives the teacher prefers. Error-step contrast measures how strongly this suppression falls on the student's mistakes. The corrupted-PI and selection results above show that when the PI is correct, the student is steered toward better actions.

W4 / Q4: Lack of multi-seed experiments

Each reported pass@1 is computed from four rollouts per task and averaged over three consecutive checkpoints, that is, 12 rollouts per task on 274 ALFWorld, 168 AppWorld, or 114 τ²-retail evaluation tasks. The resulting standard error from rollout sampling is at most 0.9 points on ALFWorld, 1.1 on AppWorld, and 1.4 on τ²-retail. We will add these intervals to the result tables.

Comments on the paper

1 comment

We thank the reviewer for the careful reading and constructive feedback. In response, we added three controlled studies to the revision (changes are marked in blue): a per-task PI selection experiment (Section 5, Figure 5), a corrupted-PI ablation (Appendix F, Figure 7), and ALFWorld-Passcode, a modified benchmark showing that PI utility can be inflated by leaked information the student can copy but not learn (Appendix E, Figure 6). We address each point below.

W1 + Q1 (small cells, checkpoint selection). The offline metrics are computed only on training or development tasks disjoint from the evaluation sets: 200 sampled ALFWorld training tasks, the AppWorld train and dev splits, and 150 τ²-retail training tasks, with 32 rollouts per task for the error-step contrast. Following prior work [1], post-training performance is the highest mean pass@1 over three consecutive checkpoint evaluations on the evaluation set, so the window is selected on the evaluation set (Section 2.3; Appendix B.3). At each checkpoint, we sample four rollouts per evaluation task, so each reported number averages 12 rollouts per task. The same rule is applied to all 14 PI configurations, and it serves to smooth checkpoint noise, not to tune any PI.

W2 (judge self-consistency). We are not sure what the reviewer means by "judge self-consistency" here, and would appreciate a clarification. In our setup, the judge that labels error steps (GPT-5.6 Sol) is a different model from the student and from both PI generators (GLM-5.2 and Qwen3.7-Plus). It sees only the student's rollout and never the PI, and every PI candidate is scored on the same labeled steps, so the labels cannot favor one PI over another.

Two experiments in the revision indicate that the contrast captures a genuine signal. First, in per-task PI selection (Section 5, Figure 5; details in Appendix G), choosing PI by error-step contrast improves held-out performance over random selection, by 1.5 ± 0.3 points with eight PI sources and by 4.6 ± 1.0 points when the pool mixes successful and failed rollouts. Second, in the corrupted-PI ablation (Section 5; Appendix F, Figure 7), incorrect PI shifts the student's distribution as much as correct PI but yields a much lower contrast and poor training performance. An artifact of the judge would neither improve performance after training nor separate correct from incorrect PI.

Q2 (lifting the two-error-step cap). We cap the judge at two error steps because, once the agent makes a mistake, later steps are conditioned on a corrupted state, so they may carry noisy signals as they depend on the previous mistake. HINT-SD [2] makes the same argument and likewise restricts feedback to a few failure-relevant steps per failed trajectory. We agree that sensitivity to the cap is important and will add this discussion in the final version.

Q3 (negative control with matched-length PI). A quick clarification: in the Self conditions the student writes the hint (a summary or skill) from a successful reference trajectory, but it does not grade its own data. Rewards come from the environment and error steps are labeled by an external judge. We ran this negative control in the new PI ablations on ALFWorld with Qwen3-4B-Instruct (Appendix F, Figure 7). Giving each task the full trajectory of a different training task, a PI of matched length and format with unrelated content, reduces pass@1 from 87.9 to 34.8. A trajectory from another benchmark degrades the student to 5.4, below the untrained baseline. As a pure length control, padding the action list to the length of the full trajectory leaves OPSD performance close to that of the unpadded list.

Q4 (Thinking-student negatives across sources). We assume "Thinking-student negatives" refers to the configurations where OPSD leaves Qwen3-4B-Thinking below its untrained baseline on τ²-retail. These are stable across sources and track the PI format: the two observation-bearing trajectory formats (Action+Obs. and Action+Obs.+Thought) perform poorly under both GLM-5.2 and Qwen3.7-Plus, while the other formats stay close to the baseline under all sources. As discussed in Section 3, the untrained Thinking student is already strong on this benchmark (88.6% pass@4), so the large distributional shift induced by these dense trajectory formats can overwrite useful behavior already present in the policy.

[1] Emiliano Penaloza et al. Privileged Information Distillation for Language Models. ICML 2026.

[2] Woongyeong Yeo et al. HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents. arXiv 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.