- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how to learn privileged context from prior experience for on-policy self-distillation. The proposed method, LOPD, uses a trainable composer to compress retrieved successful trajectories into continuous latent tokens, which condition a frozen teacher to provide token-level supervision on the student’s own rollouts. The student and composer are jointly optimized through distillation, with an outcome-weighted privilege-margin constraint and an initialization anchor intended to prevent the teacher’s supervision from becoming uninformative.
Critical strengths
-
A sensible and conceptually appealing approach. Using a trainable composer to encode retrieved experience into latent privileged context is an elegant extension of on-policy self-distillation. This design allows the representation of experience to be optimized directly for the distillation objective, providing a clear motivation for the proposed method.
-
Substantial empirical evaluation. The paper evaluates the method on multiple benchmarks spanning tool use and code generation, with comparisons against a broad set of relevant baselines. The overall experimental coverage and reported performance improvements provide reasonable empirical support for the method’s effectiveness.
Critical weaknesses
-
The effects of experience selection and latent conditioning are not sufficiently disentangled. Latent conditioning is a plausible and potentially useful design. However, we note that several baselines differ from LOPD in how they obtain and select the experiences provided to the teacher (Appendix B.1). The observed gains may therefore reflect two potentially intertwined factors: (i) which experiences the teacher receives, and (ii) how those experiences are represented and used to condition the teacher. The proposed composer primarily addresses the second factor, while the baseline comparisons also vary the first, making it difficult to attribute the improvements specifically to latent conditioning. Figure 3 provides useful ablations of composer training and the privilege margin, but all variants still use latent context. A textual-context ablation within the LOPD pipeline, using the same retrieval method, consistent experience-bank construction and update rules where applicable, and matched teacher and distillation settings, would provide stronger evidence for the contribution of latent conditioning.
-
Limited evidence for training stability and reproducibility. Jointly optimizing the student and composer introduces a risk of degenerate teacher–student agreement, making the privilege-margin constraint particularly important. Although the margin ablation is informative, the paper does not clearly report the number of independent training seeds or variability across runs, and the statistical meaning of the shaded regions in Figure 4 is unspecified. Results across multiple training seeds would help assess the reliability of the gains. Additionally, tracking the achieved margin and dual variable would clarify whether the constraint is consistently satisfied and whether the joint optimization remains stable throughout training.
Questions for authors
-
Could the authors add a textual-context ablation within the LOPD pipeline, using the same retrieval method and consistent procedures for constructing and, if applicable, updating the experience bank, while matching the teacher and distillation settings? Please clarify whether the bank is updated during training; where feasible, matching the retrieved experiences would help more directly isolate the contribution of the composer.
-
Could the authors report results across multiple independent training seeds, including the number of runs and variability, and clarify what the shaded regions in Figure 4 represent? Reporting how the achieved margin and dual variable evolve during training would also help assess whether the constraint is consistently satisfied and whether the joint optimization remains stable.
Overall, we believe this paper makes an interesting contribution and would be willing to raise our score to support acceptance if the above concerns are satisfactorily addressed.
4 comments
Response to Reviewer 12Xi (2/2)
W2 & Q2Training stability and reproducibility: results across independent training seeds, the meaning of the shaded regions in Figure 4, and how the achieved margin and the dual variable evolve during training.
Thank you for this constructive suggestion on the reliability of our results!
LOPD's gains are stable across independent training seeds. We rerun the main setting (Qwen3-4B, trained on EnvScaler) with three independent training seeds and evaluate each resulting student on EnvScaler and on the held-out ACEBench:
| Seed 1 | Seed 2 | Seed 3 | Mean ± std | Base model | GRPO | |
|---|---|---|---|---|---|---|
| EnvScaler | 63.1 | 63.4 | 64.4 | 63.7 ± 0.7 | 48.6 | 61.8 |
| ACEBench | 58.6 | 59.3 | 61.3 | 59.7 ± 1.4 | 50.6 | 56.0 |
On both benchmarks, the variation across seeds is small relative to the gains over the base model and the strongest baseline.
Figure 4: the shaded regions show the fluctuation within a moving window of a single training run, i.e., the variation of the evaluation results across neighboring checkpoints around the smoothed curve, rather than the variance across seeds, which is reported above.
The margin constraint is effectively enforced, and the joint optimization remains stable. We track the achieved margin (Eq. 12) and the dual variable over a representative training run with :
| Training progress | 0% | 25% | 50% | 75% | 100% |
|---|---|---|---|---|---|
| Achieved margin | 0.076 | 0.062 | 0.053 | 0.043 | 0.051 |
| Dual variable | 0 | 0.048 | 0.117 | 0.183 | 0.164 |
starts well above after initialization and is gradually pulled down by the distillation gradient. As it approaches , grows and pulls it back: briefly dips to 0.043 and then recovers to 0.051, while stays bounded and eases from 0.183 to 0.164 once the constraint is restored. The margin thus remains around throughout training, and neither diverges nor oscillates, so the joint optimization stays stable. This is exactly how the constraint keeps the teacher from collapsing toward the student, the degenerate agreement that otherwise lowers performance to 55.1 without the margin (Figure 3).
We hope these additional results address the reviewer's concerns. In the revision, we have included the textual-context ablation (Sec. 4.3, Table 4), the results across independent training seeds (Appendix B.6), and the dynamics of the achieved margin and the dual variable (Appendix B.7), and clarified the meaning of the shaded regions in Figure 4 (Appendix B.4). We would be glad to discuss any remaining questions.
Thank you for the detailed rebuttal and additional experiments. The controlled textual-context ablation provides useful evidence for the contribution of trainable latent conditioning under matched retrieval settings, although retrieval differences across existing baselines remain an open limitation. That said, we appreciate the authors' efforts to address our concerns and find the additional evidence convincing overall. We therefore raise our rating from 2 to 3 and support acceptance.
Thank you for your follow-up and support
We sincerely thank the reviewer for the thoughtful engagement, for raising the score, and for supporting acceptance. We are glad that the controlled textual-context ablation helped clarify the contribution of latent conditioning.
We agree that how privileged information is obtained and selected across self-distillation methods is an interesting question in its own right. Since each existing method defines what its teacher receives as part of its own design, we compared against them as originally proposed and used the controlled ablation within LOPD to isolate the effect of latent conditioning. A unified comparison of these design choices across methods would indeed be a valuable direction for this line of work.
Thank you again for your constructive feedback, which has helped strengthen the paper.
Response to Reviewer 12Xi (1/2)
We sincerely thank the reviewer for the careful and constructive review, and for finding latent conditioning an elegant extension of on-policy self-distillation! In response, we provide a textual-context ablation within the LOPD pipeline, results across three independent training seeds, and the training dynamics of the achieved margin and the dual variable. We address each comment in detail below.
Thank you for this insightful suggestion, which helps isolate the core contribution of our method!
In the main comparison, each baseline obtains its privileged information as designed in its original paper (Appendix B.1), since what the teacher receives is an integral part of each method; notably, OPSD and SDFT even condition on oracle traces partly generated by stronger models, whereas LOPD uses only the model's own rollouts. We nevertheless agree that a controlled comparison is the most direct evidence, and run it as follows.
With the experiences held fixed, latent conditioning clearly outperforms textual conditioning. Following the reviewer's suggestion, we run a textual-context ablation within the LOPD pipeline (Qwen3-4B, EnvScaler). The teacher receives the same top-3 experiences, retrieved by the same method from the same bank, but as raw text placed directly in its context instead of latent tokens; the teacher (the same frozen backbone), the distillation objective (reverse KL with the top-M teacher logits plus a tail bucket), and the training budget are all unchanged. Since the textual context is fixed, there is no composer to optimize, so the cold start and the margin do not apply.
Holding the experiences fixed, textual conditioning barely improves over the base model (50.6 vs. 48.6), whereas latent conditioning reaches 63.7, an improvement of 13.1 points with exactly the same experiences.
For completeness, which experiences are provided also matters: in our response to Reviewer CqJo, replacing the top-3 retrieval with no retrieval or random retrieval reduces LOPD to 49.7 and 50.2. Both factors therefore contribute, and latent conditioning is what turns relevant experiences into effective supervision.
The bank is fixed during training. It is built offline from the model's own successful rollouts on the training split before LOPD training and is never updated (Sec. 3.2 and Appendix A.4); all variants above share this same bank.
We hope this clarifies the contribution of latent conditioning, and we thank the reviewer again for suggesting this controlled comparison.