acceptodds
Under review as a conference paper at ICLR 2027

Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teacher

Abstract

Offline on-policy distillation gains efficiency by collecting student trajectories and teacher supervision once and reusing them throughout optimization. The same reuse makes imperfect supervision persistent. Since even strong teachers can fail, we ask what remains learnable from imperfect teacher supervision? Teacher failure is only a coarse problem-level signal and does not imply that all supervision along the associated student trajectory is unhelpful. A natural alternative is to estimate teacher recoverability along the trajectory, but repeated continuations largely erase the efficiency advantage of offline distillation. We instead use teacher-successful problems to define a cheap reference for what the student can learn. We train on teacher-successful problems and measure how the likelihood of each observed token in trajectories from teacher-failed problems changes. We use these signed likelihood changes as an operational learnability signal: larger increases indicate behavior more strongly promoted by successful-only learning. We aggregate this signal into trajectory-level weights for the original distillation loss. Unlike continuation-based estimates, our learnability requires no additional generation and can be computed once from stored trajectories and model checkpoints. Across mathematical reasoning and code generation, our method improves an offline OPD baseline by up to 2.7 percentage points and matches or outperforms online OPD variants on multiple benchmarks. Despite the additional successful-only distillation stage, it uses 2 GPUs and about 22 GPU hours, compared with 3 GPUs and 36–48 GPU hours for representative online OPD methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Offline Policy Learning, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Zs8COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Learnability-Weighted Distillation (LWD) to address imperfect teacher supervision in offline on-policy distillation. The method trains an auxiliary student on teacher-successful problems and measures token-level likelihood changes on teacher-failed trajectories. These signals are aggregated into trajectory-level weights for the final OPD objective. Experiments on mathematical reasoning and code generation demonstrate consistent improvements over Lightning-OPD across two student scales, with gains of up to 2.7 percentage points. LWD also achieves competitive performance against online OPD methods at lower computational cost.

Critical strengths
  1. Important and well-motivated problem. The paper addresses a practical limitation of offline OPD: imperfect teacher supervision can be repeatedly reinforced during training. The observation that teacher failure does not necessarily invalidate all supervision along a student trajectory is meaningful.
  2. Simple and practical methodology. LWD provides an efficient alternative to expensive continuation-based estimation. Its trajectory-level weighting preserves the original token-level OPD objective and does not require additional generation for scoring.
  3. Consistent empirical improvements. The method improves the offline OPD baseline across mathematical reasoning and code generation at two student scales. The paper also includes relevant ablations and computational cost comparisons, demonstrating competitive efficiency relative to online OPD methods.
Critical weaknesses
  1. Insufficient validation of the learnability signal. The proposed signal measures how successful-only learning changes the likelihood of observed student tokens, rather than the correctness or utility of teacher supervision. While Eq. (15) provides a gradient-alignment interpretation, it does not establish that higher learnability corresponds to more beneficial supervision. Figure 3 offers only qualitative examples, and the paper lacks systematic validation or controls against simpler proxies. In particular, a shuffled-tier baseline is needed to verify whether trajectory-specific learnability rankings actually contribute to the improvements.
  2. Limited incremental gains over simpler baselines. Although LWD consistently improves Lightning-OPD, its advantage over constant downweighting is modest. In Table 5, LWD achieves 57.1% average accuracy, compared with 56.6% for the best constant-weight baseline. Meanwhile, its GPU-hour cost approximately doubles relative to Lightning-OPD (21.96 vs. 10.64). Without independent training-seed results, statistical uncertainty estimates, or compute-matched comparisons, it remains unclear whether the additional complexity is justified.
Questions for authors
  1. Does learnability provide information beyond simpler weighting strategies? Could the authors compare against shuffled learnability tiers and inexpensive difficulty-based proxies while preserving the same weight distribution?
  2. Are the improvements over constant weighting statistically robust and compute-efficient? Could the authors provide results across multiple training seeds, uncertainty estimates, and compute-matched comparisons?
  3. How sensitive is LWD to teacher-outcome labels? Appendix E reports different mathematics verifier versions and uses response validity rather than functional correctness for code. Could the authors quantify the impact of these choices and examine whether learnability tiers correlate with independently measured supervision quality?
Reviewer_FbcLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies offline on-policy distillation, where stored student rollouts and teacher supervision are reused throughout training, potentially reinforcing unreliable guidance from an imperfect teacher. It generates an independent teacher response for each problem and argues that supervision on teacher-failed problems should be downweighted rather than discarded entirely. To distinguish these records, it trains an auxiliary student on teacher-successful problems and measures how that training changes the likelihood of stored student trajectories from teacher-failed problems. These changes determine trajectory-level weights for a final distillation run using the original teacher targets. The method improves math and coding performance over the offline baseline, although its end-to-end efficiency advantage remains unclear because the cost accounting does not explicitly include all preparation steps, particularly independent teacher-response generation.

Critical strengths
  • A new way to estimate trajectory weights: the paper uses likelihood changes after successful-only training to differentiate supervision on teacher-failed problems, without repeated continuation sampling.

  • Consistent gains over the closest offline baseline, supported by ablations: it improves Lightning-OPD across all five benchmarks at both student sizes, and tests whether selective weighting helps beyond simply discarding or uniformly downweighting teacher-failed records. These are accuracy gains under the reported protocols, not demonstrated equal-compute gains.

Critical weaknesses

One big one is that the comparisons do not isolate the benefit at equal cost. LWD adds an auxiliary training stage and teacher-outcome information, which requires very expensive teacher rollouts (these are much more expensive than trajectory scoring by the teacher). These preparation costs are not clearly included.

The “learnability” signal is only indirectly connected to supervision quality. Increased student-token likelihood after successful-only training does not establish that the original teacher targets are useful. The incremental gain over the best constant downweighting is also modest: 57.1% versus 56.6% averaged across the math ablation.

Typically coding experiments are measured for accuracy on test cases, but this paper only measures format and syntax, and not functionality.

Gains are relatively small.

Questions for authors

What is the relative cost of rolling out a teacher vs using it only to score a student trajectory ?

Reviewer_dgraOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a weighted approach to improving offline on-policy distillation. Rather than using only successful teacher rollouts or assigning equal weights to successful and failed teacher trajectories, the method assigns failed trajectories to low, medium, or high weight categories. The underlying intuition is that even unsuccessful teacher rollouts may contain useful supervision, depending on how learnable they are for the student. The method estimates learnability by measuring changes in the student’s likelihood of its generated tokens after training on successful trajectories and assigns a single weight to each trajectory rather than weighting individual tokens.

Critical strengths
  1. The problem is relevant, teacher failure at the problem level does not imply that every token level target is useless.

  2. The proposed method is relatively simple and can be applied to stored trajectories without additional continuation rollouts.

  3. The authors include useful ablations and implementation details.

Critical weaknesses
  1. It seems to me some main baselines are missing. The paper has a discussion on how recoverability solves a similar problem but with additional cost. However, the proposed method also comes with a large cost and is not free. The comparison between their approach and the recoverability estimation approaches is not tested in both the additonal cost and the performance.

  2. The paper includes many additional hyperparameters that affect the performance of the approach. However, there is no sensitivity analysis on the choice of these new params. This makes it very hard to understand how this approach can be used in practice as tuning will add its own additional cost. There is also no variance reported in the table results, making it harder to know how meaningful the reported numbers are different from each other.

  3. Given that the new approach is almost doubling the GPU cost, the model should also compare the results with the same compute given to the baseline approaches by for example additional rollout samples. Otherwise it is hard to know where exactly the performance gain is coming from.

Questions for authors
  1. The method trains a student on the successful rollout first. However, the successful or failed result will change as the sampling temperature changes. How sensitive the approach is to the choice of hyperparameters like temperature? What is the sampling strategy used in the paper? Is it argmax or temperature = 1?

  2. In Eq 12, is the action same as the offline rollouts or are they generated by the new teacher-successful trained student? Do authors have any intuition how this choice might affect the result?

  3. If I understood correctly, in the code examples, the success is the generation of a valid Python program without functional testing. Is that right? How do authors expect the functional correctness might affect the weights? What if the successful set becomes too small.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.