acceptodds
Under review as a conference paper at ICLR 2027

Pay for Rollouts Only When Needed: Compute-Aware Routing for Language Model Distillation

Abstract

On-policy distillation addresses exposure bias by training a student language model on its own generated responses, with supervision from a teacher. However, generating and scoring these responses, specially with frontier models, is more expensive compared to off-policy distillation using cached teacher responses. We investigate whether selectively allocating on-policy supervision can reduce distillation cost while maintaining student quality. In samples where a student favors one teacher-supported choice while ignoring others (i.e., anchored collapse), off-policy training with cached teacher probabilities can help recover those alternatives. In contrast, when a student assigns substantial probability to choices the teacher considers unlikely (i.e., support escape), the student's trajectory may not match the cached prefixes in off-policy distillation, making fresh rollouts via on-policy distillation helpful. We formalize this in our proposed compute-aware routing (CAR) approach for distillation, choosing among three actions: skipping an update, cached off-policy and on-policy distillation. CAR compares student and teacher probabilities on cached prefixes to predict action utilities that account for compute costs. In particular, we compute directional support statistics from cached prefixes to distinguish anchored collapse, for which off-policy may suffice, from support escape, for which fresh on-policy supervision may provide additional value. We also provide a routing-regret bound conditional on utility-estimation accuracy and an idealized support-corridor bound linking escape probability to loss-surrogate discrepancy under a common trajectory distribution. In a one-seed GSM8K study, CAR achieved 62.40% accuracy versus 62.24% for vanilla on-policy distillation while routing 71.02% of samples to fresh on-policy supervision, indicative of cost savings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge distillation, on-policy distillation, language models, compute-efficient learning, adaptive routing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QDUAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a post-training distillation algorithm, which given a prompt and teacher trajectory, selectively chooses whether to perform on-policy rollout or to use teacher trajectory, selectively choose whether to perform on-policy rollout or to use teacher trajectory for training. The method looks at directional support mismatch between student and teacher to make this decision.

Critical strengths
  • The motivation is intuitive, and it asks an important question of how to know whether on-policy benefits more than off-policy or vice versa.
Critical weaknesses

[W1] Insufficient empirical evidence. Experiments are conducted on only one teacher-student pair and one task (GSM8K). It is unclear whether the method generalizes across model scales and tasks.

[W2] It is hard to see the effect of support-aware routing due to the lack of simple selective-routing baselines. Comparison with KL-only or entropy-only baselines under matched rollout budgets will be helpful.

[W3] The central hypothesis that support escape favors on-policy supervision is insufficiently validated. The paper does not directly demonstrate that the proposed support-statistic-based diagnosis successfully predicts the relative learning benefits of on/off-policy data.

[W4] The reported compute savings are difficult to attribute solely to the proposed routing strategy. It seems like the naive OPD and CAR use different batch sizes, number of steps, and learning rate schedules. Can the authors clarify this?

Questions for authors

[Q1] Could this approach be extended to token-level or segment-level, where the choice between on-policy and off-policy rollout is made at a finer granularity?

[Q2] Can the authors compare CAR against random routing and simpler KL- or entropy-based selection strategies under the same on-policy rollout budget?

Reviewer_yZtoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes compute-aware routing (CAR) for language model distillation. It uses teacher and student probabilities on cached teacher trajectories to choose among skipping an update, cached off-policy distillation, and fresh on-policy distillation. The main idea is to distinguish missing teacher-supported alternatives from student probability mass outside the teacher’s effective support. Experiments on GSM8K show that the fixed-prior router achieves accuracy close to fully on-policy distillation while using fewer fresh rollouts.

Critical strengths

The paper addresses a practical problem: reducing the cost of on-policy distillation. Selecting which examples need fresh rollouts is a useful direction, and the distinction between anchored collapse and support escape provides an intuitive motivation.

The evaluation includes cached forward-KL distillation as a stronger offline baseline than sequence-level distillation. The paper also reports calibration overhead and acknowledges that it can outweigh the savings from routing.

Critical weaknesses

The comparison with OPD is not well controlled. CAR uses different training data, batch sizes, and numbers of updates, so it is unclear how much of the compute savings comes from routing. A random-routing baseline with the same budget is also missing.

The evaluation is limited to GSM8K, one teacher–student pair, and one seed. The paper does not directly show whether the proposed support features reliably identify which examples benefit from fresh rollouts.

Calibration is expensive and sensitive to its settings. Including probing, the larger-probe variant at λ = 0.01 costs more than OPD, while increasing λ to 0.1 substantially reduces accuracy. This limits the practical benefit of the calibrated router.

Questions for authors
  1. How does CAR compare with random routing and OPD when using the same training data, batch size, and compute budget?

  2. Can you show whether the proposed support features predict which examples benefit more from fresh rollouts than cached updates?

  3. Can calibration retain accuracy while reducing total compute, including probing costs, across different seeds and settings?

Reviewer_jfZWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets the high computational cost of on-policy distillation. To mitigate the issue, instead of generating fresh student rollouts for every training example, this paper proposes Compute-Aware Routing (CAR), which selects among three actions: skipping an update, cached off-policy and on-policy distillation. The key idea is to distinguish anchored collapse, where cached supervision may suffice, from support escape, where fresh rollouts may be beneficial. The paper also presents some theoretical bounds on it.

The authors conducted experiments on GSM8K with Qwen3.5-9B teacher and Qwen3.5-0.8B student and showed that their method can achieve competitive performance (62.4 vs 62.24) while reducing training cost.

Critical strengths
  • The motivation is significant. Reducing the cost of on-policy distillation is practical and useful.
  • The high-level idea of token routing is a feasible solution.
  • The paper is mostly very understandable.
Critical weaknesses
  • Theoretical results do not support the proposed routing mechanism. For example, Theorem 1 provides a generic regret bound under accurate utility estimation, but the authors explicitly acknowledge that it "does not establish that our support-based predictors satisfy the assumed error bound" (Section 5). Similarly, for Proposition 3.
  • Experiments only on one dataset with one model pair setup.
  • Some experimental results are confusing.
Questions for authors
  • Table 3 shows that Prior-Guided CAR routes only 0.98% of examples to CACHE, while approximately 69% are routed to ON and 30% to SKIP. Given that a central motivation of CAR is to distinguish when cached supervision suffices from when fresh on-policy rollouts are necessary, why is the CACHE action selected so rarely? Can the authors elaborate more?

  • Calibrated CAR with the larger probe and λ = 0.01 requires 11.19 H100-hours for training and an additional 22.82 H100-hours for probing, resulting in a total cost of 34.01 H100-hours, exceeding the 27.90 H100-hours of full OPD. Can the authors elaborate more on its benefits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.