acceptodds
Under review as a conference paper at ICLR 2027

SAKI: Maximal-Coupling-Routed Teacher Supervision for On-Policy Distillation

Abstract

On-policy distillation (OPD) reduces train–test state mismatch by training a student on its own generated trajectories. However, a weak student may initially visit poor, teacher-misaligned prefixes, forcing the teacher to provide supervision on states that it would rarely generate under its own policy. Teacher-guided rollout policies improve the visited state distribution, but typically retain the same per-prefix reverse-KL objective. We introduce SAKI (Supervision Allocation with KL-constrained Interpolation), which routes token-level supervision using realized accept/correction events from maximal coupling. We construct a geometrically interpolated behavior policy inside a student-centered KL trust region and realize it through maximal coupling with the student. The coupling preserves a student proposal whenever possible and exposes a correction event precisely when realizing the guided policy requires an intervention. Accepted positions retain reverse-KL supervision, whereas correction positions switch to direct supervision on the teacher's highest-probability token. Because the correction probability is exactly , the same trust-region radius constrains rollout deviation and upper-bounds intervention and specialized-supervision frequency. An engine-resident speculative verifier preserves the exact- trajectory distribution and coupling semantics while improving matched-workload rollout throughput by . Across seven mathematical reasoning benchmarks, our method improves the matched teacher-guided baseline in Mean@8 and Pass@8 for both 1.7B and 0.6B students. Placement controls show that correction-triggered routing outperforms both equal-budget random and TV-weighted placement. Fixed-prefix analysis further shows persistent teacher alignment, with larger gains over random placement at higher initial student–teacher disagreement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language models, maximal coupling, speculative decoding, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fFRVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate direct teacher supervision during on-policy distillation. SAKI builds on TRB and uses maximal-coupling acceptance and correction events to select the training objective. Accepted positions retain reverse-KL updates, while correction positions use cross-entropy on the teacher's highest-probability token. The experiments report improvements in mathematical reasoning.

Critical strengths
  1. The research question is worthwhile, and the idea makes sense. Using information from the rollout process to decide where to apply direct teacher supervision is a reasonable approach (Sections 1 and 3).

  2. The paper includes relevant placement and target ablations. It compares correction-based placement with count-matched random and TV-weighted placement, and compares teacher Top-1 targets with teacher-sampled targets (Appendices D and F).

Critical weaknesses
  1. [Medium] The experimental coverage is narrow for an incremental contribution. The behavior distribution comes from TRB, and both training losses are standard; the main algorithmic addition is the rule that switches objectives at correction positions (Section 3). All seven benchmarks concern mathematical reasoning. Both students, Qwen3-0.6B and Qwen3-1.7B, use the same Qwen3-4B teacher. There are no other model families or teacher configurations, and the placement controls are tested only on the 1.7B student (Section 4.1; Table 6).

  2. [Medium] The small gains over the closest control lack evidence of training stability. Relative to TV-Weighted-TM, SAKI improves macro Mean@8 and Pass@8 by 0.76 and 0.73 percentage points. The main results do not report means and standard deviations across independent training seeds. The prompt-bootstrap intervals in the alignment analysis do not address this uncertainty (Table 6; Appendix E).

The problem is valid and the idea is reasonable, but these limitations leave me unconvinced that this incremental design offers enough reliable value to justify acceptance.

Questions for authors
  1. How stable are the gains across independent training seeds? Please report paired repetitions for SAKI, TRB, and TV-Weighted-TM, including the variability of their final benchmark differences.

  2. Does the benefit extend beyond the current model and task setup? An additional teacher-student configuration or a non-mathematical task would help assess how broadly the routing rule applies.

  3. What does correction-based routing add over simpler supervision schedules? There is no direct budget-matched SFT-to-OPD comparison. Moreover, moving teacher-mode supervision also changes which positions lose their reverse-KL updates, so the placement controls do not fully separate these two effects (Section 4.1; Equation 17; Appendix D). A budget-matched SFT-to-OPD comparison and a control that separates teacher-label placement from reverse-KL removal would clarify the source of the gains.

Reviewer_jUUFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SAKI, an online teacher-guided distillation method that routes token-level supervision using maximal-coupling events. Following TRB, it samples rollouts from a student-centered trust-region behavior distribution. At accepted tokens it keeps standard reverse-KL supervision, while at rejected (correction) tokens it switches to teacher Top-1 supervision. To make sampling from the guided distribution practical, the authors implement the rejection-sampling loop directly inside the inference engine. Experiments on mathematical reasoning benchmarks compare SAKI against standard distillation baselines and several alternative supervision-routing rules.

Critical strengths
  1. Sensible intuition for token-level supervision routing. Using coupling acceptance/rejection events to decide where the student only needs gentle distribution matching versus explicit teacher correction makes intuitive sense. Including random and TV-weighted placement controls is also a good experimental choice, as it directly tests whether the exact coupling event matters beyond just knowing the local TV distance.

  2. Practical systems effort on guided rollout sampling. Moving the guided rejection-sampling loop inside the serving engine gives a clear throughput improvement over doing it in an external loop, which is helpful in practice when online teacher queries are on the critical path.

Critical weaknesses
  1. The empirical margin of coupling-based routing over simpler placement rules is relatively modest. Since the trust-region behavior distribution follows TRB, the main algorithmic addition in SAKI is using coupling events to route the supervision loss. On the 1.7B student, SAKI improves over the TV-weighted placement control by 0.76 points on Mean@8 and 0.73 on Pass@8. Because the marginal probability of a coupling rejection at step tt is itself determined by the local TV distance, it is not fully clear from the current analysis why conditioning the loss switch on the binary realization of the coupling event—rather than weighting or triggering supervision directly by the deterministic TV distance—yields only a small downstream difference, or when that distinction should matter most. In addition, since these are verifiable math tasks, including a direct student GRPO baseline under a comparable compute budget would be a helpful reference point—even though GRPO uses outcome rewards rather than dense teacher logits, it would help readers situate the practical trade-off of using an online teacher.

  2. Some presentation details and the scope of the efficiency comparison could be clearer. In the abstract, the opening description of OPD is slightly broad (see Question 3), and TV, ptp_t, and qtq_t appear before being defined. Figure 1 also packs a lot of information into a single diagram, so separating behavior construction, maximal coupling, and loss routing a bit more clearly would help readers follow the pipeline. There is also a minor layout issue on page 12 where references, a configuration table, and appendix text overlap. On the efficiency side, the 4.22× speedup is measured against an external-loop implementation of exact guided sampling rather than standard OPD. The paper does note this distinction, and the systems speedup itself is solid, but a clearer picture of the end-to-end training cost relative to simpler OPD or TRB baselines would make the overall cost–benefit trade-off more compelling.

Questions for authors
  1. In maximal coupling, the marginal probability of a rejection at position tt is already governed by the local TV distance TV(pt,qt)\mathrm{TV}(p_t, q_t). Could you clarify what drives the difference between routing on the realized binary coupling event (SAKI) versus routing directly by TV(pt,qt)\mathrm{TV}(p_t, q_t) (TV-weighted placement)—for example, is the benefit mainly tied to matching Top-1 supervision with the exact steps where a replacement token actually alters the rollout prefix, and how sensitive is this behavior to the trust-region radius schedule?

  2. Have you tried comparing SAKI against running GRPO directly on the student model with the same prompts? Even an approximate comparison with compute costs would be a helpful reference for understanding the practical trade-off of keeping an online teacher in the loop.

  3. As a small wording suggestion for the first sentence of the abstract: would you consider making the OPD framing a bit more specific—e.g., clarifying that compared with offline distillation on teacher trajectories, OPD reduces prefix-distribution shift by querying the teacher on student-generated rollouts (just so "train–test mismatch" isn't read as system-level train–inference discrepancy)?

2 comments

Thank you again for your time and consideration. We have provided detailed responses and additional experimental results addressing the concerns raised in the reviews. We would greatly appreciate it if you could take another look at our responses and let us know if any concerns remain or if further clarification is needed. Thank you for your consideration.

Thanks for your rebuttal. However, my concerns remain, so I’ll keep my score.

Reviewer_FZCmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper builds on top of TRB (Trust-Region Behavior Blending) in order to "guide" the student rollouts towards the teacher, while staying within a prespecified constraint around the student. On top of TRB which find the maximum interpolation coefficient within the constraint, the authors introduce a probabilistic accept/reject mechanism based on what they call "maximal coupling". Accepted tokens use the standard reverse KL loss, while rejected tokens are supervised with the teachers most likely token. On top, they use speculative decoding to speed things up.

Critical strengths
  • The resulting method is elegant and well-motivated (if one buys the assumption/intuition that maximal coupling is a good idea to select supervision locations).
  • The idea of using speculative decoding to speed up the proposed approach is clever.
Critical weaknesses
  • The empirical gains compared to the strongest baselines are modest at best and it's not clear whether they are statically significant. Suspiciously, even though Random-TM and TV-Weighted-TM are perhaps the closest baselines (in addition to TRB on which the proposed method builds) they are not included in the main results (Table 1 and 2). The paper does evaluate these baselines, but not throughly and only briefly compares to them in the main paper. Results such as only +0.76 Mean@8 and +0.73 Pass@8 for the .17B student are not convincing without considering random variability.
  • The paper makes the point that maximal coupling minimises interventions, but it is not clear at all why this is something desirable, and why this should be exactly the positions where we need the alternative ("stronger") supervision.
  • Since the paper used Base models as students, they have terrible starting performance. This is fine, as it can be an interesting and challenging scenario, but it's not clear how much the gain in performance is from better reasoning vs. better format following (i.e. learning the task and answer format). The paper does not disentangle those effects.
  • The paper is quite difficult to read on a first try, despite the simplicity of the main ideas. There is also a general lack of polish. The exposition in the introduction relies on terminology that is ether never formally defined or only introduced much later. The key idea of maximal coupling is not explained as clearly or intuitively as it could be. In particular the presentation also makes it seem more complicated than necessary. The delineation from TRB is also not very sharp. Several derivation of standard results are included but it's not clear what purpose they serve exactly.
Questions for authors
  1. Can you do a more comprehensive comparison with Random-TM and TV-Weighted-TM? Including reporting results across multiple seeds and different student sizes.
  2. Can you explain why supervising the rejected positions leads to better learning?
  3. Can you evaluate with non "-Base" teachers and also non "-Base" student-teacher pairs?
  4. Can you disentangle improvement in reasoning vs. instruction following?
2 comments

Thank you again for your time and consideration. We have provided detailed responses and additional experimental results addressing the concerns raised in the reviews. We would greatly appreciate it if you could take another look at our responses and let us know if any concerns remain or if further clarification is needed. Thank you for your consideration.

Thank you again for your time and thoughtful review. Please let us know if you have any other questions. If our responses well resolve your concerns, we would be grateful if you would consider raising your rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.