acceptodds
Under review as a conference paper at ICLR 2027

Mini-OPD: On-Policy Relation Distillation

Abstract

On-policy distillation (OPD) offers a natural remedy for exposure bias: rather than imitating a teacher on static data, a student is supervised on states it visits itself. Yet making the states on-policy does not make the supervision richer. Existing OPD still observes the teacher mainly through its next-token distribution, leaving the context processing behind that prediction largely underdetermined—especially after the student's trajectory has gone wrong. We introduce Mini-OPD, which extends on-policy action matching to on-policy state-processing alignment. Alongside output reverse KL, it distills response-level Q–Q, K–K, and V–V self-relations from one fixed teacher–student layer pair on the student's realized response. These relation rows inhabit a shared token-position simplex, enabling heterogeneous models to be compared without learned feature bridges when positions are aligned. The objective admits either KL order and uses the teacher-weighted branch by default; both targets use the same student rollout. Across AIME 2024, AMC, OlympiadBench, LiveCodeBench, and SWE-Bench, Mini-OPD is the strongest student method, improving over matched output-only OPD by 1.1–4.0 points on every evaluation. In a representative run, it reaches the low-divergence regime 3.41× earlier and attains a 66% lower reverse KL. These results suggest that correcting where a student learns is only part of on-policy distillation; enriching what it learns at those states matters as well.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KcyrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of relying solely on output-space reverse KL divergence in on-policy distillation (OPD). Existing OPD methods typically use a stopped-gradient student snapshot for rollouts and compute output reverse KL divergence against the teacher's next-token distribution on student-generated response prefixes. The authors argue that such output-level supervision primarily constrains token distributions but does not explicitly transfer structured relationships among response states, particularly the attention and representation geometry encoded by QKV relations. To address this limitation, the paper proposes Mini-OPD, which introduces a lightweight relational transfer term in addition to output KL. Specifically, the method aligns teacher and student QKV relations at a selected response position within a selected pair of response layers. The overall objective can be summarized as a combination of output-space distillation and process-space relational transfer, while restricting the additional supervision to a single response-layer pair and a single response position to reduce training overhead. The experimental evaluation compares Mini-OPD with relevant baselines, including OPD, OPRD-Bridge, and feature-projector-based methods, on mathematical reasoning tasks. The paper also employs common-response diagnostics and a full layer sweep to analyze cross-layer alignment patterns and sensitivity to layer selection.

Critical strengths

The paper accurately identifies a limitation of output-level OPD: it supervises the teacher's next-token distribution without explicitly modeling the relational structure among response states. Extending supervision from the output token space to QKV relations and the process space is a natural and meaningful direction. Rather than adopting full-layer projection or comprehensive OPRD-style alignment, Mini-OPD performs relational transfer using only a single response-layer pair and a single response position. This design is more lightweight than dense feature alignment and is better suited to practical LLM post-training scenarios.

Critical weaknesses

The core of Mini-OPD can be summarized as "output-level OPD + response-level QKV relational transfer." However, substantial prior work already exists on internal representation distillation, attention/relation distillation, and sequence-level distillation. The causal attribution of the reported performance gains remains insufficiently established. The observed improvements could potentially arise from several factors, including additional response-level supervision, teacher noise filtering, longer training time, position matching and layer-level regularization. End-to-end performance improvements alone are insufficient to demonstrate that the gains primarily originate from QKV relational transfer itself. The paper would benefit from more explicit relation-specific ablation studies, such as shuffled relations, random layer pairs, shuffled token positions, and a projector-only baseline.

Questions for authors

Are the performance improvements of Mini-OPD over OPD and OPRD-Bridge still statistically significant across multiple random seeds? Are confidence intervals or variance estimates reported? Were OPD, OPRD-Bridge, and feature-projector-based baselines compared under matched computational budgets?

Reviewer_asStOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose Mini-OPD, which augments on-policy distillation (OPD) with an auxiliary MiniLMv2-style relation loss. At a single teacher–student layer pair, temperature-scaled cosine Q–Q, K–K, and V–V self-relations over the student-generated response are matched with forward KL, alongside the standard output reverse KL. Specifically, the paper shows that under direct position alignment with unit-normalized rows and a shared temperature, matching the relation distributions is equivalent to matching the within-response cosine Gram matrix, and that this target is invariant to per-relation-head orthogonal transforms. Additionally, Mini-OPD improves over OPD by 1.0–4.2 points on five math, code, and SWE-Bench benchmarks with same-width Qwen3-4B pairs, and by 0.7–2.6 points in a DeepSeek-R1-Distill 32B→7B setting. It also improves over SimpleOPD by 0.8–3.7 points for Qwen3-4B→Llama-3.2-3B.

Critical strengths

1. Simple, projector-free plug-in: The relation target lives on the token-position simplex, so it compares teacher and student with unequal widths without a learned bridge. It adds no teacher rollouts or inference-time modules, which makes it easy to bolt onto existing OPD pipelines.

2. Broad empirical coverage: Evaluation spans mathematics, code, agentic SWE-Bench, a 32B→7B size gap, and a cross-family/cross-tokenizer pair. Mini-OPD improves every reported primary metric over its output-only baseline in all three settings.

3. Clean formalization of what is transferred: Proposition 1, the high-temperature expansion, and the softmax-gradient analysis state precisely which quantity the loss encodes. The cross-tokenizer transport is a well-defined, mass-preserving interface for position mismatch.

Critical weaknesses

1. Why do we need to distill relations? The paper lack preliminary study or any Insights about what knowledge the within-response Q/K/V relations carry that output KL fails to transfer. Proposition 1 shows only what the loss encodes, not why preserving this angular geometry should benefit the student’s reasoning. The downside metrics improvement only is hard to distinguish it from helpful regularization.

2. What distinguishes Mini-OPD from MiniLMv2 beyond moving it on-policy? The re-split relation heads, Q–Q/K–K/V–V self-relations, teacher-to-student KL, and single-layer transfer are all inherited from MiniLMv2. Since the paper does not explain why relational supervision is specifically needed in the OPD setting (e.g., which failure of output-only OPD on student-generated prefixes it addresses), the method reads as an incremental transplant of MiniLMv2 into OPD, i.e., an auxiliary regularization term.

3. Are the baselines and ablations sufficient to isolate the contribution? Apart from output-only OPD, the only internal-supervision baseline is OPRD, which the authors concede is not a loss-level controlled comparison. Mini-OPD should be compared against other OPD variants that add auxiliary regularization or internal supervision to show that the relational target is preferable to alternative regularizers. Moreover, the individual effects of the Q–Q, K–K, and V–V relations are not studied, so it remains unclear which component drives the gains and whether all three, inherited directly from MiniLMv2, are necessary.

Questions for authors

Please check Weaknesses, especially the point1 and point 2.

Reviewer_dt2qOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether on-policy distillation (OPD) can transfer more than the teacher’s next-token distribution—specifically, whether it can also convey relations among the internal states of student-generated responses. The authors propose Mini-OPD, which adds a loss matching Q–Q, K–K, and V–V relation distributions computed by the teacher and student on the same student-generated response. The method matches relations at one selected layer pair and does not require a learned projector for hidden states of different widths. The authors also discuss a position-mapping approach for models with different tokenizers. In domain-specific 4B experiments, Mini-OPD improves all five primary metrics over OPD across mathematics, coding, and software engineering. The paper also reports gains when distilling from a 32B teacher to a 7B student and in cross-tokenizer distillation from Qwen3 to Llama. A code diagnostic shows a further 54–56% reduction in residual relation KL. These results support the usefulness of relation supervision in the tested settings, but do not establish its general applicability, practical training-speed gains, or a causal mechanism for the performance improvements.

Critical strengths
  1. Consistent point-estimate gains across multiple tasks and distillation settings. In the 4B experiments, Mini-OPD outperforms OPD on all five primary metrics across mathematics, coding, and SWE-Bench. Experiments with a larger teacher and across model families also show positive results, suggesting that the gains are not confined to a single benchmark.

  2. The method has a clearly defined supervision target and a geometric interpretation. The teacher and student process the same student-generated response, and the relation loss operates on Q/K/V relations between response-token positions rather than directly aligning hidden-state coordinates. The authors also examine layer sensitivity, computational cost, and limitations of cross-tokenizer alignment.

Critical weaknesses
  1. The uncertainty of the results is not adequately quantified. Most headline results are single-run point estimates, without independent training seeds or confidence intervals. Some benchmarks are small, and several metrics rely on multiple generations through Avg@N/pass@N. It is therefore difficult to determine whether the reported gains would remain stable across runs or evaluation samples.

  2. The cross-tokenizer results are difficult to verify, and the computational cost is not fully compared. The appendix notes that some cross-tokenizer experiments lack details such as model versions, layer extraction, alignment coverage, and evaluation procedures. This makes it difficult to determine whether the gains can be attributed to the relation loss itself. Mini-OPD also adds feature-extraction forward passes and relation computation that scales quadratically with response length. The paper clarifies that faster convergence in training steps does not imply wall-clock speedup, but does not provide a thorough comparison under matched compute budgets.

Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.