acceptodds
Under review as a conference paper at ICLR 2027

OPRD: On-Policy Representation Distillation

31 views

Abstract

On-policy distillation (OPD) is a key technique in LLM post-training, where the student generates responses and a frozen teacher provides next-token supervision at each prefix. Such guidance offers an opportunity to pass on more than correct answers, helping the student learn how the teacher reasons and behaves. Yet output-space OPD communicates only next-token probabilities: these tell the student what the teacher would say next, but matching those choices may not be enough to acquire its broader behavior. To provide richer guidance, we introduce **On-Policy Representation Distillation (OPRD)**, a framework that uses the teacher's already-computed hidden states to directly supervise the student on its own responses. When models share representation coordinates, we align their normalized states directly, an approach we call **OPRD-Vanilla**. However, distillation often involves teachers and students with different architectures, making direct comparison of their hidden states difficult. Luckily, the Platonic Representation Hypothesis suggests that different models can capture similar underlying structures despite expressing them in different coordinates. Motivated by this hypothesis, we develop **OPRD-Bridge** to first learn a low-rank mapping that makes their representations comparable and then distill through this fixed mapping. Our experiments show that *i)* in multi-teacher RL model merging, one of the most important real-world applications of OPD, OPRD-Vanilla combines mathematics and code specialists into one student that matches their combined average accuracy, exhibits teacher-like behavior such as concise responses, and achieves up to greater training-data efficiency; and *ii)* on open-ended tasks, where accuracy alone cannot capture abilities such as empathy, OPRD-Bridge substantially outperforms OPD-style methods in transferring these abilities from larger models. Together, these results show that aligning internal representations helps students learn both what teachers can solve and how they behave.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Representation Distillation, Knowledge Distillation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vPmFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OPRD distills teacher hidden states along student-generated trajectories. It directly aligns normalized representations for models with shared coordinates and uses a calibrated, frozen low-rank bridge for heterogeneous models. Results are strongest in same-backbone specialist merging and KardiaBench response usability, while cross-scale mathematics accuracy is mixed. The paper also reports layer, coefficient, rank, and data-budget studies and a single-teacher time/memory comparison.

Critical strengths
  1. Strong same-backbone merging results. OPRD approaches the routed teachers' aggregate performance and improves over the strongest reported output-space baselines. The preliminary time and memory measurements also support practical value for Vanilla.

  2. Useful empirical analysis. The paper studies layer choice, loss coefficients, training-data budgets, and bridge rank, with additional Qwen3 and cross-scale experiments. The method clearly separates direct alignment from heterogeneous transfer.

Critical weaknesses
  1. The incremental contribution is not fully isolated. Hidden-state distillation is established, but matched offline-representation and alternative bridge-training controls are missing. Without these and simple SFT or parameter-merging baselines, the necessity of the proposed combination remains unclear.

  2. The open-ended transfer claim needs stronger validation. Six output-space baselines have 100% response failure, and evaluation relies on one judge with a specific output format. The provided OPD launcher also defaults to head-only token-loss updates; the actual Table 3 configurations must be clarified before attributing the gap to representation supervision.

  3. The interpretation of the bridge-rank ablation is incomplete. Changing rank also changes the explicit 1/r scaling of the normalized representation loss, so the results do not isolate capacity from optimization scale. A controlled loss-scale comparison is needed to explain high-rank degradation.

  4. Robustness and efficiency claims exceed the reported coverage. Cross-scale mathematics gains are small or involve lower average accuracy, without reported multi-seed uncertainty. The 8× result concerns one-epoch prompt usage, and the full calibration cost of Bridge is not established by the Vanilla timing study.

Questions for authors
  1. How does OPRD compare with teacher-response SFT, parameter merging for shared-backbone teachers, and representation distillation on fixed trajectories under matched budgets?
  2. What is gained by freezing the bridge rather than jointly training or periodically recalibrating it? Does the rank trend persist with loss or gradient scales controlled?
  3. Were sampled/top-16 OPD baselines in Table 3 trained through the full backbone? Please provide resolved configurations and explain the six complete-failure outcomes after task-specific tuning.
  4. Does the KardiaBench content-quality advantage persist with independent judges or human assessment and format/length-controlled comparisons, alongside the original full-set results?
  5. Can the authors report multiple training seeds, and explain the differing accuracy outcomes across teacher–student pairs?
9 comments

Thank you for the careful review and valuable suggestions. We respond point by point below.

W1 & Q1: The incremental contribution is not fully isolated. Hidden-state distillation is established, but matched offline-representation and alternative bridge-training controls are missing. How does OPRD compare with teacher-response SFT, parameter merging for shared-backbone teachers, and representation distillation on fixed trajectories under matched budgets?

Response: We would first like to defend the central contribution of our work: it is not simply an application of existing representation-distillation methods to OPD. Rather, we explain why representation supervision matters specifically in the on-policy setting and make it work in the main settings of modern LLM distillation.

(1) Motivation and insight.

Since OPD emerged in 2024, almost every OPD method developed before our submission has continued to distill knowledge through logits. This choice was natural because previous LLM post-training methods, including SFT and RL, also train models through token-probability objectives. Over time, however, this convention formed a rigid mindset that post-training supervision should operate on token probabilities.

As a result, the missing idea was not simply that “we can also match features.” The more important yet underexplored question was: what does logit-level OPD fail to transfer? Standard OPD feeds the student’s rollout to the teacher but retains only the teacher’s next-token probabilities. These probabilities show what the teacher would say next, yet they do not fully reveal how the teacher understands the partial response, identifies mistakes, or decides how to continue.

Our experiments first show this problem clearly: output-space OPD improves accuracy but can leave the student’s behavior largely unchanged. The student may produce probabilities closer to the teacher’s while keeping its original response length or still performing poorly on open-ended tasks. It learns what token the teacher prefers, but not necessarily how the teacher processes the context.

OPRD provides a direct channel for this “dark knowledge.” The teacher’s hidden states capture how it understands the student’s rollout, so matching these states transfers information that logits miss. OPRD-Bridge further carries this knowledge across models with different sizes, architectures, or vocabularies.

(2) Methodology. Our work also makes substantial technical contributions. a. OPRD-Vanilla is simple, but it addresses an important practical setting. Modern post-training pipelines such as DeepSeekv4 [1], GLM-5 [2], and Instella-MoE [3] often produce several RL specialists from the same base model. Because these specialists share representation coordinates, OPRD-Vanilla can directly merge their capabilities and behavior into one student. Our experiments show that this simple method is effective and data-efficient.

b. OPRD-Bridge extends the idea beyond shared coordinates. We study whether representation-level OPD still works when the teacher and student differ in depth, width, model size, or tokenizer. Our results show that heterogeneous models can still share transferable representation structure, allowing teacher knowledge and behavior to pass across architectures rather than remaining tied to one model family.

c. OPRD differs from conventional feature distillation in where and how supervision is collected. Conventional methods match features on fixed offline data. OPRD instead follows the student’s changing rollout: the student generates a response, and the teacher provides representation targets on the prefixes the student actually visits. Therefore, OPRD can guide the student on its own mistakes as training evolves. Our contribution is not feature matching itself, but bringing it into on-policy LLM distillation and making it work within and across architectures.

Finally, to our knowledge, OPRD is the first representation-level OPD method and has already attracted broad community interest. Several teams told us that they had tried similar ideas but struggled to make them work or explain why they should work. This reminds us of the lesson from ResNet: making CNNs deeper was not difficult to imagine, but progress required identifying the real optimization barrier and finding a simple mechanism that overcame it. Likewise, representation matching may appear simple in hindsight, but the key contribution of OPRD is to identify the information bottleneck in logit-level OPD and show how representation supervision can overcome it across practical distillation settings.

Within a short period, OPRD has inspired a rapidly growing body of follow-up work across diverse settings and application domains:

  1. On-policy self-distillation: PR-OPD [4], SCOPE-OPSD [5], OPASD [6], and PHF [7];
  2. Agentic reinforcement learning: PR-OPD [4];
  3. Mechanism and transition distillation: OPMD [8], OPTD [9], and PHF [7];
  4. Relational supervision: SimOPD [10];
  5. Multi-teacher consolidation: Latent-MOPD [11];
  6. Collapse mitigation and representation extrapolation: LastOPD [12] and RIDE [13].

We respectfully hope that the reviewer recognizes OPRD’s pioneering contribution and broad impact.🥹

For the three OPRD-Vanilla controls raised by the reviewer, we conduct matched experiments: teacher-response SFT on routed teacher outputs, parameter merging of the two specialists’ deltas, and fixed-trajectory representation distillation on trajectories collected once from the initial student. All trainable methods use the same prompts, initialization, optimizer, update budget, sequence length, and response-token budget; parameter merging serves as a zero-training reference. Results follow the complete Table 1 evaluation protocol:

MethodAIMOA24M500Min.Oly.HEMBPPLCLCBID Avg.
Code teacher----------75.055.153.311.655.1
Math teacher77.146.677.244.156.3--------55.1
Initial student47.030.061.226.535.456.148.613.90.035.4
Teacher-response SFT74.740.075.443.449.873.249.645.60.750.3
Parameter merging73.533.366.834.952.170.150.446.10.047.5
Fixed-trajectory RD51.833.360.034.645.561.047.443.90.242.0
OPRD-Vanilla80.746.778.444.155.076.856.651.110.455.5

OPRD-Vanilla exceeds teacher-response SFT, parameter merging, and fixed-trajectory RD by 5.2, 8.0, and 13.5 ID-average points, respectively. SFT shows that teacher-generated answers alone do not explain the gain, while parameter merging shows that directly combining specialist weights is insufficient. Most importantly, the fixed-trajectory comparison isolates the benefit of refreshing representation supervision along the student’s evolving on-policy distribution. We have added the matched teacher-response SFT, parameter-merging, and fixed-trajectory representation-distillation controls in Appendix C.9.2 and Table 13 of the revised paper and highlighted them in blue.

References

[1] Xu, Anyi, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026.

[2] Zeng A, Lv X, Hou Z, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026.

[3] Liu J, Ranjan S, Mishra P, et al. Instella-MoE Technical Report. arXiv preprint arXiv:2609.00791, 2026.

[4] M. Li, J. Yang, Z. Fang, et al. PR-OPD: Privileged Representation On-Policy Self-Distillation for Agentic Reinforcement Learning. arXiv:2609.36642, 2026.

[5] Y. Chen, K. Wang, P. Li, et al. SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation. arXiv:2609.12579, 2026.

[6] S. H. Arib, R. Akter, I. N. Swapnil, et al. Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning. arXiv:2609.33200, 2026.

[7] Y. Li, M. Zhang, D. Shen, et al. PHF: Privileged Hidden Flow for On-Policy Self-Distillation. arXiv:2606.29340, 2026.

[8] Y. Shi, Z. Li, C. Li, et al. OPMD: On-Policy Mechanism Distillation Beyond Output Supervision. arXiv:2610.14022, 2026.

[9] R. You, Y. Li, T. Yu, et al. On-Policy Transition Distillation. arXiv:2610.28048, 2026.

[10] K. Tang, X. Zheng, H. Wang, et al. SimOPD: Relational Knowledge Transfer for On-Policy Distillation. arXiv:2610.23306, 2026.

[11] Z. Fang, S. Hong, J. Yang, et al. Latent-MOPD: Latent Multi-Teacher On-Policy Distillation. arXiv:2610.02381, 2026.

[12] J. Yang, Z. Fang, Z. Xu, et al. LastOPD: Taming Collapse in Latent On-Policy Distillation. arXiv:2609.28845, 2026.

[13] H. Li, M. J. Chen, W. Ren, et al. The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation. arXiv:2609.36484, 2026.

W2 & Q3: Were sampled/top-16 OPD baselines in Table 3 trained through the full backbone? Please provide resolved configurations and explain the six complete-failure outcomes after task-specific tuning.

Response: Yes. Every method in Table 3 updates the full backbone. The teacher is frozen; OPRD-Bridge additionally freezes the pretrained bridge. All methods use the same student initialization, training data, update budget, and decoding settings, ensuring a fully controlled and fair comparison.

MethodResolved method-specific configuration
OPD (Sampled)Sampled-token OPD
OPD (Top-16)Top-16 student support
ExOPDλ=1.25\lambda=1.25
AOPDk=32, τ=0k=32,\ \tau=0
EOPDk=16k=16, entropy threshold =0.8=0.8, coefficient =1=1
PowerOPDα=5\alpha=5
FiReOPDpercentile =20=20, α=β=1\alpha=\beta=1
TIDEk=16k=16, positive/negative retention =0.2=0.2, λ=1\lambda=1
OPRD-BridgeFrozen rank-64 bridge

All methods use the same student, teacher, 1,024 training prompts, batch size, learning rate, 8,192-token response budget, and evaluation protocol. Method-specific hyperparameters are selected on the validation set.

The reported failure rate is the union of length-limit termination and invalid output format; these categories may overlap:

MethodLength-limit termination ↓Format failure ↓Aggregate failure ↓
OPD (Sampled)21.942.242.8
OPD (Top-16)14.143.343.5
ExOPD100.027.3100.0
AOPD98.425.0100.0
EOPD100.025.1100.0
PowerOPD87.512.6100.0
FiReOPD100.025.0100.0
TIDE100.034.7100.0
OPRD-Bridge7.715.915.9

The six complete-failure outcomes are mainly caused by training-induced non-termination: their generations overwhelmingly continue to the 8,192-token limit. Because all methods share the same prompting, decoding, stopping conditions, and evaluation interface, this behavior cannot be attributed to an interface mismatch. We have added the full-backbone training clarification, resolved baseline configurations, and detailed failure diagnostics in Appendix C.9.3–C.9.4 and Tables 14–15 of the revised paper and highlighted them in blue.

W3 & Q2: What is gained by freezing the bridge rather than jointly training or periodically recalibrating it? Does the rank trend persist with loss or gradient scales controlled?

Response: Freezing the bridge keeps the representation coordinates fixed during Stage 2. Joint training allows the projector to absorb alignment error without improving the student backbone, while periodic recalibration creates a moving target and repeats the calibration cost.

Bridge strategyFail. ↓Tok. ↓Rel. ↑Flu. ↑Emp. ↑Per. ↑Safe. ↑Avg. ↑
Joint bridge–student training100.07953.620.320.320.320.320.320.3
Periodic recalibration22.9934.754.054.053.854.054.454.0
Frozen bridge (ours)15.9825.753.753.953.653.654.253.8

Following your suggestion, we remove the 1/r1/r factor and repeat the rank ablation under otherwise identical settings:

RankOriginal scaling ↑Fixed scale (without 1/r) ↑
455.254.8
1654.455.0
6453.855.4
12856.654.6
25647.143.7
51234.622.2
102421.220.4

The same trend persists without 1/r: ranks 4–128 remain effective, while performance deteriorates at ranks 256 and above. This confirms that the high-rank degradation is not caused by the original loss scaling. We have added comparisons of frozen, jointly trained, and periodically recalibrated bridges, together with the loss-scale-controlled rank ablation, in Appendix C.9.5 and Tables 17–18 of the revised paper and highlighted them in blue.

Reviewer_zNnPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies knowledge distillation through teacher hidden states on student-generated trajectories. OPRD-Vanilla directly matches normalized states at corresponding layers and positions when the models share their architecture, tokenizer, and initialization. OPRD-Bridge first constructs a low-rank comparison space using teacher PCA projections and learned student mappings, then freezes this interface during on-policy student training. The paper also analyzes conditional token-sampling noise and the extent to which output KL constrains hidden-state differences. Experiments cover single-teacher mathematical distillation, mathematics/code specialist consolidation, out-of-domain evaluation, and cross-scale transfer in mathematics and emotional support. The strongest results concern related specialists: OPRD-Vanilla achieves a nine-benchmark ID average of 55.5, compared with 52.3 for the strongest output-space baseline and 55.1 for the routed teacher reference.

Critical strengths
  1. The specialist-consolidation results demonstrate a practically relevant benefit. With a shared student initialization and domain routing, OPRD-Vanilla achieves a nine-benchmark mathematics/code average of 55.5, exceeding PowerOPD’s 52.3 and approaching the composite teacher’s 55.1. PRIME results support the same direction. These gains accompany response lengths closer to the teachers rather than simply longer generation.
  2. The method is simple, and the auxiliary experiments help assess its practical use. Direct normalized-state matching requires no additional task-specific prediction head, while the bridge separates interface calibration from student training and is discarded at deployment.
Critical weaknesses
  1. The open-ended transfer results do not sufficiently distinguish content improvements from improved generation stability. On KardiaBench, OPRD-Bridge raises the average quality score from sampled OPD’s 44.6 to 53.8 while reducing termination and formatting failures, which is practically valuable. However, six competing methods have a 100% aggregate failure rate, making it difficult to determine how much of the quality gap reflects content-level transfer versus the avoidance of generation degradation. Although the failure metric does not directly alter quality scores, repetition, truncation, and response extraction affect the text being judged. The current analysis therefore does not adequately establish the contribution of transferring content-level abilities such as empathy to the overall improvement. This limits the experiment’s support for the open-ended capability-transfer conclusion, rather than constituting merely an evaluation-detail concern.
  2. The comparative advantage in heterogeneous transfer is less firmly established than in consolidation under shared initialization, limiting the breadth of the contribution. The strongest results involve models with shared representation coordinates. Bridge demonstrates the feasibility of cross-scale transfer, but its accuracy advantages over output-space baselines are inconsistent: Table 2 reports micro accuracy of 43.5 versus ExOPD’s 43.3; in Table 12, its average Avg@16 is 11.2 versus sampled OPD’s 15.3, while both achieve 30.8 Best@16. Its principal advantages in the latter setting are shorter and less repetitive generations. Although the authors acknowledge some trade-offs, framework applicability should be distinguished more clearly from demonstrated comparative advantages. These findings establish useful cross-scale transfer, but do not demonstrate a consistent accuracy advantage over strong output-space baselines. Consequently, the breadth of OPRD-Bridge’s comparative advantage is less firmly established than that of specialist consolidation under shared initialization.
Questions for authors
  1. On KardiaBench, do the severe termination or formatting failures of six methods primarily arise from training-induced generation degradation or from interface mismatches involving output templates and stopping conditions? Can existing diagnostics distinguish these explanations? Additionally, when no response span can be extracted, what text is submitted to the judge, and how is it scored?
  2. For heterogeneous teacher–student settings, does the evidence most directly support improved task accuracy, improved generation behavior, or their joint improvement? What conditions delimit this conclusion, and how should it be reconciled with the mixed results in Tables 2 and 12? Furthermore, should learning how the teacher “reasons and behaves” be understood as closer observable output behavior, or also as transfer of internal processing? If the latter is intended, which analyses directly support it?
4 comments

Thank you for recognizing our work and for your careful review. We provide detailed responses below.

W1/Q1: On KardiaBench: (1) are the termination and formatting failures caused by training degradation or interface mismatch; (2) which diagnostics distinguish these causes; and (3) when no response span is extracted, what is judged and how is it scored?

Response: We address the three questions in order. We have added these explanations and diagnostics in Appendix C.9.3–C.9.4 and Tables 14–16 of the revised paper and highlighted them in blue.

(1) Cause of failure. All distilled models use the same student architecture and tokenizer, KardiaBench prompt template, response markers, decoding settings, stopping rules, and 8,192-token generation budget. The large differences across methods therefore come from training-induced generation degradation rather than interface mismatch. We observe two main failure modes: some models generate malformed response structures, while others repeatedly generate until reaching the length limit.

(2) Generation and content diagnostics.

MethodNon-stop ↓Format fail. ↓Any fail. ↓Missing span ↓Tokens ↓Quality: All ↑Quality: Extractable ↑
Kardia teacher0.218.718.714.4182.071.580.2
Initial student9.450.850.938.11,454.238.349.6
OPD (Sampled)21.942.242.826.42,278.344.653.4
OPD (Top-16)14.143.343.528.31,729.043.853.2
ExOPD100.027.3100.019.56,724.220.320.4
AOPD98.425.0100.022.18,064.420.620.8
EOPD100.025.1100.023.48,191.820.020.0
PowerOPD87.512.6100.012.07,196.622.422.7
FiReOPD100.025.0100.023.88,192.020.020.0
TIDE100.034.7100.032.68,192.020.420.6
OPRD-Bridge7.715.915.913.8825.753.859.2

Non-stop means that generation reaches the maximum length without normal termination. Any failure means that the response either fails to stop or violates the required format; these two failures may overlap. Missing span means that no nonempty final response can be extracted from the designated response markers.

The results distinguish the failure patterns clearly. Sampled and top-16 OPD usually stop, but often produce invalid response structures. The other six output-space methods frequently generate until the length limit. OPRD-Bridge reduces both problems, lowering overall failure to 15.9% and average response length to 825.7 tokens.

We also separate content quality from response extraction. Quality: All evaluates the entire test set, where a missing response span receives the minimum score of 20/100. Quality: Extractable is the average judge score computed only over examples with a successfully extracted response; missing-span examples are excluded. OPRD-Bridge still achieves the highest quality on extractable responses: 59.2, compared with 53.4 for sampled OPD and 53.2 for top-16 OPD. It also reduces their missing-span rates from 26.4% and 28.3% to 13.8%. Its improvement therefore reflects both stronger content transfer and more reliable generation.

(3) Missing response spans. When a nonempty response span can be extracted, it is submitted to Qwen3-8B and scored normally, even if the complete generation has formatting or termination problems. When no response span can be extracted, the malformed full output is not submitted to the judge and receives the minimum score of 20/100.

W2/Q2: In heterogeneous teacher–student settings: (1) does the evidence support improved task accuracy, generation behavior, or both; (2) under what conditions, and how should the mixed results in Tables 2 and 12 be reconciled; and (3) does “reasoning and behavior” refer only to observable outputs or also to internal processing, and what evidence supports the latter?

Response: Thank you for these careful questions. The evidence supports a joint but task-dependent improvement: OPRD-Bridge transfers task capability while improving how that capability is expressed during generation. We have added these explanations in Section 3.2, Appendix C.9.1 and C.9.6, Figure 5, and Tables 19–20 of the revised paper, with all additions highlighted in blue.

(1) What improves? In Table 2, OPRD-Bridge achieves the highest micro accuracy, 43.5%, slightly above ExOPD’s 43.3%, demonstrating that representation transfer can improve task accuracy across model scales. On KardiaBench, it improves both extractable content quality and generation reliability. In Table 12, its main advantage is generation behavior: compared with sampled OPD, it preserves the same Best@16 accuracy (30.8%) while producing responses that are substantially shorter (5,333 vs. 16,020 tokens) and more diverse (20.4 vs. 9.4 Dist-4g).

(2) Why are Tables 2 and 12 different? The two tables evaluate different model pairs and different aspects of performance. Table 2 uses deterministic benchmark accuracy, whereas Table 12 evaluates 16 sampled responses and separates average accuracy, best-of-16 capability, diversity, and length. OPRD-Bridge preserves the same best-of-16 capability in Table 12, but does not exceed sampled OPD in Avg@16 (11.2 vs. 15.3). Continuing OPD from the OPRD-Bridge checkpoint then raises Best@16 to the best result, 33.9%, while retaining much shorter and more diverse generations. This shows that representation and output supervision are complementary: the bridge establishes a useful generation regime, while output-space OPD further sharpens token-level accuracy.

The relevant condition is whether the teacher and student contain a stable shared low-rank representation space. We measure this using held-out Stage-1 alignment before student training and select the bridge rank without downstream test feedback. When this alignment is strong, OPRD-Bridge supports cross-scale transfer.

(3) What does “reasoning and behavior” mean? It refers to both: OPRD directly aligns the student’s internal representations with the teacher’s at student-visited prefixes, and this internal transfer is reflected in the student’s observable generation behavior. Internal transfer is evidenced by the decreasing teacher–student representation distance during Stage 2, 98.4% held-out bridge alignment, and bridge-rank ablation. Behavioral transfer is measured independently through accuracy, response length, diversity, termination, formatting, and response quality.

Thanks for your response. I'll keep my positive score.

Thank you so much for your thoughtful follow-up and continued support! We sincerely appreciate your constructive comments, which have strengthened our analysis and experimental evidence, and we will ensure that all clarifications and additional results are clearly incorporated into the final manuscript. Your suggestions really helped us make this a better paper!

Reviewer_C2N7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies OPD. The paper argues that the current output-space OPD communicates only next-token probabilities is not good solution. The standard OPD methods tell the student what the teacher would say next, but matching those choices may not be enough to acquire its broader behavior. Thus the paper proposes new methods OPRD-Vanilla and OPRD-Bridge. They can provide richer guidance. The proposed On-Policy Representation Distillation (OPRD) is to use the teacher’s already-computed hidden states to directly supervise the student on its own responses. OPRD-Bridge is to learn a low-rank bridge between teacher and student representations while keeping both models frozen. The paper provides experiments on 24 task datasets. The results show that achieves up to 8× greater training-data efficiency and on open-ended tasks OPRD-Bridge substantially outperforms OPD-style methods in transferring these abilities from larger models.

Critical strengths

The motivation using internal representation is clear.

The method On-Policy Representation Distillation (OPRD) is interesting.

The paper provides multiple experiments to support the idea.

Critical weaknesses

The paper lacks the comparison with feature-based distillation methods.

For OPRD-Bridge, there are two stages. It would be better to provide more explanation about Calibration set.

Theory part may be not strong. For example, in the Theorem 2, it would be better to explain the assumption about τ∈(0,1/3)\tau \in (0, 1/3).

Questions for authors

Figure 5 shows r = 64 works best. How about different models? Do they need different rrs?

Is there any other work about feature-based distillation?

Are there any weaknesses to this method? Like, does the student always just follow the teacher, even when the teacher makes mistakes?

6 comments

Thank you for the careful review and valuable suggestions. We respond point by point below.

W1/Q2: The paper lacks the comparison with feature-based distillation methods.

Response: We would first like to emphasize that OPRD is not a simple adaptation of feature distillation: it explains why representations matter for OPD and makes them work in practical LLM distillation settings.

(1) Motivation and Insight.

Since OPD emerged in 2024, almost every OPD method developed before our submission has continued to distill knowledge through logits. This choice was natural because previous LLM post-training methods, including SFT and RL, also train models through token-probability objectives. Over time, however, this convention formed a rigid mindset that post-training supervision should operate on token probabilities.

As a result, the missing idea was not simply that “we can also match features.” The more important yet underexplored question was: what does logit-level OPD fail to transfer? Standard OPD feeds the student’s rollout to the teacher but retains only the teacher’s next-token probabilities. These probabilities show what the teacher would say next, yet they do not fully reveal how the teacher understands the partial response, identifies mistakes, or decides how to continue.

Our experiments first show this problem clearly: output-space OPD improves accuracy but can leave the student’s behavior largely unchanged. The student may produce probabilities closer to the teacher’s while keeping its original response length or still performing poorly on open-ended tasks. It learns what token the teacher prefers, but not necessarily how the teacher processes the context.

OPRD provides a direct channel for this “dark knowledge.” The teacher’s hidden states capture how it understands the student’s rollout, so matching these states transfers information that logits miss. OPRD-Bridge further carries this knowledge across models with different sizes, architectures, or vocabularies.

(2) Methodology. Our work also makes substantial technical contributions. a. OPRD-Vanilla is simple, but it addresses an important practical setting. Modern post-training pipelines such as DeepSeekv4 [1], GLM-5 [2], and Instella-MoE [3] often produce several RL specialists from the same base model. Because these specialists share representation coordinates, OPRD-Vanilla can directly merge their capabilities and behavior into one student. Our experiments show that this simple method is effective and data-efficient.

b. OPRD-Bridge extends the idea beyond shared coordinates. We study whether representation-level OPD still works when the teacher and student differ in depth, width, model size, or tokenizer. Our results show that heterogeneous models can still share transferable representation structure, allowing teacher knowledge and behavior to pass across architectures rather than remaining tied to one model family.

c. OPRD differs from conventional feature distillation in where and how supervision is collected. Conventional methods match features on fixed offline data. OPRD instead follows the student’s changing rollout: the student generates a response, and the teacher provides representation targets on the prefixes the student actually visits. Therefore, OPRD can guide the student on its own mistakes as training evolves. Our contribution is not feature matching itself, but bringing it into on-policy LLM distillation and making it work within and across architectures.

Finally, to our knowledge, OPRD is the first representation-level OPD method and has already attracted broad community interest. Several teams told us that they had tried similar ideas but struggled to make them work or explain why they should work. This reminds us of the lesson from ResNet: making CNNs deeper was not difficult to imagine, but progress required identifying the real optimization barrier and finding a simple mechanism that overcame it. Likewise, representation matching may appear simple in hindsight, but the key contribution of OPRD is to identify the information bottleneck in logit-level OPD and show how representation supervision can overcome it across practical distillation settings.

Within a short period, OPRD has inspired a rapidly growing body of follow-up work across diverse settings and application domains:

  1. On-policy self-distillation: PR-OPD [4], SCOPE-OPSD [5], OPASD [6], and PHF [7];
  2. Agentic reinforcement learning: PR-OPD [4];
  3. Mechanism and transition distillation: OPMD [8], OPTD [9], and PHF [7];
  4. Relational supervision: SimOPD [10];
  5. Multi-teacher consolidation: Latent-MOPD [11];
  6. Collapse mitigation and representation extrapolation: LastOPD [12] and RIDE [13].

These follow-up studies establish representation-level on-policy supervision, initiated by OPRD, as a distinct and productive research direction. We respectfully hope that the reviewer recognizes OPRD’s pioneering contribution and broad impact.🥹

References

[1] Xu, Anyi, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026.

[2] Zeng A, Lv X, Hou Z, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026.

[3] Liu J, Ranjan S, Mishra P, et al. Instella-MoE Technical Report. arXiv preprint arXiv:2609.00791, 2026.

[4] M. Li, J. Yang, Z. Fang, et al. PR-OPD: Privileged Representation On-Policy Self-Distillation for Agentic Reinforcement Learning. arXiv:2609.36642, 2026.

[5] Y. Chen, K. Wang, P. Li, et al. SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation. arXiv:2609.12579, 2026.

[6] S. H. Arib, R. Akter, I. N. Swapnil, et al. Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning. arXiv:2609.33200, 2026.

[7] Y. Li, M. Zhang, D. Shen, et al. PHF: Privileged Hidden Flow for On-Policy Self-Distillation. arXiv:2606.29340, 2026.

[8] Y. Shi, Z. Li, C. Li, et al. OPMD: On-Policy Mechanism Distillation Beyond Output Supervision. arXiv:2610.14022, 2026.

[9] R. You, Y. Li, T. Yu, et al. On-Policy Transition Distillation. arXiv:2610.28048, 2026.

[10] K. Tang, X. Zheng, H. Wang, et al. SimOPD: Relational Knowledge Transfer for On-Policy Distillation. arXiv:2610.23306, 2026.

[11] Z. Fang, S. Hong, J. Yang, et al. Latent-MOPD: Latent Multi-Teacher On-Policy Distillation. arXiv:2610.02381, 2026.

[12] J. Yang, Z. Fang, Z. Xu, et al. LastOPD: Taming Collapse in Latent On-Policy Distillation. arXiv:2609.28845, 2026.

[13] H. Li, M. J. Chen, W. Ren, et al. The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation. arXiv:2609.36484, 2026.

W2: For OPRD-Bridge, there are two stages. It would be better to provide more explanation about Calibration set.

Response: Thank you for the suggestion. The calibration set introduces no external data: it is randomly sampled from the same training prompt pool used in Stage 2 and by all baselines. We use 2,000 DAPO-Math prompts for Skywork-to-DeepSeek and 1,024 KardiaBench prompts for Kardia-to-DeepSeek, each split 90/10 for fitting and validation. Hence, the experimental setup is fair to all baselines in terms of data access. The calibration set is used only to learn the shared representation space; Stage 2 trains the student on fresh on-policy rollouts. We have added these explanations in Appendix C.9.1 of the revised paper and highlighted them in blue.

W3: Theory part may be not strong. For example, in the Theorem 2, it would be better to explain the assumption about τ∈(0,1/3)\tau \in (0, 1/3).

Response: Thank you for the suggestion. The complete proof is provided under Appendix A, “Global KL-to-hidden conditioning.” Let pip_i and pi′p'_i denote the teacher and student probabilities assigned to token ii, respectively. The assumption means that both output distributions share a positive probability floor:

For finite logits, softmax assigns every token positive probability, so τ>0\tau>0. Since each distribution sums to one,

for any vocabulary with more than three tokens. Thus, τ∈(0,1/3)\tau\in(0,1/3) is automatically satisfied for modern LLMs and imposes no additional modeling restriction. The threshold 1/31/3 is used only to obtain the simplified coefficient in the proof. In practice, τ\tau can be extremely small, making the resulting KL-to-hidden-state bound very loose. This exposes the limited control of logit distillation over hidden representations and motivates OPRD’s direct representation supervision. We have added these explanations in Theorem 2 of the revised paper and highlighted them in blue.

Q1: Figure 5 shows r = 64 works best. How about different models? Do they need different rrs?

Response: The optimal rank is model-pair-dependent because it reflects the dimensionality of their shared representation subspace. We select it using held-out Stage-1 alignment, without Stage-2 training or test-set feedback. This yields r=8r=8 for Skywork-to-DeepSeek and r=64r=64 for Kardia-to-DeepSeek. Stage 1 accounts for only about 10% of the total runtime, enabling rapid rank selection without running the substantially more expensive Stage 2. Figure 5 further shows a broad effective range of moderate ranks, making this selection stable and inexpensive. We have added these explanations in the relevant paragraph on page 6 of the revised paper and highlighted them in blue.

Q3: Are there any weaknesses to this method? Like, does the student always just follow the teacher, even when the teacher makes mistakes?

Response: OPD is less prone to blind teacher imitation because it trains on the student’s own state distribution rather than fixed teacher trajectories, reducing train–inference mismatch and catastrophic forgetting [14, 15]. OPRD preserves this property: the student generates its own trajectories, while the teacher provides representation guidance only at student-visited prefixes. Accordingly, Table 1 shows that OPRD-Vanilla achieves an OOD average of 38.7, exceeding the math teacher’s 37.0, while OPRD-Bridge reaches 38.9.

Of course, if the teacher is consistently wrong, the student may also learn those mistakes. This applies to any distillation method, which is why the teacher is normally chosen to be stronger than the student in the target domain.

[14] Agarwal et al., On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024.
[15] Shenfeld et al., Self-Distillation Enables Continual Learning, ICML 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers, AC, SAC, and PC,

We sincerely thank all reviewers for their constructive feedback. We appreciate their recognition of the clear motivation and interesting approach (R1), strong specialist-consolidation results (R2, R3), and useful empirical analyses (R2, R3).

Our rebuttal further strengthens these findings. We have addressed the reviewers’ questions through the following updates:

  • [R1] Clarified the theoretical analysis, calibration data, and bridge-rank selection (Section 2.2; Appendix C.9.1).
  • [R3] Added matched comparisons supporting the value of on-policy representation supervision (Table 13).
  • [R2, R3] Strengthened the open-ended transfer evaluation with generation diagnostics, extractable-response analysis, and an independent judge (Tables 14–16).
  • [R3] Expanded bridge ablations, multiple-seed experiments, and data-efficiency and training-cost analyses (Tables 17–22).

These additions reinforce the effectiveness of both OPRD variants in their main applications. Following the rebuttal, R2 explicitly reaffirmed their positive recommendation.

OPRD’s contribution also extends beyond the reported experiments. To our knowledge, OPRD is the first representation-level OPD method. It identifies the limitations of logit-level supervision and makes representation supervision effective in practical distillation settings, both within shared architectures and across heterogeneous models.

Within a short period, OPRD has inspired follow-up work across diverse settings, as detailed in our individual responses:

  • On-policy self-distillation: PR-OPD, SCOPE-OPSD, OPASD, and PHF.
  • Agentic reinforcement learning: PR-OPD.
  • Mechanism and transition distillation: OPMD, OPTD, and PHF.
  • Relational supervision: SimOPD.
  • Multi-teacher consolidation: Latent-MOPD.
  • Collapse mitigation and representation extrapolation: LastOPD and RIDE.

These follow-ups demonstrate the broader value of representation-level on-policy supervision as a distinct and productive research direction. Representation matching may look simple in hindsight, but identifying why it matters and making it work across practical settings are central contributions of OPRD.

We sincerely hope the reviewers and AC will consider the strong empirical evidence, the questions addressed during rebuttal, and OPRD’s pioneering contribution and broader impact in their overall assessment.

R1, R2, and R3 refer to reviewers C2N7, zNnP, and vPmF, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.