- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OPRD distills teacher hidden states along student-generated trajectories. It directly aligns normalized representations for models with shared coordinates and uses a calibrated, frozen low-rank bridge for heterogeneous models. Results are strongest in same-backbone specialist merging and KardiaBench response usability, while cross-scale mathematics accuracy is mixed. The paper also reports layer, coefficient, rank, and data-budget studies and a single-teacher time/memory comparison.
Critical strengths
-
Strong same-backbone merging results. OPRD approaches the routed teachers' aggregate performance and improves over the strongest reported output-space baselines. The preliminary time and memory measurements also support practical value for Vanilla.
-
Useful empirical analysis. The paper studies layer choice, loss coefficients, training-data budgets, and bridge rank, with additional Qwen3 and cross-scale experiments. The method clearly separates direct alignment from heterogeneous transfer.
Critical weaknesses
-
The incremental contribution is not fully isolated. Hidden-state distillation is established, but matched offline-representation and alternative bridge-training controls are missing. Without these and simple SFT or parameter-merging baselines, the necessity of the proposed combination remains unclear.
-
The open-ended transfer claim needs stronger validation. Six output-space baselines have 100% response failure, and evaluation relies on one judge with a specific output format. The provided OPD launcher also defaults to head-only token-loss updates; the actual Table 3 configurations must be clarified before attributing the gap to representation supervision.
-
The interpretation of the bridge-rank ablation is incomplete. Changing rank also changes the explicit 1/r scaling of the normalized representation loss, so the results do not isolate capacity from optimization scale. A controlled loss-scale comparison is needed to explain high-rank degradation.
-
Robustness and efficiency claims exceed the reported coverage. Cross-scale mathematics gains are small or involve lower average accuracy, without reported multi-seed uncertainty. The 8× result concerns one-epoch prompt usage, and the full calibration cost of Bridge is not established by the Vanilla timing study.
Questions for authors
- How does OPRD compare with teacher-response SFT, parameter merging for shared-backbone teachers, and representation distillation on fixed trajectories under matched budgets?
- What is gained by freezing the bridge rather than jointly training or periodically recalibrating it? Does the rank trend persist with loss or gradient scales controlled?
- Were sampled/top-16 OPD baselines in Table 3 trained through the full backbone? Please provide resolved configurations and explain the six complete-failure outcomes after task-specific tuning.
- Does the KardiaBench content-quality advantage persist with independent judges or human assessment and format/length-controlled comparisons, alongside the original full-set results?
- Can the authors report multiple training seeds, and explain the differing accuracy outcomes across teacher–student pairs?
9 comments
Within a short period, OPRD has inspired a rapidly growing body of follow-up work across diverse settings and application domains:
- On-policy self-distillation: PR-OPD [4], SCOPE-OPSD [5], OPASD [6], and PHF [7];
- Agentic reinforcement learning: PR-OPD [4];
- Mechanism and transition distillation: OPMD [8], OPTD [9], and PHF [7];
- Relational supervision: SimOPD [10];
- Multi-teacher consolidation: Latent-MOPD [11];
- Collapse mitigation and representation extrapolation: LastOPD [12] and RIDE [13].
We respectfully hope that the reviewer recognizes OPRD’s pioneering contribution and broad impact.🥹
For the three OPRD-Vanilla controls raised by the reviewer, we conduct matched experiments: teacher-response SFT on routed teacher outputs, parameter merging of the two specialists’ deltas, and fixed-trajectory representation distillation on trajectories collected once from the initial student. All trainable methods use the same prompts, initialization, optimizer, update budget, sequence length, and response-token budget; parameter merging serves as a zero-training reference. Results follow the complete Table 1 evaluation protocol:
Method AIMO A24 M500 Min. Oly. HE MBPP LC LCB ID Avg. Code teacher -- -- -- -- -- 75.0 55.1 53.3 11.6 55.1 Math teacher 77.1 46.6 77.2 44.1 56.3 -- -- -- -- 55.1 Initial student 47.0 30.0 61.2 26.5 35.4 56.1 48.6 13.9 0.0 35.4 Teacher-response SFT 74.7 40.0 75.4 43.4 49.8 73.2 49.6 45.6 0.7 50.3 Parameter merging 73.5 33.3 66.8 34.9 52.1 70.1 50.4 46.1 0.0 47.5 Fixed-trajectory RD 51.8 33.3 60.0 34.6 45.5 61.0 47.4 43.9 0.2 42.0 OPRD-Vanilla 80.7 46.7 78.4 44.1 55.0 76.8 56.6 51.1 10.4 55.5 OPRD-Vanilla exceeds teacher-response SFT, parameter merging, and fixed-trajectory RD by 5.2, 8.0, and 13.5 ID-average points, respectively. SFT shows that teacher-generated answers alone do not explain the gain, while parameter merging shows that directly combining specialist weights is insufficient. Most importantly, the fixed-trajectory comparison isolates the benefit of refreshing representation supervision along the student’s evolving on-policy distribution. We have added the matched teacher-response SFT, parameter-merging, and fixed-trajectory representation-distillation controls in Appendix C.9.2 and Table 13 of the revised paper and highlighted them in blue.
References
[1] Xu, Anyi, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026.
[2] Zeng A, Lv X, Hou Z, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026.
[3] Liu J, Ranjan S, Mishra P, et al. Instella-MoE Technical Report. arXiv preprint arXiv:2609.00791, 2026.
[4] M. Li, J. Yang, Z. Fang, et al. PR-OPD: Privileged Representation On-Policy Self-Distillation for Agentic Reinforcement Learning. arXiv:2609.36642, 2026.
[5] Y. Chen, K. Wang, P. Li, et al. SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation. arXiv:2609.12579, 2026.
[6] S. H. Arib, R. Akter, I. N. Swapnil, et al. Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning. arXiv:2609.33200, 2026.
[7] Y. Li, M. Zhang, D. Shen, et al. PHF: Privileged Hidden Flow for On-Policy Self-Distillation. arXiv:2606.29340, 2026.
[8] Y. Shi, Z. Li, C. Li, et al. OPMD: On-Policy Mechanism Distillation Beyond Output Supervision. arXiv:2610.14022, 2026.
[9] R. You, Y. Li, T. Yu, et al. On-Policy Transition Distillation. arXiv:2610.28048, 2026.
[10] K. Tang, X. Zheng, H. Wang, et al. SimOPD: Relational Knowledge Transfer for On-Policy Distillation. arXiv:2610.23306, 2026.
[11] Z. Fang, S. Hong, J. Yang, et al. Latent-MOPD: Latent Multi-Teacher On-Policy Distillation. arXiv:2610.02381, 2026.
[12] J. Yang, Z. Fang, Z. Xu, et al. LastOPD: Taming Collapse in Latent On-Policy Distillation. arXiv:2609.28845, 2026.
[13] H. Li, M. J. Chen, W. Ren, et al. The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation. arXiv:2609.36484, 2026.
W2 & Q3: Were sampled/top-16 OPD baselines in Table 3 trained through the full backbone? Please provide resolved configurations and explain the six complete-failure outcomes after task-specific tuning.
Response: Yes. Every method in Table 3 updates the full backbone. The teacher is frozen; OPRD-Bridge additionally freezes the pretrained bridge. All methods use the same student initialization, training data, update budget, and decoding settings, ensuring a fully controlled and fair comparison.
Method Resolved method-specific configuration OPD (Sampled) Sampled-token OPD OPD (Top-16) Top-16 student support ExOPD AOPD EOPD , entropy threshold , coefficient PowerOPD FiReOPD percentile , TIDE , positive/negative retention , OPRD-Bridge Frozen rank-64 bridge All methods use the same student, teacher, 1,024 training prompts, batch size, learning rate, 8,192-token response budget, and evaluation protocol. Method-specific hyperparameters are selected on the validation set.
The reported failure rate is the union of length-limit termination and invalid output format; these categories may overlap:
Method Length-limit termination ↓ Format failure ↓ Aggregate failure ↓ OPD (Sampled) 21.9 42.2 42.8 OPD (Top-16) 14.1 43.3 43.5 ExOPD 100.0 27.3 100.0 AOPD 98.4 25.0 100.0 EOPD 100.0 25.1 100.0 PowerOPD 87.5 12.6 100.0 FiReOPD 100.0 25.0 100.0 TIDE 100.0 34.7 100.0 OPRD-Bridge 7.7 15.9 15.9 The six complete-failure outcomes are mainly caused by training-induced non-termination: their generations overwhelmingly continue to the 8,192-token limit. Because all methods share the same prompting, decoding, stopping conditions, and evaluation interface, this behavior cannot be attributed to an interface mismatch. We have added the full-backbone training clarification, resolved baseline configurations, and detailed failure diagnostics in Appendix C.9.3–C.9.4 and Tables 14–15 of the revised paper and highlighted them in blue.
W3 & Q2: What is gained by freezing the bridge rather than jointly training or periodically recalibrating it? Does the rank trend persist with loss or gradient scales controlled?
Response: Freezing the bridge keeps the representation coordinates fixed during Stage 2. Joint training allows the projector to absorb alignment error without improving the student backbone, while periodic recalibration creates a moving target and repeats the calibration cost.
Bridge strategy Fail. ↓ Tok. ↓ Rel. ↑ Flu. ↑ Emp. ↑ Per. ↑ Safe. ↑ Avg. ↑ Joint bridge–student training 100.0 7953.6 20.3 20.3 20.3 20.3 20.3 20.3 Periodic recalibration 22.9 934.7 54.0 54.0 53.8 54.0 54.4 54.0 Frozen bridge (ours) 15.9 825.7 53.7 53.9 53.6 53.6 54.2 53.8 Following your suggestion, we remove the factor and repeat the rank ablation under otherwise identical settings:
Rank Original scaling ↑ Fixed scale (without 1/r) ↑ 4 55.2 54.8 16 54.4 55.0 64 53.8 55.4 128 56.6 54.6 256 47.1 43.7 512 34.6 22.2 1024 21.2 20.4 The same trend persists without 1/r: ranks 4–128 remain effective, while performance deteriorates at ranks 256 and above. This confirms that the high-rank degradation is not caused by the original loss scaling. We have added comparisons of frozen, jointly trained, and periodically recalibrated bridges, together with the loss-scale-controlled rank ablation, in Appendix C.9.5 and Tables 17–18 of the revised paper and highlighted them in blue.
Thank you for the careful review and valuable suggestions. We respond point by point below.
W1 & Q1: The incremental contribution is not fully isolated. Hidden-state distillation is established, but matched offline-representation and alternative bridge-training controls are missing. How does OPRD compare with teacher-response SFT, parameter merging for shared-backbone teachers, and representation distillation on fixed trajectories under matched budgets?