- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Knowledge distillation and preference alignment are both established paradigms to improve LLMs performance. However, the combination of knowledge distillation loss and preference alignment losses (DPO) can provide contradictory signals in the training process. Bloop (Hsieh et al.) propose an optimization technique for such joint-loss function training. In this optimization process, only the gradient signal from the auxiliary loss function which is orthogonal to the primary loss function’s gradient is retained. The gradients parallel to the primary loss function’s gradient are discarded.
In this paper, the authors examine the performance of this optimization technique for knowledge distillation + DPO and show that it performs better than baselines like SFT, SFT + DPO, scalar combination of KD + DPO, KD only on three benchmarks and two model families. They further conduct ablation studies on specific design choices like using exponential moving average of past preference losses for projection instead of the most recent loss gradients, orthogonality of the gradients.
Critical strengths
- This paper shows that it is possible to make Knowledge Distillation and Preference Alignment work simultaneously. The experiments and the ablation studies comparing full gradient combination, parallel only, orthogonal only, norm-matched and random projections helps distinguish the importance of the orthogonal gradient updates.
- The presentation of this paper is highly appreciated. The authors made it very easy to understand different sections of the paper, the design choices, the experiments and the empirical results.
Critical weaknesses
- The novelty in this work is not clear. Bloop [1] examines the performance of orthogonal gradient updates in multi-task learning settings. To the best of my understanding, the authors apply Bloop for a knowledge distillation + preference alignment task and present it's empirical improvements over simple baselines. While the results validate the application of Bloop in this specific setting, it is not clear that there is sufficient novelty for this paper to be accepted.
- The baselines for the experiments are also limited. There are other preference alignment methodologies like IPO, KTO etc which are missing from this work. Similarly there are other multi-task learning papers related to the geometry of gradient updates[2][3][4] which have not been discussed.
[1] Yu-Guan Hsieh, James Thornton, Eugene Ndiaye, Michal Klein, Marco Cuturi, and Pierre Ablin. Careful with that scalpel: improving gradient surgery with an ema. arXiv preprint arXiv:2402.02998, 2024. [2] Dery, L. M., Dauphin, Y., and Grangier, D. (2021). “Auxiliary Task Update Decomposition: The Good, the Bad and the Neutral.” ICLR. [3] Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. (2020). “Gradient Surgery for Multi-Task Learning.” NeurIPS. [4] Shamsian, A., Navon, A., Glazer, N., Kawaguchi, K., Chechik, G., and Fetaya, E. (2023). “Auxiliary Learning as an Asymmetric Bargaining Game.” ICML.
Questions for authors
- Can you elaborate on why the experiment was warm started from n intermediate DPO checkpoint [line16] and whether this has any interaction effects on the subsequent training?
- As the experiments are conducted using LoRA, will this also generalize to full-parameter training of the LLMs?
4 comments
Question 1: DPO Warm Start and Training Interactions
We thank the reviewer for this question. The intermediate DPO checkpoint is specific to the gradient-component diagnostic in Table 1 in the paper (Section 2.2). It allows us to examine how introducing different teacher-gradient components affects ongoing preference learning from a common state with a shared training history. All branches inherit identical model parameters, optimizer states, and EMA histories and follow the same optimizer-batch sequence (Appendix D).
The shared warm start controls the conditions at the branching point. Subsequent interactions remain possible: different auxiliary updates can lead to different parameter trajectories, affecting later preference gradients and optimizer states. Accordingly, Table 1 in the paper measures the effects of the continuation rules conditional on that shared state, including their subsequent interactions with preference learning.
OrDiPO’s main experiments and the full-training ablations in Table 4 in the paper start directly from the initial student and LoRA weights, without a preceding DPO-only stage (Appendices K and E). These runs also show improvements over DPO and scalar DPO+KD in both preference accuracy and generation win rate, demonstrating that the reported benefits can be obtained without this warm start. We will clarify the scope of the continuation diagnostic and its relationship to the main training protocol in the main text.
Question 2: Generalization to Full-Parameter Training
We thank the reviewer for raising this question. OrDiPO’s update rule does not require a low-rank parameterization. In Section 3.2, the projection is defined over the trainable student parameters. The same construction can therefore be applied to full-parameter fine-tuning by computing both objective gradients, the preference-gradient EMA, and the orthogonal residual over all student parameters.
Our empirical evaluation focuses on LoRA, with matched training settings within each model and dataset configuration. Changing the parameterization changes the gradient geometry and optimization dynamics, so the observed gains do not by themselves establish equivalent benefits under full-parameter training. The current results support the utility of orthogonal teacher guidance across the evaluated LoRA settings. We will explicitly clarify this empirical scope and identify full-parameter evaluation as a direction for further study in the limitations.
[1] Hsieh, Y. G., Thornton, J., Ndiaye, E., Klein, M., Cuturi, M., & Ablin, P. (2024). Careful with that scalpel: improving gradient surgery with an EMA. arXiv preprint arXiv:2402.02998.
[2] Dery, L. M., Dauphin, Y., & Grangier, D. (2021). Auxiliary task update decomposition: The good, the bad and the neutral. arXiv preprint arXiv:2108.11346.
[3] Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., & Finn, C. (2020). Gradient surgery for multi-task learning. Advances in neural information processing systems, 33, 5824-5836.
[4] Shamsian, A., Navon, A., Glazer, N., Kawaguchi, K., Chechik, G., & Fetaya, E. (2023, July). Auxiliary learning as an asymmetric bargaining game. In International Conference on Machine Learning (pp. 30689-30705). PMLR.
[5] Nguyen, D., Xiao, H., Prasad, A., Stengel-Eskin, E., Lee, H., & Bansal, M. (2026). Conflict-Resolving and Sharpness-Aware Minimization for Generalized Knowledge Editing with Multiple Updates. arXiv preprint arXiv:2602.03696.
[6] Chen, W., Wu, Y., Yang, J., Zeng, D., Zhao, Q., Paisley, J., ... & Wang, Z. (2026). Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner. arXiv preprint arXiv:2604.18239.
[7] Gao, S., Wan, F., Guo, J., Quan, X., & Wang, Q. (2025, May). Advantage-guided distillation for preference alignment in small language models. In International Conference on Learning Representations (Vol. 2025, pp. 18916-18938).
[8] Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., & Calandriello, D. (2024, April). A general theoretical paradigm to understand learning from human preferences. In International conference on artificial intelligence and statistics (pp. 4447-4455). PMLR.
[9] Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024, July). Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning.
| Method | Auxiliary supervision | Integration | UltraMedical PAcc ↑ | UltraMedical WR ↑ | SHP–AskDocs PAcc ↑ | SHP–AskDocs WR ↑ |
|---|---|---|---|---|---|---|
| DPO | — | — | 73.62 | 38.36 | 59.96 | 56.69 |
| SFT→DPO | Chosen tokens | Sequential, 1+3 epochs | 71.20 | 33.87 | 59.01 | 53.91 |
| KD→DPO | Teacher distributions | Sequential, 1+3 epochs | 74.19 | 38.19 | 59.28 | 66.22 |
| DPO+SFT | Chosen tokens | Scalar | 73.04 | 35.54 | 58.07 | 58.11 |
| DPO+KD | Teacher distributions | Scalar | 69.82 | 38.65 | 58.30 | 60.91 |
| DPO + Projected SFT | Chosen tokens | EMA projection | 75.81 | 39.57 | 60.08 | 62.16 |
| OrDiPO | Teacher distributions | EMA projection | 76.96 | 42.86 | 60.83 | 71.01 |
Table 1. Auxiliary supervision and integration strategies with Qwen3-8B-Base. All results use terminal checkpoints.
| Method | Primary objective | Projected gradient | UltraMedical PAcc ↑ | UltraMedical WR ↑ | SHP–AskDocs PAcc ↑ | SHP–AskDocs WR ↑ |
|---|---|---|---|---|---|---|
| DPO | DPO | — | 73.62 | 38.36 | 59.96 | 56.69 |
| DPO+KD (scalar) | — | — | 69.82 | 38.65 | 58.30 | 60.91 |
| Reverse projection | KD | DPO | 71.89 | 39.29 | 59.48 | 64.19 |
| OrDiPO | DPO | KD | 76.96 | 42.86 | 60.83 | 71.01 |
Table 2. Effect of reversing the primary–auxiliary assignment with Qwen3-8B-Base (all metrics in %).
| Dataset | Method | PAcc ↑ | WR ↑ | K-QA Comp ↑ | K-QA Hall ↓ | HealthBench ↑ |
|---|---|---|---|---|---|---|
| DPO | 73.62 | 38.36 | 56.79 | 0.18 | 21.78 | |
| IPO | 72.81 | 35.60 | 55.44 | 0.22 | 21.48 | |
| UltraMedical | KTO | 73.73 | 39.00 | 57.04 | 0.20 | 23.14 |
| DPO+KD | 69.82 | 38.65 | 55.46 | 0.16 | 20.56 | |
| OrDiPO | 76.96 | 42.86 | 58.61 | 0.14 | 28.37 | |
| DPO | 59.96 | 56.69 | 29.83 | 0.23 | 19.42 | |
| IPO | 60.26 | 61.07 | 30.84 | 0.42 | 19.36 | |
| SHP–AskDocs | KTO | 58.92 | 58.66 | 30.89 | 0.31 | 19.92 |
| DPO+KD | 58.30 | 60.91 | 42.12 | 0.18 | 24.10 | |
| OrDiPO | 60.83 | 71.01 | 55.07 | 0.21 | 30.19 |
Table 3. IPO and KTO comparisons with Qwen3-8B-Base.
We appreciate your insightful feedback and we hope our additional experiments and explanations could address your concerns and questions.
Dear Reviewer U3R9,
As the discussion period draws to a close, we wanted to briefly follow up on our detailed responses. We clarified our task-specific contributions relative to prior gradient-projection work, the diagnostic-only DPO warm start, and the empirical scope of LoRA training. We also added IPO and KTO comparisons on both health datasets, alongside projected-SFT and reverse-projection controls.
We hope these clarifications and additional experiments address your concerns, and we would be happy to discuss any further questions.
Dear Reviewer U3R9:
We sincerely appreciate your valuable comments on our work. We hope our response will adequately address your concerns.
Concern 1: Novelty Relative to Bloop
We thank the reviewer for highlighting the relationship to Bloop. As acknowledged in Sections 3.2.1 and 6, OrDiPO builds on Bloop’s EMA-based projection [1]. Building on this established framework, our contribution lies in the empirical insights and validated design for selectively integrating teacher guidance into preference learning. Figure 2 shows that closer imitation of teacher distributions, even on chosen-response prefixes, does not necessarily improve preference accuracy or generation quality. This motivates our central question: can useful teacher guidance be retained after removing its component along an estimated preference direction? Table 1 in the paper investigates this through controlled gradient-component interventions. The teacher-derived orthogonal residual improves both preference accuracy and generation win rate over preference-only continuation, while norm-matched full-gradient and random-orthogonal controls do not reproduce this joint improvement. These controls support the residual’s directional value beyond the tested effects of magnitude reduction or arbitrary orthogonal perturbations, a task-specific finding not established by the general projection rule alone. Full-training ablations from the initial student favor unconditional over conditional projection on both PAcc and WR in the evaluated setting (Table 4 in the paper; Appendix E). Figure 4 links the PAcc advantage over DPO+KD primarily to fewer regressions. OrDiPO also exceeds CoRSA, DPO+RC, and DCKD on both internal metrics across all five evaluated configurations (Tables 2–3 and 7–9 in the paper). Our added controls further show that OrDiPO outperforms both projected SFT and reversed-hierarchy projection on PAcc and WR across the two datasets, supporting teacher-distribution supervision and preference-primary optimization under the tested settings (Tables 1 and 2 below). We will clarify these findings and their relationship to the inherited projection rule.
Concern 2: Baseline Coverage and Related Work
We thank the reviewer for suggesting broader comparisons. Our controlled comparisons among DPO, scalar DPO+KD, and OrDiPO hold the DPO objective fixed to isolate how teacher guidance is incorporated. We also evaluate CoRSA, DPO+RC, and DCKD, covering joint optimization, preference-gradient calibration, and teacher-guided distillation, respectively (Tables 2, 3 and Appendix G in the paper) [5][6][7]. To broaden the comparison beyond DPO, we added standalone IPO and KTO on both datasets with Qwen3-8B-Base using the same evaluation protocol (Table 3 below) [8][9]. We trained IPO and KTO for three epochs using the same student initialization, data splits, LoRA configuration, optimizer, and dataset-specific learning rates as DPO (Appendix K), and evaluated the terminal checkpoints. PAcc uses the common initial-student reference and response-token log-probability sums from Appendix H, independently of the training objective. Their performance relative to DPO varies by dataset, while OrDiPO achieves higher PAcc and WR than IPO and KTO in each setting, along with better K-QA completeness, lower contradiction counts, and higher HealthBench scores. These comparisons complement our DPO, scalar DPO+KD, and OrDiPO controls, which examine how teacher guidance is incorporated. Regarding the gradient geometry literature, the cited studies on auxiliary-gradient decomposition, gradient conflict handling, and asymmetric auxiliary learning are already discussed in Section 6 under “Hierarchical Optimization” [2][3][4]. Our CoRSA baseline also incorporates PCGrad within joint SFT and DPO training together with sharpness-aware minimization, providing a related experimental comparison involving gradient conflict handling [3][5]. We will make these connections and the scope of the comparisons more explicit.