acceptodds
Under review as a conference paper at ICLR 2027

OrDiPO: Orthogonal Distillation for Preference Optimization in Language Models

Abstract

Knowledge distillation provides token-level teacher guidance that can complement pairwise preference learning for language model alignment. However, closer imitation of the teacher’s predictive distributions does not necessarily improve alignment with the target preferences. Through controlled gradient-component interventions, we show that the distillation residual orthogonal to an estimated preference-gradient direction retains useful generative guidance. Motivated by this observation, we propose Orthogonal Distillation for Preference Optimization (OrDiPO), which treats Direct Preference Optimization as the primary objective and chosen-prefix distillation as an auxiliary signal. Specifically, OrDiPO uses an exponential moving average of past preference gradients to define the projection normal. It removes the distillation-gradient component parallel to this normal and combines the remaining orthogonal residual with the unmodified current preference gradient. Experiments on UltraMedical-Preference and SHP–AskDocs demonstrate improvements over DPO and DPO+KD in both reference-corrected pair accuracy and generation win rate on internal test sets, with further evidence from external benchmarks. Additional experiments on HelpSteer3-Preference support applicability in a general-domain setting, while ablations and gradient analyses support the roles of orthogonal guidance and preference-gradient averaging.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
language model alignment, preference optimization, knowledge distillation, gradient projection, healthcare

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_U3R9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Knowledge distillation and preference alignment are both established paradigms to improve LLMs performance. However, the combination of knowledge distillation loss and preference alignment losses (DPO) can provide contradictory signals in the training process. Bloop (Hsieh et al.) propose an optimization technique for such joint-loss function training. In this optimization process, only the gradient signal from the auxiliary loss function which is orthogonal to the primary loss function’s gradient is retained. The gradients parallel to the primary loss function’s gradient are discarded.

In this paper, the authors examine the performance of this optimization technique for knowledge distillation + DPO and show that it performs better than baselines like SFT, SFT + DPO, scalar combination of KD + DPO, KD only on three benchmarks and two model families. They further conduct ablation studies on specific design choices like using exponential moving average of past preference losses for projection instead of the most recent loss gradients, orthogonality of the gradients.

Critical strengths
  1. This paper shows that it is possible to make Knowledge Distillation and Preference Alignment work simultaneously. The experiments and the ablation studies comparing full gradient combination, parallel only, orthogonal only, norm-matched and random projections helps distinguish the importance of the orthogonal gradient updates.
  2. The presentation of this paper is highly appreciated. The authors made it very easy to understand different sections of the paper, the design choices, the experiments and the empirical results.
Critical weaknesses
  1. The novelty in this work is not clear. Bloop [1] examines the performance of orthogonal gradient updates in multi-task learning settings. To the best of my understanding, the authors apply Bloop for a knowledge distillation + preference alignment task and present it's empirical improvements over simple baselines. While the results validate the application of Bloop in this specific setting, it is not clear that there is sufficient novelty for this paper to be accepted.
  2. The baselines for the experiments are also limited. There are other preference alignment methodologies like IPO, KTO etc which are missing from this work. Similarly there are other multi-task learning papers related to the geometry of gradient updates[2][3][4] which have not been discussed.

[1] Yu-Guan Hsieh, James Thornton, Eugene Ndiaye, Michal Klein, Marco Cuturi, and Pierre Ablin. Careful with that scalpel: improving gradient surgery with an ema. arXiv preprint arXiv:2402.02998, 2024. [2] Dery, L. M., Dauphin, Y., and Grangier, D. (2021). “Auxiliary Task Update Decomposition: The Good, the Bad and the Neutral.” ICLR. [3] Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. (2020). “Gradient Surgery for Multi-Task Learning.” NeurIPS. [4] Shamsian, A., Navon, A., Glazer, N., Kawaguchi, K., Chechik, G., and Fetaya, E. (2023). “Auxiliary Learning as an Asymmetric Bargaining Game.” ICML.

Questions for authors
  1. Can you elaborate on why the experiment was warm started from n intermediate DPO checkpoint [line16] and whether this has any interaction effects on the subsequent training?
  2. As the experiments are conducted using LoRA, will this also generalize to full-parameter training of the LLMs?
4 comments

Dear Reviewer U3R9:

We sincerely appreciate your valuable comments on our work. We hope our response will adequately address your concerns.

Concern 1: Novelty Relative to Bloop

We thank the reviewer for highlighting the relationship to Bloop. As acknowledged in Sections 3.2.1 and 6, OrDiPO builds on Bloop’s EMA-based projection [1]. Building on this established framework, our contribution lies in the empirical insights and validated design for selectively integrating teacher guidance into preference learning. Figure 2 shows that closer imitation of teacher distributions, even on chosen-response prefixes, does not necessarily improve preference accuracy or generation quality. This motivates our central question: can useful teacher guidance be retained after removing its component along an estimated preference direction? Table 1 in the paper investigates this through controlled gradient-component interventions. The teacher-derived orthogonal residual improves both preference accuracy and generation win rate over preference-only continuation, while norm-matched full-gradient and random-orthogonal controls do not reproduce this joint improvement. These controls support the residual’s directional value beyond the tested effects of magnitude reduction or arbitrary orthogonal perturbations, a task-specific finding not established by the general projection rule alone. Full-training ablations from the initial student favor unconditional over conditional projection on both PAcc and WR in the evaluated setting (Table 4 in the paper; Appendix E). Figure 4 links the PAcc advantage over DPO+KD primarily to fewer regressions. OrDiPO also exceeds CoRSA, DPO+RC, and DCKD on both internal metrics across all five evaluated configurations (Tables 2–3 and 7–9 in the paper). Our added controls further show that OrDiPO outperforms both projected SFT and reversed-hierarchy projection on PAcc and WR across the two datasets, supporting teacher-distribution supervision and preference-primary optimization under the tested settings (Tables 1 and 2 below). We will clarify these findings and their relationship to the inherited projection rule.

Concern 2: Baseline Coverage and Related Work

We thank the reviewer for suggesting broader comparisons. Our controlled comparisons among DPO, scalar DPO+KD, and OrDiPO hold the DPO objective fixed to isolate how teacher guidance is incorporated. We also evaluate CoRSA, DPO+RC, and DCKD, covering joint optimization, preference-gradient calibration, and teacher-guided distillation, respectively (Tables 2, 3 and Appendix G in the paper) [5][6][7]. To broaden the comparison beyond DPO, we added standalone IPO and KTO on both datasets with Qwen3-8B-Base using the same evaluation protocol (Table 3 below) [8][9]. We trained IPO and KTO for three epochs using the same student initialization, data splits, LoRA configuration, optimizer, and dataset-specific learning rates as DPO (Appendix K), and evaluated the terminal checkpoints. PAcc uses the common initial-student reference and response-token log-probability sums from Appendix H, independently of the training objective. Their performance relative to DPO varies by dataset, while OrDiPO achieves higher PAcc and WR than IPO and KTO in each setting, along with better K-QA completeness, lower contradiction counts, and higher HealthBench scores. These comparisons complement our DPO, scalar DPO+KD, and OrDiPO controls, which examine how teacher guidance is incorporated. Regarding the gradient geometry literature, the cited studies on auxiliary-gradient decomposition, gradient conflict handling, and asymmetric auxiliary learning are already discussed in Section 6 under “Hierarchical Optimization” [2][3][4]. Our CoRSA baseline also incorporates PCGrad within joint SFT and DPO training together with sharpness-aware minimization, providing a related experimental comparison involving gradient conflict handling [3][5]. We will make these connections and the scope of the comparisons more explicit.

Question 1: DPO Warm Start and Training Interactions

We thank the reviewer for this question. The intermediate DPO checkpoint is specific to the gradient-component diagnostic in Table 1 in the paper (Section 2.2). It allows us to examine how introducing different teacher-gradient components affects ongoing preference learning from a common state with a shared training history. All branches inherit identical model parameters, optimizer states, and EMA histories and follow the same optimizer-batch sequence (Appendix D).

The shared warm start controls the conditions at the branching point. Subsequent interactions remain possible: different auxiliary updates can lead to different parameter trajectories, affecting later preference gradients and optimizer states. Accordingly, Table 1 in the paper measures the effects of the continuation rules conditional on that shared state, including their subsequent interactions with preference learning.

OrDiPO’s main experiments and the full-training ablations in Table 4 in the paper start directly from the initial student and LoRA weights, without a preceding DPO-only stage (Appendices K and E). These runs also show improvements over DPO and scalar DPO+KD in both preference accuracy and generation win rate, demonstrating that the reported benefits can be obtained without this warm start. We will clarify the scope of the continuation diagnostic and its relationship to the main training protocol in the main text.

Question 2: Generalization to Full-Parameter Training

We thank the reviewer for raising this question. OrDiPO’s update rule does not require a low-rank parameterization. In Section 3.2, the projection is defined over the trainable student parameters. The same construction can therefore be applied to full-parameter fine-tuning by computing both objective gradients, the preference-gradient EMA, and the orthogonal residual over all student parameters.

Our empirical evaluation focuses on LoRA, with matched training settings within each model and dataset configuration. Changing the parameterization changes the gradient geometry and optimization dynamics, so the observed gains do not by themselves establish equivalent benefits under full-parameter training. The current results support the utility of orthogonal teacher guidance across the evaluated LoRA settings. We will explicitly clarify this empirical scope and identify full-parameter evaluation as a direction for further study in the limitations.


[1] Hsieh, Y. G., Thornton, J., Ndiaye, E., Klein, M., Cuturi, M., & Ablin, P. (2024). Careful with that scalpel: improving gradient surgery with an EMA. arXiv preprint arXiv:2402.02998.

[2] Dery, L. M., Dauphin, Y., & Grangier, D. (2021). Auxiliary task update decomposition: The good, the bad and the neutral. arXiv preprint arXiv:2108.11346.

[3] Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., & Finn, C. (2020). Gradient surgery for multi-task learning. Advances in neural information processing systems, 33, 5824-5836.

[4] Shamsian, A., Navon, A., Glazer, N., Kawaguchi, K., Chechik, G., & Fetaya, E. (2023, July). Auxiliary learning as an asymmetric bargaining game. In International Conference on Machine Learning (pp. 30689-30705). PMLR.

[5] Nguyen, D., Xiao, H., Prasad, A., Stengel-Eskin, E., Lee, H., & Bansal, M. (2026). Conflict-Resolving and Sharpness-Aware Minimization for Generalized Knowledge Editing with Multiple Updates. arXiv preprint arXiv:2602.03696.

[6] Chen, W., Wu, Y., Yang, J., Zeng, D., Zhao, Q., Paisley, J., ... & Wang, Z. (2026). Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner. arXiv preprint arXiv:2604.18239.

[7] Gao, S., Wan, F., Guo, J., Quan, X., & Wang, Q. (2025, May). Advantage-guided distillation for preference alignment in small language models. In International Conference on Learning Representations (Vol. 2025, pp. 18916-18938).

[8] Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., & Calandriello, D. (2024, April). A general theoretical paradigm to understand learning from human preferences. In International conference on artificial intelligence and statistics (pp. 4447-4455). PMLR.

[9] Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024, July). Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning.

MethodAuxiliary supervisionIntegrationUltraMedical PAcc ↑UltraMedical WR ↑SHP–AskDocs PAcc ↑SHP–AskDocs WR ↑
DPO——73.6238.3659.9656.69
SFT→DPOChosen tokensSequential, 1+3 epochs71.2033.8759.0153.91
KD→DPOTeacher distributionsSequential, 1+3 epochs74.1938.1959.2866.22
DPO+SFTChosen tokensScalar73.0435.5458.0758.11
DPO+KDTeacher distributionsScalar69.8238.6558.3060.91
DPO + Projected SFTChosen tokensEMA projection75.8139.5760.0862.16
OrDiPOTeacher distributionsEMA projection76.9642.8660.8371.01

Table 1. Auxiliary supervision and integration strategies with Qwen3-8B-Base. All results use terminal checkpoints.



MethodPrimary objectiveProjected gradientUltraMedical PAcc ↑UltraMedical WR ↑SHP–AskDocs PAcc ↑SHP–AskDocs WR ↑
DPODPO—73.6238.3659.9656.69
DPO+KD (scalar)——69.8238.6558.3060.91
Reverse projectionKDDPO71.8939.2959.4864.19
OrDiPODPOKD76.9642.8660.8371.01

Table 2. Effect of reversing the primary–auxiliary assignment with Qwen3-8B-Base (all metrics in %).



DatasetMethodPAcc ↑WR ↑K-QA Comp ↑K-QA Hall ↓HealthBench ↑
DPO73.6238.3656.790.1821.78
IPO72.8135.6055.440.2221.48
UltraMedicalKTO73.7339.0057.040.2023.14
DPO+KD69.8238.6555.460.1620.56
OrDiPO76.9642.8658.610.1428.37
DPO59.9656.6929.830.2319.42
IPO60.2661.0730.840.4219.36
SHP–AskDocsKTO58.9258.6630.890.3119.92
DPO+KD58.3060.9142.120.1824.10
OrDiPO60.8371.0155.070.2130.19

Table 3. IPO and KTO comparisons with Qwen3-8B-Base.


We appreciate your insightful feedback and we hope our additional experiments and explanations could address your concerns and questions.

Dear Reviewer U3R9,

As the discussion period draws to a close, we wanted to briefly follow up on our detailed responses. We clarified our task-specific contributions relative to prior gradient-projection work, the diagnostic-only DPO warm start, and the empirical scope of LoRA training. We also added IPO and KTO comparisons on both health datasets, alongside projected-SFT and reverse-projection controls.

We hope these clarifications and additional experiments address your concerns, and we would be happy to discuss any further questions.

Reviewer_S2uxOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper applies Bloop's primary–auxiliary gradient projection method to the joint optimization of DPO and knowledge distillation and shows its effectiveness through controlled experiments. The projection method itself has limited novelty, but applying an existing optimization tool to this problem and showing empirical gains over the tested scalar-weighted joint training settings still makes a valuable contribution.

Critical strengths
  1. The problem is clearly defined. Imitating the teacher and meeting the target preferences are different objectives. The paper directly studies how to retain the benefits of teacher guidance while reducing interference with the primary objective, which is more targeted than only adjusting the weights of the two losses.
  2. The main finding has strong controlled evidence. The authors compare the full gradient, parallel component, and orthogonal component from the same training state, and add norm-matched and random orthogonal controls. These experiments support the view that the gains depend on the selected direction, rather than only on a smaller auxiliary gradient or the effectiveness of arbitrary perturbations (Section 2.2).
  3. The experiments assess both objectives. The authors report results for both preference ranking and response generation, allowing readers to assess the trade-off between the two types of metrics after projection instead of judging joint training by a single metric.
  4. Results from multiple settings support the method's value. Results across two model families, health-related data, and independent general-domain experiments generally support a better training trade-off, showing value beyond a local geometric construction.
Critical weaknesses
  1. The added methodological contribution is limited. Combining an average of past primary gradients with orthogonal auxiliary updates has already been studied, and the paper explicitly cites and adopts Bloop's approach. Therefore, the paper's contribution mainly needs to be supported by new findings and practical gains in preference distillation. The paper should explain more clearly which difficulties are specific to this task and which conclusions go beyond existing experience with auxiliary learning (Sections 3.2 and 6).
  2. The choice of primary and auxiliary objectives is not sufficiently justified. The paper aims to use teacher guidance while keeping preference learning as the primary objective, which provides a motivation for the current design. However, since the final evaluation considers both preference ranking and generation quality, why should DPO be the primary objective and distillation be the auxiliary objective? What would happen if the full distillation gradient were retained and the DPO gradient were projected instead? The current ablations do not report this reversed setup. Further explanation and controlled comparisons would help determine whether the gains come from the projection method itself or the specific choice of preference learning as the primary objective and distillation as the auxiliary objective.
  3. The link between gains in both metrics and actual quality improvements needs more independent evidence. Pair accuracy measures the ranking of given responses, while generation win rate measures the model judge's preferences. Improvements in both are positive evidence, and external benchmarks provide further support, but the main generation evaluations still rely on the same judge. Therefore, the evidence that the teacher's generation ability is effectively retained remains indirect, and the influence of response style on the evaluation results cannot yet be fully ruled out.
Questions for authors

See weaknesses.

4 comments

Dear Reviewer S2ux:

We sincerely appreciate your valuable comments on our work. We hope our response will adequately address your concerns.

Concern 1: Contributions to Preference Distillation

We appreciate this constructive suggestion. OrDiPO builds on Bloop’s EMA projection; our contribution concerns which teacher guidance remains useful for preference learning and the evidence supporting its selective integration. The task-specific difficulty is that DPO optimizes reference-corrected rankings between responses, whereas KD matches token distributions. Even conditioning KD on chosen responses does not ensure that teacher imitation reinforces the preference margin. Figure 2 shows that closer imitation does not consistently improve preference accuracy or generation outcomes. Our controlled interventions provide task-specific evidence beyond the general projection rule: the teacher-derived orthogonal residual improves both preference accuracy and generation win rate over preference-only continuation, while norm-matched full-gradient and random-orthogonal controls do not reproduce this joint improvement (Table 1 in the paper). This supports the residual’s directional value beyond the tested effects of auxiliary magnitude and arbitrary orthogonal perturbations. Figure 4 further links its preference advantage over scalar DPO+KD primarily to fewer regressions. Full-training ablations (Table 4 in the paper) and evaluations across models and datasets support the practical value of this design. The added reverse-projection comparison further supports retaining the full DPO gradient under matched coefficients (Table 1 below). We will make these findings and their relationship to prior auxiliary learning more explicit in Sections 3.2 and 6.

Concern 2: Primary and Auxiliary Objective Assignment

We thank the reviewer for this helpful suggestion. DPO directly supervises the target preference ranking, whereas KD optimizes teacher imitation rather than generation quality itself. Figure 2 shows that closer imitation does not consistently improve preference accuracy or generation outcomes. We added reverse projection on both datasets with Qwen3-8B-Base: dt=rP,t+0.1gD,td_t=r_{P,t}+0.1g_{D,t}, where rP,tr_{P,t} is the DPO residual orthogonal to the historical KD-gradient EMA. DPO/KD coefficients remain 1/0.1 to isolate projection orientation. Both projected variants use an EMA rate of 0.05, with matched initialization, optimizer, training budget, evaluation, and terminal checkpoints. Reverse projection improves both metrics over scalar DPO+KD but remains below DPO in PAcc on both datasets. OrDiPO exceeds the reversed variant in both PAcc and WR while also improving both over DPO (Table 1 below). These comparisons support the preference-primary assignment under matched settings: projection helps under either hierarchy, but retaining DPO yields better joint performance.

Concern 3: Independent Evidence of Generation Quality

We appreciate this careful distinction. Our claim concerns retaining useful teacher-derived guidance, as stated in the abstract. Beyond pairwise WR, K-QA assesses coverage and contradictions against physician-authored statements, while HealthBench evaluates 5,000 conversations using physician-written rubrics. Across both datasets and model families, OrDiPO improves K-QA completeness and HealthBench scores over DPO and scalar DPO+KD, with the lowest or tied-lowest reported K-QA contradiction count in three of four settings (Tables 2, 3 and 7–8 in the paper). PAcc provides judge-independent evidence of preference ranking, but we do not treat it as direct evidence of generated-answer quality. Our pairwise evaluations anonymize responses, reverse presentation order, and instruct the judge not to reward verbosity, confidence, or formatting alone. These safeguards are designed to mitigate bias, not establish its absence. We acknowledge that all model-based grading uses GPT-5.5. The results therefore support the usefulness of teacher-derived guidance across the evaluated criteria. These benchmarks provide complementary evaluation criteria but share the same grader, so judge-specific preferences and response-style effects remain possible.

MethodPrimary objectiveProjected gradientUltraMedical PAcc ↑UltraMedical WR ↑SHP–AskDocs PAcc ↑SHP–AskDocs WR ↑
DPODPO—73.6238.3659.9656.69
DPO+KD (scalar)——69.8238.6558.3060.91
Reverse projectionKDDPO71.8939.2959.4864.19
OrDiPODPOKD76.9642.8660.8371.01

Table 1. Effect of reversing the primary–auxiliary assignment with Qwen3-8B-Base (all metrics in %).


We appreciate your insightful feedback and we hope our additional experiments and explanations could address your concerns .

Thank you for your detailed responses and additional experiments. They have addressed my main concerns. I have increased my confidence score to 4.

We sincerely thank you for your positive feedback and are glad that our responses and additional experiments have addressed your main concerns.

Reviewer_e7FoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how to add token-level distillation from a larger teacher to DPO without hurting preference learning. A preliminary sweep shows that a larger distillation weight brings the student closer to the teacher and raises win rate, but lowers pair accuracy. The proposed method keeps the DPO gradient as is and, following the EMA-based projection of Hsieh et al. 2024, removes from the distillation gradient its component along a moving average of past DPO gradients before adding it back. Experiments with Qwen3-8B-Base and Llama-3.1-8B-Instruct students on two health preference datasets and HelpSteer3 report gains over DPO, DPO plus distillation and several adapted baselines on pair accuracy, win rate and external health benchmarks.

Critical strengths
  1. Well-motivated problem and a principled design direction. How to use a stronger teacher's token-level distributions without sacrificing preference learning is a practical question

  2. Consistent gains across settings. In all five model–dataset configurations, OrDiPO beats both DPO and DPO+KD on pair accuracy and win rate, and the implementation is documented in enough detail to be reproduced.

Critical weaknesses
  1. The method does not deliver the orthogonality promised by the title and introduction. The paper presents OrDiPO as distillation that does not interfere with DPO, but it adopts the EMA-based projection of Hsieh et al. 2024, so the residual is orthogonal only to a historical average and not to the current preference gradient, as the authors themselves acknowledge. Fig. 4 further shows that this average has an absolute cosine of only 0.12 to 0.31 with the audit preference gradient, so the residual is not orthogonal to the true preference direction either.

  2. Table 1 is non-additive and partly contradicts the motivation. In PAcc, both the parallel component at 74.65 and the orthogonal residual at 77.42 beat the no-distillation branch at 73.85, yet their sum, the full gradient, falls to 72.70, and the paper offers no explanation.

  3. The teacher is not aligned to the target preferences, so the conflict may come from the teacher rather than from gradient interference. Qwen3-32B and Llama-3.3-70B-Instruct are general-purpose models never tuned on the UltraMedical or SHP labels, and the paper reports neither their preference accuracy nor their win rate.

Questions for authors
  1. Does the gain come from the teacher distribution or from the projection? The only sequential baseline is SFT→DPO, with no KD→DPO, and DPO+SFT appears only as a scalar mixture in Fig. 2. Applying the same projection to an SFT auxiliary loss would show whether the teacher distribution is actually needed.

  2. How is the thinking mode of Qwen3-32B handled when caching teacher distributions? Qwen3-32B is a hybrid reasoning model whose chat template differs with thinking enabled or disabled, while the paper only states that teacher and student share the same serialization. Please clarify the template used and whether an empty think block precedes the chosen response.

  3. How sensitive is OrDiPO to λ and ρ, and how were they chosen? All OrDiPO runs and ablations fix λ = 0.1 and ρ = 0.05, and Appendix L sweeps λ only for DPO+KD. Since OrDiPO weights the residual rather than the full distillation gradient, the same λ does not imply the same auxiliary strength, as Appendix K itself notes.

4 comments

Dear Reviewer e7Fo:

We sincerely appreciate your valuable comments on our work. We hope our response will adequately address your concerns.

Concern 1: Orthogonality and EMA Alignment

Thank you for raising this distinction. OrDiPO’s orthogonality is defined with respect to an estimated preference-gradient direction, as stated in the abstract (L15–21) and introduction (L49–50). The residual is orthogonal to this normal before optimizer preconditioning, while the current DPO gradient enters the combined direction unchanged. We explicitly distinguish this property from orthogonality to the instantaneous gradient and from a guarantee of non-increasing preference loss after every step (L254–255, L289–290).

The historical EMA averages DPO gradients to reduce sensitivity to minibatch noise (L260–269). Fig. 4 compares EMA alignment with the mean alignment of eight 32-pair probe gradients to a 1,024-pair audit gradient (Appendix F); the audit is a finite-sample proxy, not the true full-data gradient. Absolute alignment is modest, but the EMA is higher at all four checkpoints. Crucially, we directly evaluate projection against the current DPO gradient in the w/o EMA ablation (L398–404). Table 4 of the paper shows that using the EMA improves PAcc from 74.88 to 76.96 and WR from 34.68 to 42.86. Fig. 4 (right) also shows fewer preference regressions than scalar DPO+KD across all four settings.

Concern 2: Non-additivity of Gradient Components

Thank you for highlighting this pattern. While gD=gD∥+rDg_D = g_D^{\parallel} + r_D holds at a fixed parameter state and projection normal, it does not imply additive final PAcc across separate multi-step runs. The branches in Table 1 of the paper share an initial checkpoint and batch sequence, but their subsequent parameter and optimizer states diverge (L930–933). Clipping and AdamW act on the combined gradient, so adding a component can alter the effective update and subsequent trajectory (L789–798). Thus, improvement from each component alone does not guarantee improvement when combined.

Our motivation is to investigate whether useful teacher guidance remains after removing the component along the estimated preference direction (L148–158). We explicitly acknowledge that parallel-only guidance also improves both metrics (L188–193). The central finding is that the orthogonal residual achieves the highest PAcc and WR among the tested interventions, including norm-matched full and random orthogonal controls. These comparisons support the value of selectively retaining informative teacher guidance; this conclusion does not require parallel guidance to be universally harmful, as already clarified in Appendix D (L955–958).

Concern 3: Teacher Alignment and Performance

We have added teacher-only evaluations using the student protocols, including the same initial-student reference within each model family for PAcc in Appendix H. Across all four model–dataset settings, the teachers outperform OrDiPO by 2.04–4.03 percentage points in PAcc and 4.78–8.04 points in WR, with stronger results on K-QA and HealthBench as well (Table 1). Thus, the teachers perform well on the target evaluations without additional target-specific fine-tuning.

Strong teacher performance does not, however, guarantee that chosen-prefix distribution matching reinforces the student’s pairwise preference objective (Section 2.1). With the same teachers, scalar DPO+KD still exhibits preference–generation trade-offs, whereas OrDiPO improves both metrics over DPO and DPO+KD. Together with the gradient-component interventions and projection ablation (Tables 1 and 4 in the paper), these findings argue against attributing the trade-offs solely to poor overall teacher performance and support our focus on how useful teacher guidance is incorporated into preference optimization.


DatasetTeacher modelPAcc ↑WR ↑K-QA Comp ↑K-QA Hall ↓HealthBench ↑
UltraMedical subsetQwen3-32B80.4147.6465.170.1144.02
UltraMedical subsetLlama-3.3-70B-Instruct79.9543.6159.200.1740.08
SHP–AskDocs subsetQwen3-32B64.8679.0565.170.1144.02
SHP–AskDocs subsetLlama-3.3-70B-Instruct62.1682.4359.200.1740.08

Table 1. Teacher-only performance on the internal tests and external benchmarks. External scores are shared across dataset rows for each frozen teacher.

Question 1: Teacher Supervision and Projection

We added KD→DPO, terminal DPO+SFT, and projected SFT, which replaces KD with chosen-token supervision while retaining OrDiPO’s DPO-gradient EMA projection. All joint methods use auxiliary weight 0.10 and train for three epochs; projected SFT uses ρ = 0.05 and the same remaining training protocol. These terminal comparisons extend Fig. 2’s preliminary intermediate-stage study.

Projection improves the SFT auxiliary baseline on both metrics across both datasets (Table 2). Under the same projection framework, however, OrDiPO further improves PAcc by 1.15/0.75 percentage points and WR by 3.29/8.85 points on UltraMedical/SHP–AskDocs, respectively. Together with the original DPO+KD versus OrDiPO ablation (Table 4 of the paper), these comparisons support complementary contributions from teacher-distribution supervision and projection. KD→DPO uses one epoch of KD followed by three epochs of DPO, with the frozen KD checkpoint as the DPO training reference and the optimizer and scheduler reset between stages. All PAcc evaluations use the common initial-student reference. This sequential baseline remains below OrDiPO on both metrics across both datasets (Table 2), supporting the value of how teacher guidance is incorporated into preference optimization.

Question 2: Qwen3 Thinking Mode and Serialization

Thank you for requesting this clarification. We use the official Qwen3 non-thinking serialization, with an empty <think>\n\n</think>\n\n block included in the fixed assistant prefix before the chosen response. Teacher targets are obtained by forward scoring the recorded chosen responses, without generating a new answer or reasoning trace.

The same serialization is used for teacher caching, student training, and reference scoring. The assistant prefix, including the empty thinking block, remains part of the conditioning context but is excluded from the loss mask. Only response tokens and one end-of-turn token are scored: KD uses chosen responses, while DPO scores both chosen and rejected responses, consistent with Appendix A (L682–687). We will explicitly document this prefix in the revised appendix.

Question 3: Hyperparameter Selection and Sensitivity

We used λ = 0.10 and ρ = 0.05 as shared defaults across the reported settings in Appendix K. Our added 3 × 3 sensitivity study (Table 3) on UltraMedical-Preference with Qwen3-8B-Base covers λ ∈ {0.03, 0.10, 0.30} and ρ ∈ {0.01, 0.05, 0.20}. All nine configurations improve both metrics over DPO: PAcc ranges from 75.35–78.34, versus 73.62, and WR from 40.32–44.87, versus 38.36. All also outperform fixed-weight DPO+KD on both metrics; their PAcc exceeds every scalar configuration in Appendix L. Thus, the joint improvements persist across the tested range. We agree that equal coefficients do not imply equal auxiliary-gradient magnitudes, as explicitly noted in Appendix K. Our comparisons additionally include a scalar DPO+KD weight sweep in Appendix L and a norm-matched full-gradient control in the continuation study (Table 1 in the paper), which the orthogonal residual outperforms on both metrics. Together with the new sensitivity results, these findings support the benefit of directional selection beyond simply adjusting distillation strength.

MethodAuxiliary supervisionIntegrationUltraMedical PAcc ↑UltraMedical WR ↑SHP–AskDocs PAcc ↑SHP–AskDocs WR ↑
DPO——73.6238.3659.9656.69
SFT→DPOChosen tokensSequential, 1+3 epochs71.2033.8759.0153.91
KD→DPOTeacher distributionsSequential, 1+3 epochs74.1938.1959.2866.22
DPO+SFTChosen tokensScalar73.0435.5458.0758.11
DPO+KDTeacher distributionsScalar69.8238.6558.3060.91
DPO + Projected SFTChosen tokensEMA projection75.8139.5760.0862.16
OrDiPOTeacher distributionsEMA projection76.9642.8660.8371.01

Table 2. Auxiliary supervision and integration strategies with Qwen3-8B-Base. All results use terminal checkpoints.



ρPAcc ↑ (λ = 0.03)PAcc ↑ (λ = 0.10)PAcc ↑ (λ = 0.30)WR ↑ (λ = 0.03)WR ↑ (λ = 0.10)WR ↑ (λ = 0.30)
0.0176.5078.1176.0441.4744.3043.78
0.0576.7376.9678.3441.4242.8644.87
0.2075.8177.6575.3540.3242.1743.38

Table 3. OrDiPO sensitivity to λ and ρ on UltraMedical-Preference with Qwen3-8B-Base.


We appreciate your insightful feedback and we hope our additional experiments and explanations could address your concerns and questions.

Dear Reviewer e7Fo,

As the discussion period draws to a close, we wanted to briefly follow up on our detailed responses. We clarified the EMA-relative scope of orthogonality, multi-step gradient non-additivity, and Qwen3 serialization, and provided additional teacher-only evaluations, KD→DPO and projected-SFT comparisons, and a 3×3 (λ, ρ) sensitivity study.

We hope these clarifications and additional results address your concerns, and we would be glad to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.