acceptodds
Under review as a conference paper at ICLR 2027

Lexicographic Multi-Objective On-Policy Distillation

Abstract

Reinforcement learning from verifiable rewards (RLVR) usually optimizes answer correctness, yet useful language-model behavior also requires high-quality reasoning and concise responses. Existing multi-reward post-training methods typically scalarize rewards or combine specialists without explicitly protecting a reward priority order. This is problematic when trade-offs are asymmetric: conciseness, for example, should not improve at the cost of correctness. We introduce **Lexicographic Multi-Objective On-Policy Distillation** (LMOPD), a multi-teacher method for integrating reward-specialized policies under explicit priorities. For each student rollout, LMOPD selects the specialist for the first objective whose gate detects a deficiency, then locally projects its centered log-policy correction to remove components that oppose higher-priority specialists. We evaluate 30B-A3B mixture-of-experts transformer models in two- and four-expert settings on three math benchmarks, measuring retained specialist gains. With two experts, LMOPD's point estimates fully retain the accuracy and reasoning-quality gains while acquiring % of the conciseness gain. With four experts, it retains % of both the accuracy gain and reasoning-correctness gain, compared to only % by the next best evaluated baseline. Matched four-expert ablations show that lexicographic routing outperforms random routing and that projection further strengthens both top-priority capabilities. Across both scales, LMOPD preserves the highest-priority capabilities more effectively than the existing baselines we evaluate, demonstrating the value of explicit priorities for specialist integration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Mathematics, Large language models (LLMs), Model Merging, Multi-Objective Optimization, Reinforcement Learning, Natural Language Processing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SRcHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper introduces Lexicographic Multi-Objective On-Policy Distillation (LMOPD). The paper argues that random/equal contribution from the existing method is not optimal, so they detect the first priority that exhibits a deficiency. Upon detection, the specialized policy is identified. Further filtering with a negative inner product with the high-priority expert is added to ensure all the policies align with each other. The paper reports the highest retained gain compared to baselines. No ablation is reported.

Critical strengths
  • The hypothesis is sound and potentially helpful for post-training.
  • The paper evaluates the models on a large amount of public data and shows improvement compared to baselines.
Critical weaknesses
  • The paper reported relative retained gains seem to be odd and not very conventional. The relative gains get inflated quickly when the potential gain is small. I saw that the paper reported the raw performance in the Appendix and evaluated the reported raw performance.
  • The evaluation protocol seems to be weak. The paper assumes that the used order is optimal; however, it might not be the case. The paper should evaluate different priority orders to avoid such a prior.
  • The ablation study is virtually poor and insufficient to address the modules added in this paper.
  • The results are impossible to reproduce because the paper uses a private training set.
  • Missing baselines: SPA (https://arxiv.org/abs/2511.06222, AAAI, published March 2026)
Questions for authors
  • How was the upper bound of the performance estimate over all possible priority orders?
3 comments

Thank you for the feedback and helpful suggestions. We appreciate the opportunity to clarify several aspects of the evaluation and methodology.

Retained gain: We agree that normalization can amplify differences when the base-to-specialist gap is small. We report raw performance in the appendix and have made it more prominent in the revision. We also corrected an inconsistency in the four-expert normalization, detailed in our response to Reviewer az51. Thank you for highlighting this issue.

Priority ordering: We apologize that this distinction was not sufficiently clear. In LMOPD, the priority ordering is an externally specified preference, not a hyperparameter whose optimal value we seek to discover. For example, prioritizing correctness over conciseness expresses the requirement that shorter responses should not come at the expense of correctness. Reversing this ordering would represent a different preference, not necessarily a better solution. Selecting an optimal ordering across conflicting objectives would require an additional criterion for comparing those preferences, which is outside our scope. We do not claim lexicographic optimality for a given ordering either; LMOPD uses the supplied priorities to guide teacher selection and projection. We have clarified this distinction in the revised manuscript.

Ablations: We apologize that these were difficult to locate. Table 3 of the original submission compares full LMOPD, routing-only LMOPD, and random routing in both the two- and four-expert settings, with additional results in the supplementary appendix. These isolate the contributions of lexicographic routing and projection. We have made these comparisons more prominent. SPA: Thank you for pointing out this relevant work. We have added SPA to Related Work. While both methods incorporate lexicographic priorities, SPA constructs preference pairs through self-evaluation and refinement, then performs preference optimization. In contrast, our setting is the integration of existing reward-specialized experts through on-policy distillation. SPA does not directly operate on these experts or provide an alternative teacher-selection mechanism; adapting it would require a separate preference-generation and optimization pipeline. We therefore believe the existing multi-task RLVR, GDPO, Rewarded Soup, and routing ablations provide more directly comparable baselines for evaluating our contributions.

Reproducibility: We agree that proprietary training data limits exact reproduction. However, the use of unreleased training data is not uncommon in accepted large-scale LLM research (e.g., Jamba, ICLR 2025). Our contribution is a general training algorithm rather than a dataset or model release. We provide the complete mathematical formulation, algorithms, hyperparameters, and experimental details, with evaluations on three public benchmarks. We believe these allow the methodology and empirical findings to be meaningfully assessed despite the proprietary training infrastructure.

We appreciate the suggestions and hope the revisions address the main concerns.

Lenz, Barak, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram et al. "Jamba: Hybrid transformer-mamba language models." In International Conference on Learning Representations, vol. 2025, pp. 67959-67984. 2025.

We wanted to briefly follow up before the discussion period closes on November 18. In particular, we hope our clarifications regarding the externally specified priority ordering and the existing component ablations (Table 3 of the original submission) help address your concerns. We have also made the ablations more prominent and added a discussion of SPA in the revised manuscript.

We would be very grateful if you could reconsider your assessment in light of these clarifications and revisions, and would be happy to address any remaining concerns before the discussion period closes. Thank you again for your time and feedback!

Thanks for the rebuttal. The rebuttal addressed my concerns. I increased the rating to 3.

Reviewer_az51OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LMOPD, which routes student trajectories to the expert corresponding to the first unmet objective according to a preset priority. Subsequently, the logarithmic strategy of the expert is projected into the function space to correct the trajectories, removing components that conflict with higher-priority experts. The authors conducted two-expert and four-expert experiments on a MoE model with approximately 30B total parameters and 3B activation parameters, examining the integration of correctness, inference quality, and simplicity. The accuracy of the two-expert setup improved from .711 to .745, while also achieving some simplicity gains; the four-expert ablation projection further shifts the benefits towards higher-priority objectives.

Critical strengths

The problem is clearly defined, and the routing rules directly address asymmetric objective requirements. The constraint that "simplicity cannot compensate for incorrect answers" is a reasonable multi-objective constraint. Selecting experts based on the actual deficiencies of each trajectory is more targeted than a fixed mix of all teachers; using absolute gating for discrete correctness also avoids the problem of zero relative advantage and inability to trigger error correction when the entire set of answers is wrong. Component ablation supports the role of priority routing and provides preliminary evidence of the effectiveness of projection. In the four-expert setting, the accuracy gain retention rate of the routing version alone is 54.0%, while the point estimate after adding projection is 90.6%; random routing, however, is lower than the base in high-priority metrics. The authors also disclose the sacrifices of low-priority gains and clarify that the method is only a local heuristic and does not claim global lexicographical optimality. This clarifies the scope of the empirical conclusions.

Critical weaknesses

The current document lacks the necessary materials for verifying the complete methodology and experiments. This PDF is only 12 pages long, ending at the references, but the main text repeatedly relies on missing appendices A–D, Algorithm 1/2, complete loss formulas (13)–(15), original results table 4, and various sub-results table 5. Therefore, it is impossible to verify the complete training objective, reward definition, evaluation hints, implementation details, and some statistical evidence. This problem stems from the coverage limitations of the currently provided document; until these materials are supplemented, a sufficient judgment on reproducibility and complete evidence cannot be made.

There is an inconsistency between the key retention rate and the definition of the metric in the abstract. Formula (11) defines the denominator as the gain of the "best available expert" for the objective, but Table 2 sets the reasoning correctness of RQ-Correctness Expert to 100%, while RQ-Verification Expert reaches 124.4% in the same column. If normalized according to the best performing expert in this table, the retention rate of LMOPD should be approximately 88.6/124.4*100%=71.2.

Questions for authors

Materials and Implementation Clarification: Could you please complete the appendix, especially the loss formulas (13)–(15), the reward definition, the evaluation hints, and the original metrics? How does the hard-label cross-entropy term align with the statement "self-distillation does not generate gradients when all gates pass"?

Metric Clarification: Why does the reasoning correctness in Table 2 use RQ-Correctness Expert as the denominator, while another expert reaches 124.4%? After correcting according to the unified definition of "best expert," how should the summary and retention rates change?

Supplementary Evidence Affecting Judgment: Can four-expert projection ablation provide paired results for multiple independent training seeds? The dual-expert projection setup is almost inactive, and the current four-expert error bar only reflects evaluation sampling, which is insufficient to determine the stability of projection gain on training randomness.

2 comments

Important clarification regarding the missing appendices: Appendices A–E were included in the original submission as a separate supplementary PDF, containing Algorithms 1–2, full implementation details, reward definitions, raw metrics, and additional experimental results. We sincerely apologize that these materials were difficult to locate. To make the paper easier to review, we have now consolidated the appendices into the revised manuscript after the references. We hope this addresses the reviewer's primary concern regarding the availability of the methodology and supporting evidence.

Retained-gain normalization: Thank you for catching this inconsistency. You are correct that the RQ-verification expert should define the denominator for reasoning correctness. We have corrected the normalization throughout the manuscript. LMOPD's four-expert RQ-correctness retention changes from 88.6% to approximately 71.2%, without changing the relative ranking of methods on that objective.

Self-distillation: We apologize for the imprecise wording. The KL-to-anchor term initially has zero gradient when the student equals the frozen anchor, but the sampled hard-token cross-entropy can still produce gradients. We have clarified this distinction in the revised manuscript.

Projection: We agree that additional seeds would strengthen the evidence. Given the substantial compute cost of four-expert OPD, we prioritized a checkpoint-matched ablation, which shows accuracy retention increasing from 54.0% to 90.6% with projection. We believe this provides meaningful evidence of the intended effect, while acknowledging that training-seed variability remains unmeasured.

We appreciate the reviewer's detailed feedback, particularly the normalization correction, which has helped us improve the manuscript.

Thank you again for your detailed review. We wanted to briefly follow up before the discussion period closes on November 18.

In particular, we hope our clarification regarding the appendices addresses your primary concern about the availability of the methodology and supporting evidence. These materials were available under the Supplementary Material attachment on the original OpenReview submission, and have now been consolidated into the revised manuscript for easier reference.

We would be very grateful if you could reconsider your assessment in light of these clarifications and revisions, and would be happy to address any remaining concerns before the discussion period closes.

Reviewer_5bgnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Lexicographic Multi-Objective On-Policy Distillation (LMOPD) for integrating reward-specialized policies under an explicit objective priority. For each student rollout, LMOPD routes supervision to the specialist corresponding to the highest-priority unmet objective and projects the selected specialist’s token-level correction to reduce conflicts with higher-priority specialists. The method is evaluated in two- and four-expert mathematical reasoning settings on a proprietary 30B-scale MoE model, with comparisons against multi-objective RL and policy-merging baselines.

Critical strengths

The problem setting is meaningful. The paper targets multi-objective post-training where lower-priority objectives should not compensate for degradation on more important ones, which is a relevant limitation of standard scalarized reward optimization. The method has a clear and structured design. LMOPD separates the problem into lexicographic rollout routing and priority-aware projection, making the role of objective priority explicit and the overall approach relatively easy to interpret. The paper includes useful component ablations. The comparisons between random routing, routing-only LMOPD, and full LMOPD help isolate the contributions of teacher assignment and projection. The two-expert setting is also replicated across three training seeds.

Critical weaknesses
  1. The experimental scope is still limited. The main experiments are conducted on a proprietary 30B-scale MoE model, using a proprietary math training set, and the final evaluation is limited to three math benchmarks. It would be useful to see whether the method can generalize to open-weight models and non-math tasks.

  2. The evidence for the projection component is not very strong. In the two-expert setting, projection is almost inactive, while the main improvement from projection appears in the four-expert setting. However, the four-expert result is only reported with one training seed. It would be helpful to know whether this improvement is stable across different runs.

  3. The comparison with GDPO may not be fully fair. The conciseness specialist is trained with a correctness-gated conciseness reward, while GDPO directly optimizes the raw conciseness score. Since part of the GDPO degradation is attributed to reward hacking, it is unclear how much of the improvement comes from LMOPD itself and how much comes from the different reward design.

  4. Several important evaluation metrics depend on the same LLM judge. Reasoning correctness, verification, clarity, and efficiency are all evaluated by the same 31B Gemma judge. Some validation with another judge or human evaluation would make these results more convincing.

  5. The presentation could be improved. Several tables are quite dense, and the font sizes are small and not fully consistent across the paper. This makes the main comparisons harder to read than necessary.

Questions for authors

Please see the questions and clarification requests raised in the "Critical Weaknesses" above.

1 comment

Thank you for the thoughtful feedback and constructive suggestions. We appreciate the concerns about presentation and have revised the manuscript, particularly the readability of the tables and component ablations.

Projection: We agree that additional seeds would strengthen the evidence. However, four-expert OPD requires substantial on-policy generation and token-level distillation with four ~30B-parameter MoE teachers. Given this cost, we prioritized controlled ablations over repeated full-scale training runs. In the checkpoint-matched four-expert comparison, projection improves accuracy retained gain from 54.0% to 90.6%, and reasoning-correctness retained gain from 55.9% to 71.2% (corrected normalization). We believe this provides meaningful evidence for projection's intended effect, while acknowledging that training-seed variance remains unmeasured. The two-expert setting additionally includes three training seeds.

GDPO: We appreciate this observation and agree that the reward-design difference deserves clarification. Our correctness-gated multi-task RLVR baseline helps address whether correctness gating alone explains the improvement. We have clarified this distinction in the revised manuscript and the limitation of the ungated GDPO comparison.

Judge dependence: We agree this is a valid concern. Answer accuracy is independently verifiable, and we have clarified the limitations of relying on one judge for the remaining metrics in the revised manuscript.

Experimental scope: We agree that evaluation on additional models and domains would strengthen the paper. We focus on mathematical reasoning because it allows us to study multiple competing objectives on the same prompts, with independently verifiable answer correctness. While the proprietary training data limits exact reproduction, we provide the full algorithmic specification and evaluate on three public benchmarks. We have clarified the scope of our conclusions accordingly in the revised manuscript.

Thank you again for the suggestions, which have helped us improve the presentation and precision of our claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.