- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper introduces Lexicographic Multi-Objective On-Policy Distillation (LMOPD). The paper argues that random/equal contribution from the existing method is not optimal, so they detect the first priority that exhibits a deficiency. Upon detection, the specialized policy is identified. Further filtering with a negative inner product with the high-priority expert is added to ensure all the policies align with each other. The paper reports the highest retained gain compared to baselines. No ablation is reported.
Critical strengths
- The hypothesis is sound and potentially helpful for post-training.
- The paper evaluates the models on a large amount of public data and shows improvement compared to baselines.
Critical weaknesses
- The paper reported relative retained gains seem to be odd and not very conventional. The relative gains get inflated quickly when the potential gain is small. I saw that the paper reported the raw performance in the Appendix and evaluated the reported raw performance.
- The evaluation protocol seems to be weak. The paper assumes that the used order is optimal; however, it might not be the case. The paper should evaluate different priority orders to avoid such a prior.
- The ablation study is virtually poor and insufficient to address the modules added in this paper.
- The results are impossible to reproduce because the paper uses a private training set.
- Missing baselines: SPA (https://arxiv.org/abs/2511.06222, AAAI, published March 2026)
Questions for authors
- How was the upper bound of the performance estimate over all possible priority orders?
3 comments
We wanted to briefly follow up before the discussion period closes on November 18. In particular, we hope our clarifications regarding the externally specified priority ordering and the existing component ablations (Table 3 of the original submission) help address your concerns. We have also made the ablations more prominent and added a discussion of SPA in the revised manuscript.
We would be very grateful if you could reconsider your assessment in light of these clarifications and revisions, and would be happy to address any remaining concerns before the discussion period closes. Thank you again for your time and feedback!
Thanks for the rebuttal. The rebuttal addressed my concerns. I increased the rating to 3.
Thank you for the feedback and helpful suggestions. We appreciate the opportunity to clarify several aspects of the evaluation and methodology.
Retained gain: We agree that normalization can amplify differences when the base-to-specialist gap is small. We report raw performance in the appendix and have made it more prominent in the revision. We also corrected an inconsistency in the four-expert normalization, detailed in our response to Reviewer az51. Thank you for highlighting this issue.
Priority ordering: We apologize that this distinction was not sufficiently clear. In LMOPD, the priority ordering is an externally specified preference, not a hyperparameter whose optimal value we seek to discover. For example, prioritizing correctness over conciseness expresses the requirement that shorter responses should not come at the expense of correctness. Reversing this ordering would represent a different preference, not necessarily a better solution. Selecting an optimal ordering across conflicting objectives would require an additional criterion for comparing those preferences, which is outside our scope. We do not claim lexicographic optimality for a given ordering either; LMOPD uses the supplied priorities to guide teacher selection and projection. We have clarified this distinction in the revised manuscript.
Ablations: We apologize that these were difficult to locate. Table 3 of the original submission compares full LMOPD, routing-only LMOPD, and random routing in both the two- and four-expert settings, with additional results in the supplementary appendix. These isolate the contributions of lexicographic routing and projection. We have made these comparisons more prominent. SPA: Thank you for pointing out this relevant work. We have added SPA to Related Work. While both methods incorporate lexicographic priorities, SPA constructs preference pairs through self-evaluation and refinement, then performs preference optimization. In contrast, our setting is the integration of existing reward-specialized experts through on-policy distillation. SPA does not directly operate on these experts or provide an alternative teacher-selection mechanism; adapting it would require a separate preference-generation and optimization pipeline. We therefore believe the existing multi-task RLVR, GDPO, Rewarded Soup, and routing ablations provide more directly comparable baselines for evaluating our contributions.
Reproducibility: We agree that proprietary training data limits exact reproduction. However, the use of unreleased training data is not uncommon in accepted large-scale LLM research (e.g., Jamba, ICLR 2025). Our contribution is a general training algorithm rather than a dataset or model release. We provide the complete mathematical formulation, algorithms, hyperparameters, and experimental details, with evaluations on three public benchmarks. We believe these allow the methodology and empirical findings to be meaningfully assessed despite the proprietary training infrastructure.
We appreciate the suggestions and hope the revisions address the main concerns.
Lenz, Barak, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram et al. "Jamba: Hybrid transformer-mamba language models." In International Conference on Learning Representations, vol. 2025, pp. 67959-67984. 2025.