-
Limited novelty relative to closely related work, particularly Preference Orchestrator (major concern).
The central components of MEAL substantially overlap with established multi-objective alignment techniques. Most notably, Preference Orchestrator (Liu et al., 2025) already introduces a prompt-conditioned preference network that predicts objective weights from input prompts and learns these weights using normalized reward-model scores. This closely resembles MEAL's preference-weight network and its reference preference construction in Eq. (5). Rewards-in-Context (Yang et al., 2024) and Directional Preference Alignment (Wang et al., 2024) also establish preference-conditioned generation and rejection-sampling-based alignment.
The main remaining distinction appears to be the alternating optimization of the preference network and policy using downstream SFT loss. However, Appendix A explicitly acknowledges that the implementation does not differentiate through the inner optimization trajectory and instead uses a direct first-order gradient with the updated policy treated as fixed. This makes the claimed bilevel meta-learning mechanism substantially less distinctive than suggested by the paper's framing.
Furthermore, the latent preference injection in Eqs. (2–4) effectively introduces a preference-dependent additive bias to the output logits. The paper does not sufficiently establish why this mechanism, combined with ordinary alternating optimization, represents a substantial methodological advance. A direct comparison with Preference Orchestrator and an ablation isolating the additional task-gradient feedback are particularly important.
-
Insufficient experimental evidence to establish the claimed superiority (major concern).
The experimental evaluation is limited primarily to reward trade-off curves on two benchmarks, without numerical hypervolume comparisons, uncertainty estimates, or repeated-seed results for the main methods. More importantly, the same reward-model objectives appear to be used both for optimizing the policy and for assessing the reported trade-offs, making it difficult to determine whether improved proxy rewards translate into better alignment with independent human preferences.
The fairness of the baseline comparisons also requires clarification. Section 2 states that some reported baseline results are shared with closely related work, while the present study uses Qwen3-based models. The earlier Preference Orchestrator study used different base models in its corresponding experiments. It is therefore important to establish whether all baselines were rerun under comparable models, reward evaluators, and training budgets.
Finally, the relationship between automatically predicted preference weights and the externally varied preference weights used to construct the evaluation curves is insufficiently explained. Since the proposed preference network generates weights as a function of the prompt alone, the paper should clarify how arbitrary user-specified preferences are supported and evaluated. Together, these issues substantially weaken the empirical evidence for the central claims.
[1] Liu, B., Xu, N., Yang, J., & Geng, X. (2025). Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models. arXiv preprint arXiv:2511.10656. https://arxiv.org/abs/2511.10656
[2] Yang, R., Pan, X., Luo, F., Qiu, S., Zhong, H., Yu, D., & Chen, J. (2024). Rewards-in-Context: Multi-Objective Alignment of Foundation Models with Dynamic Preference Adjustment. Proceedings of the 41st International Conference on Machine Learning (ICML), 235, 56276–56297. https://proceedings.mlr.press/v235/yang24q.html
[3] Wang, H., Lin, Y., Xiong, W., Yang, R., Diao, S., Qiu, S., Zhao, H., & Zhang, T. (2024). Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 8642–8655. https://aclanthology.org/2024.acl-long.468/