acceptodds
Under review as a conference paper at ICLR 2027

MOMoE: Instantiating Mixture-of-Experts for Multi-Objective LLM Alignment

Abstract

Aligning Large Language Models (LLMs) for diverse users and contexts requires balancing objectives such as helpfulness and harmlessness, yet no single trade-off between them suits all. Existing methods steer a single model to any user-specified preference at inference time, typically by composing objective-specific models through merging coefficients. However, they do not consistently surpass MORLHF, which trains a separate policy per preference. We trace this limitation to these methods adopting only the routing interface of a Mixture-of-Experts (MoE), with frozen, objective-specific experts and unregularized routing. We thus propose Multi-Objective Mixture-of-Experts (MOMoE) through four design principles: (1) joint optimization trains experts and router together rather than the router alone over frozen experts; (2) free experts are added with no objective assigned, reaching trade-offs the objective-specific ones cannot; (3) blend selection chooses each objective-specific expert by its synergy with the others rather than by its own objective; and (4) coverage regularization adds preference-conditioned losses that keep experts reachable and distinct. Extensive experiments show MOMoE outperforms state-of-the-art methods, attaining a broader Pareto front and higher utility (over 20% higher linear utility than MORLHF). We validate each principle through ablation and confirm gains through human evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-objective alignment, large language models, mixture-of-experts, controllable generation, RLHF, Pareto front, multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jHeuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MOMoE, a preference-conditioned mixture of LoRA experts for multi-objective LLM alignment. A shared frozen backbone is augmented with objective-specific and free experts, with layer-wise routers conditioned on the hidden state and user preference. The framework combines joint expert–router training, additional free experts, synergy-aware checkpoint selection, and coverage regularization. Experiments evaluate reward trade-offs and preference-conditioned utility across several tasks, with comparisons to seven baselines and targeted design ablations.

Critical strengths
  1. The framework is well motivated and coherently combines joint expert–router training with preference-aware initialization and regularization to improve the trade-offs served by a single controllable model.
  2. The method shows substantial gains across multiple tasks and baselines, supported by repeated runs and targeted ablations that provide useful insights into joint training and router warm-up.
Critical weaknesses
  1. The contribution of free experts is not fully separated from additional capacity. The default construction adds nn free experts to nn objective-specific experts, whereas removing them halves the expert count. This comparison therefore changes both expert roles and adapter capacity. The matched-count comparison of base and SFT initialization is informative, but it does not establish whether free experts are preferable to an equal-sized pool containing additional objective-specific experts. Such a control would clarify whether the gains depend on the proposed free-expert role or primarily on a larger trainable pool.

  2. Blend selection is not established as an effective independent contribution. In the ablations on pages 8–9, the default blend criterion does not consistently outperform simpler diagonal-only checkpoint selection, and the ordering varies across tasks and backbones. Appendix F also reports a stronger alternative selection criterion on Beaver, although that result is limited to one seed. These findings leave the practical benefit of the proposed criterion uncertain and weaken the claim that each of the four principles is independently validated. The overall framework remains useful, but the contribution attributed to blend selection should be more cautiously stated.

Questions for authors
  1. At matched capacity and training budget, how does adding nn objective-specific experts compare with adding free experts?
  2. Given the mixed ablations, what evidence supports blend selection as an independently effective component?
2 comments

We thank the reviewer for the constructive comments and for finding the framework useful. We address each point below.

W1 and Q1: Isolating the free-expert role from capacity.

Removing free experts (m = 0) changes both expert roles and pool size, and results in the original submission already bear on this: at the same N, replacing all objective-specific experts with free experts collapses the front (HV 0.119 vs. 0.588; pure free experts in Appendix F), so a larger pool alone does not yield the gains.

To test the requested control directly, we have added an equal-sized pool on all four tasks with two seeds in the revised manuscript (Sec. 4.4, l. 442–445; Table 14), with raw results in results/additional_experiments of our anonymous repository: the free experts are replaced with additional objective-specific experts, initialized from another checkpoint of each single-objective run, keeping N, routing, and training budget unchanged. This lowers every metric in all eight task–seed runs, e.g., mean HV from 0.60 to 0.56 on Beaver, indicating that the gains depend on the free-expert role rather than a larger pool.

W2 and Q2: Evidence for blend selection.

We agree that blend selection is not established as an independent contribution, as noted in Limitation (1). Since submission, we have extended the own-objective variant in Appendix F to all tasks and seeds: it raises mean HV on Beaver and LFQA but lowers it on Assistant and Summarization, so its gains likewise vary across tasks. In the revised manuscript, we have stated the contribution of blend selection more cautiously and narrowed the original claim that each principle is independently validated (abstract, l. 26–37; Sec. 4.4, l. 450-451; Sec. 5, l. 494–496; Appendix F, l. 1507–1511). In the final version, we will change blend selection from one of the design principles to an observation, describing it instead as part of expert initialization and moving its analysis (Sec. 3.4 and Theorem 3) to the appendix. All experts remain initialized as before, so the experimental setup and main results are unchanged.

We hope the added control for free experts and the revised treatment of blend selection address the reviewer’s concerns, and we welcome further discussion if any remain.

Thanks for the response. I am keeping my original rating.

Reviewer_fKkDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MOMoE jointly trains experts and a preference-conditioned router. Across five tasks, it reports strong reward-model utility and hypervolume, plus ablations, cost results, and human evaluation on two tasks.

Critical strengths

The problem is worth studying and the architecture is coherent. The main study covers five tasks, several baselines, ablations, costs, and human evaluation. A smaller second backbone is used in the ablations.

Critical weaknesses

The evidence does not isolate the proposed mechanism. Jointly trained and free experts add capacity and optimization freedom over frozen-expert baselines, but no control matches parameters, training tokens, and compute. The results therefore cannot separate the four proposed principles from added capacity and joint training. The alignment-specific novelty of these established MoE components is also unclear.

Independent validation is limited. Human evaluation covers Beaver and LFQA; LFQA is not significant (p=0.36), and agreement is low (Fleiss’ κ=0.26 overall; 0.10 on LFQA). Reward metrics use models closely tied to optimization, and the two-seed results have no significance tests; Beaver hypervolume is nearly tied with MORLHF (0.603 vs. 0.602). The theory gives no optimization guarantee, while latency is 4.2–5.8 ms/token versus 1.8–2.3 ms for several baselines.

Questions for authors

Add a jointly trained control matched for parameters, training tokens, and compute, but without free experts or the proposed routing design. How much gain remains after controlling for capacity and optimization freedom?

Add independent evaluation and uncertainty tests on more tasks. Narrow the human and theoretical claims, and report quality against memory and latency.

3 comments
AuthorsOpenReview ↗

Response to Reviewer fKkD (1/2)

We thank the reviewer for the feedback. Several of the main concerns, especially the alignment-specific novelty, reward-based evaluation, Beaver hypervolume, inference latency, and quality against cost, are addressed by results in the original submission or follow standard protocols. We clarify each point below, numbering the weaknesses and questions by paragraph (W1–W2, Q1–Q2).

W1 (a) and Q1: Isolating the design principles from capacity.

MOMoE differs from frozen-expert baselines in both joint training and free experts, each of which also adds trainable parameters or experts. Results in the original submission already bear on both. At the same N, replacing all objective-specific experts with free experts collapses the front (HV 0.119 vs. 0.588; pure free experts in Appendix F). MOMoE also achieves 21% higher linear utility on Beaver than MORLHF, which jointly trains policies without free experts or routing at a much higher total cost (34.9 h for 11 policies vs. 10.8 h for MOMoE including its experts).

To isolate each principle directly, we have added two capacity-matched controls in the revised manuscript (Sec. 4.4, l. 434–437 and 442–445; Table 14), with raw results in results/additional_experiments of our anonymous repository:

(1) Joint training. A single LoRA of rank N·r, matching the trainable parameters of MOMoE’s expert pool, is initialized as the equal-weight average of the n objective-specific experts, conditioned on the preference via the prompt, and trained with the same steps, batch size, and group size (hence the same training tokens and comparable compute), but without routing or free experts. On Beaver, its mean HV of 0.43 versus MOMoE’s 0.60 leaves a gain of 0.17 after matching capacity and optimization freedom, and across all eight task–seed runs it trails MOMoE on all metrics in seven (all but Assistant seed 42).

(2) Free experts. The free experts are replaced with additional objective-specific experts, initialized from another checkpoint of each single-objective run, keeping N, routing, and training budget unchanged. On all four tasks with two seeds, this lowers every metric in all eight task–seed runs, e.g., mean HV from 0.60 to 0.56 on Beaver.

W1 (b): Alignment-specific novelty.

We respectfully disagree. Each MoE design in MOMoE is tailored to multi-objective alignment, where expert roles are imposed by the reward models rather than emerging during training (Sec. 2):

(1) Joint optimization uses shared reward normalization and a preference-aligned router warm-up. Without the warm-up, the router ignores the preference (μ-sensitivity drops from 0.313 to 0.014; Table 15).

(2) Free experts are reserved a minimum routing share so that they can fill the interior of the Pareto front (Theorem 2), and objective-specific experts are initialized by their synergy with the other objectives rather than their own reward alone (Eq. 4).

(3) Coverage regularization balances routing across the preference simplex, preserving the per-preference concentration that extreme trade-offs require (Table 15).

Without these designs, an MoE with purely emergent expert roles collapses to a near-single point on the front (HV 0.119 vs. 0.588; pure free experts in Appendix F).

W2 (a) and Q2 (b): Human evaluation and agreement.

Agreement is inherently modest when judging trade-offs. On Beaver, annotator pairs give the same label (A, B, or tie) on 57.6% of comparisons, and agree on 72.0% of those where neither marked a tie; for reference, InstructGPT reports 72.6% agreement among its labelers (Ouyang et al., 2022), without specifying how ties are handled. Agreement on LFQA is lower, consistent with prior work: Su et al. (2022) report Fleiss’ κ as low as 0.12 for relevance in pairwise LFQA comparisons with three annotators, and Xu et al. (2023) find that even domain experts often disagree on long-form answers. On Beaver, MOMoE’s advantage is significant (p = 0.003). Accordingly, in the revised manuscript, we have qualified the human-evaluation claims in the abstract (l. 28), Sec. 4.5 (l. 474–477), and Sec. 5 (l. 496–497), stating that MOMoE is preferred on both tasks but significantly only on Beaver (p = 0.36 on LFQA).

W2 (b): Reward-based evaluation.

Our reward-based evaluation follows the standard protocol in multi-objective LLM alignment, training and evaluating policies with the same publicly released reward models as our baselines (Ramé et al., 2023; Yang et al., 2024; Xie et al., 2025; Li et al., 2026; Kong et al., 2026). These reward models are given and define the objectives. Their quality is orthogonal to our task of realizing the requested trade-off among them, and MOMoE is agnostic to their choice. All methods are evaluated on the same normalized scale (Sec. 3.2), ensuring a fair comparison.

AuthorsOpenReview ↗

Response to Reviewer fKkD (2/2)

W2 (c) and Q2 (a): Significance tests and evaluation scope.

We evaluated five tasks with two to five objectives and reported seed standard deviations for the four main tasks (Tables 8–12): MOMoE’s margin over the best baseline exceeds three standard deviations on 10 of 12 task–metric pairs, the exceptions being Beaver HV and Assistant Tchebyshev utility. In the revised manuscript, we have also added paired Wilcoxon signed-rank tests over the preference grid with Holm correction (Table 13): MOMoE’s advantage is significant in 51 of 54 comparisons, and no baseline is significantly better in any comparison. Human evaluation covered the two-objective tasks, as trade-offs among more objectives are hard for annotators to judge.

W2 (d): Beaver hypervolume.

MORLHF trains a separate policy for each of the 11 preferences (34.9 h vs. 10.8 h for MOMoE) and cannot serve unseen ones. At matched HV, MOMoE still achieves much higher linear (0.748 vs. 0.617) and Tchebyshev utility (−0.111 vs. −0.221), and leads HV on all other tasks, by up to 2.2× on LFQA.

W2 (e) and Q2 (b): Scope of the theory.

Our theorems characterize what joint training and free experts can reach beyond the frozen-expert front, rather than providing convergence guarantees, which are beyond the scope of this work. In the revised manuscript, we have made this explicit in Sec. 3 (l. 187–188) and revised the statement of Theorem 1’s implication accordingly (Sec. 3.2, l. 219–220).

W2 (f) and Q2 (c): Inference latency and quality against cost.

Table 6 of the original submission already reports training time, per-token latency, preference-switching time, and training memory for every method. MOMoE’s per-token latency is higher (4.2–5.8 ms vs. 1.8 ms for dense baselines), but it switches preferences instantly, versus up to 7.9 s for methods that reload or merge weights per preference, attains the best utility and HV on all four tasks, and trains far faster than MORLHF (10.8–23.3 h including the shared experts vs. 34.9–156.0 h). We acknowledge this overhead in Limitation (4), which top-k routing would reduce.

We hope the added controls, significance tests, qualified claims, and clarifications address the reviewer’s concerns, and we welcome further discussion if any remain.

References: Su et al. (2022). Read before Generate! Faithful Long Form Question Answering with Machine Reading. Findings of ACL. Xu et al. (2023). A Critical Evaluation of Evaluations for Long-form Question Answering. ACL.

Dear Reviewer fKkD, thank you again for your comments. We wanted to briefly follow up on our response of Nov 11, in particular the new capacity-matched controls (Sec. 4.4, Table 14) and significance tests (Table 13) in the revised manuscript. We would be grateful to know whether these address your concerns, or whether any questions remain.

Reviewer_sBz3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies controllable multi-objective alignment for large language models, where different users may prefer different trade-offs among objectives such as helpfulness and harmlessness. The authors argue that existing MoE-based methods are limited because they primarily train routers over frozen objective-specific experts. To address this issue, they propose MOMoE, a framework that jointly optimizes experts and routers, introduces free experts without predefined objectives, selects objective-specific experts based on their compatibility with other experts, and applies coverage regularization to maintain expert utilization and diversity. Experiments across multi-objective and many-objective alignment tasks show that MOMoE achieves broader Pareto fronts and higher utility than existing methods.

Critical strengths
  1. The motivation of this paper is novel. On the one hand, it adjusts the allocation among multiple objectives at the architectural level; on the other hand, it jointly optimizes the routing mechanism and the experts themselves, thereby improving overall performance through joint training.
  2. The experimental evaluation is comprehensive, with comparisons against multiple state-of-the-art methods, and the results demonstrate clear performance improvements.
  3. The paper provides theoretical justification for the effectiveness of the proposed method.
Critical weaknesses
  1. The proposed method is primarily applicable to MoE-based architectures and may offer limited benefits for improving dense models. Moreover, the architectural modifications may introduce additional challenges and overhead for inference optimization.
  2. The paper should cite and discuss some recent related work, such as Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models and Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment. Both methods learn improved routing or preference-allocation mechanisms, which is related to some of the key ideas explored in this work.
Questions for authors

See weaknesses.

3 comments

We sincerely thank the reviewer for the constructive comments. Below are our point-by-point responses:

W1 (a): Applicability to dense models.

We would like to clarify that MOMoE does not require an MoE model as its starting point. It starts from a dense LLM and constructs the MoE by attaching LoRA adapters as experts, together with per-layer routers (Eq. 2); all our experiments start from dense models (Llama-2-7B and Qwen2.5-0.5B). Relative to MORLHF, which aligns the same dense model directly for each preference, MOMoE achieves 20–54% higher linear utility (Table 1). We have made this more explicit in Secs. 1 and 3.1 of the revised manuscript (l. 60–61 and 210–213).

W1 (b): Inference overhead.

We acknowledge the inference overhead (Table 6, Limitation (4)): per-token latency is 4.2–5.8 ms vs. 1.8 ms for dense baselines such as MORLHF, RS, and RiC. In exchange, MOMoE serves all preferences from one model with zero preference-switching time, versus up to 7.9 s for methods that reload or merge weights per preference, and attains the best utility and HV on all four tasks. Beyond this overhead, the architectural modifications pose no further challenges for inference optimization: the backbone, attention, and KV cache remain unchanged, and unlike standard MoE, there is no expert-parallel communication or token dispatch. Top-k routing would further reduce the latency.

W2: Related work on preference allocation.

Preference Orchestrator (PRO) and Meta-Aligner (MEAL) address a different question from MOMoE: they learn which preference weights to target, whereas MOMoE realizes a given preference by routing over a jointly trained expert pool. They are thus complementary to MOMoE, as a preference predicted by these methods could serve as the input to its router. To make this distinction explicit, we have added a discussion of PRO and MEAL, together with other preference-inference work, to the related work in Appendix A (l. 714–718).

We hope these clarifications address the reviewer’s concerns, and we welcome further discussion if any remain.

Dear Reviewer sBz3, thank you again for your comments. We wanted to briefly follow up on our response of Nov 11. The corresponding changes are highlighted in blue in the revised manuscript (Secs. 1 and 3.1 and Appendix A). We would be grateful to know whether these address your concerns.

Thank you for your detailed response. It has addressed my concerns. Taking the other reviewers’ comments into account, I have decided to maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of the discussion

  1. Dense backbones (sBz3): MOMoE builds on dense LLMs (Llama-2-7B, Qwen2.5-0.5B), with LoRA adapters as experts.

  2. Related work (sBz3): PRO and MEAL are discussed in Appendix A.

  3. Capacity (fKkD, jHeu; Table 14): a single-LoRA control matched in parameters, training tokens and compute trails MOMoE in 7 of 8 task–seed runs (Beaver HV 0.43 vs. 0.60). An equal-N pool with extra objective-specific experts in place of the free experts lowers every metric in all 8 runs (Beaver HV 0.60 → 0.56).

  4. Significance (fKkD; Tables 8–13): MOMoE is significantly better in 51 of 54 paired Wilcoxon comparisons with Holm correction, and no baseline is significantly better in any.

  5. Cost (fKkD; Table 6): training time, latency, preference switching and memory are reported for every method; MOMoE trains in 10.8 h vs. 34.9 h for MORLHF on Beaver.

  6. Blend selection (jHeu): its gains are setting-dependent, as stated in Limitation (1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.