acceptodds
Under review as a conference paper at ICLR 2027

Portfolio of Experts

Abstract

Sparse mixture-of-experts (MoE) models increase model capacity without proportionally increasing per-token computation by activating only a small subset of experts. Route quality therefore depends on both expert utility and complementarity within the selected subset. However, most routers rank experts independently, potentially selecting experts with redundant task contributions. Effective routing must instead identify a high-utility, complementary expert subset under a fixed budget. We introduce Portfolio of Experts (PoE), which formulates sparse routing as conditional portfolio optimisation under outcome uncertainty. PoE defines counterfactual gradient returns (CGRs), which quantify the local effect of each expert's allocation on downstream loss. A lagged critic predicts each candidate expert's expected CGR and the pairwise covariance of these returns. A constrained solver then jointly selects and weights a small set of experts, favouring high predicted utility while avoiding redundant combinations under a fixed routing budget. Across seven zero-shot downstream tasks spanning science question answering, commonsense reasoning, reading comprehension, and contextual prediction, PoE attains the highest mean accuracy and lowest held-out NLL at every evaluated expert-pool size. Frozen-route studies show that modelling pairwise dependence improves local routing utility, while routing diagnostics show that PoE lowers joint failure without activating more experts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Sparse Routing, Portfolio Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g6a7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed PoE which borrowed financial portfolio concept to improve moe model performance and lower the joint failure. While the application of it into llm domain is not very clear working for me. I also think the paper didn't discuss the weakness of the mean variance model, there are few reasons it's not well adopted in real life finance decision, although it's a great invention.

Critical strengths
  1. Idea is simple and seems effective, it's very interesting to think the problem as an investment problem
  2. Presentation is clear and easy to understand
Critical weaknesses

We don't really know how efficient this type of methods are for modern moe models e.g. is it useful for bigger models or smaller models? Are the method transferrable? Is it more useful when doing pretraining or post training? Is there any design criteria needs to be satisfied with the critic? Also we already know that mean variance model is very sensitive and there are some criticisms called "error minimization" say if one expert give you 10.2% gain and the other give you 10%, you will be dropping the other one completely but we also know there are super experts in moe arch: https://arxiv.org/abs/2507.23279 which might give you trouble

Questions for authors

Do you have any test results for modern open source models e.g. Qwen, Kimi or Deepseek? Expert is MLP how you make it as vector which can be comparable with the gradient?

7 comments
AuthorsOpenReview ↗

Response to Reviewer g6a7 (1/3)

We thank the reviewer for recognizing the simplicity, effectiveness, and clarity of PoE. We address the five concerns below, following the order in the review.

1. Is PoE useful for bigger models or smaller models?

PoE improves performance across all three evaluated expert-pool sizes, and its advantage becomes more pronounced as expert capacity increases.

We vary the number of fixed-width routed experts over N∈16,24,32N\in\\{16,24,32\\} while keeping the nominal activation budget at k=8k=8. The models have 12 layers and approximately 1B parameters at N=16N=16, with total capacity increasing with NN. Within each configuration, methods share initialization, data order, training horizon, and evaluation.

Table 1 shows that PoE achieves the highest mean downstream accuracy and lowest held-out NLL at every evaluated expert-pool size:

Routed expertsTop-kPoERelative gain
1648.1%50.0%4.0%
2450.9%54.3%6.7%
3253.9%59.4%10.2%

The relative improvement increases from 4.0% to 6.7% and 10.2%, providing encouraging evidence that PoE benefits from additional expert capacity. At N=32N=32, PoE achieves 59.4% mean accuracy compared with 56.0% for the group-selection baseline MP-MoE, an improvement of 3.4 percentage points.

Figure 3 further shows approximately an order-of-magnitude reduction in joint failure relative to Top-k while activating slightly fewer experts. This supports the effectiveness of selecting complementary expert combinations under a limited activation budget.

We also measure complete training cost. At fixed N=16N=16, PoE adds approximately 7.6-12.1% training step time relative to Top-k across k∈2,4,8k\in\\{2,4,8\\}, including critic updates, shadow evaluations, and support optimization. Its additional step time over MP-MoE is only 1.69% and 1.76% at k=4k=4 and 88. These measurements concern training overhead; inference retains the critic and solver while removing task gradients and shadow evaluations.

2. Is the method transferable?

PoE's formulation is applicable across MoE architectures that combine expert outputs through routing weights and provide a differentiable downstream task loss. The return definition and covariance-aware selection criterion are independent of a particular model family.

The transferable element is the routing formulation: estimate task-conditioned expert utility and dependence, then jointly select and weight experts under the activation budget. Adapting an existing checkpoint requires fitting the critic to that model's expert contributions and routing distribution; the critic is learned for the model and training setting.

Regarding the concern about Qwen, Kimi, and DeepSeek, the current manuscript does not report experiments on those pretrained checkpoints. We use controlled from-scratch training to isolate routing effects under matched conditions. The consistent gains across three expert-pool sizes support the routing principle.

AuthorsOpenReview ↗

Response to Reviewer g6a7 (2/3)

3. Is PoE more useful during pretraining or post-training?

Our experiments directly establish effectiveness during pretraining on approximately 100B Dolma tokens, where PoE improves both held-out language-modeling loss and downstream accuracy. Pretraining allows expert representations, routing, and the critic to develop together, and our results demonstrate that this joint learning is effective.

The return definition also applies naturally to gradient-based post-training, such as supervised fine-tuning: expert contributions can be measured using the fine-tuning loss, with the critic adapted to the target distribution. This provides a concrete route to task-specific routing during supervised post-training.

4. What design criteria must the critic satisfy?

The essential requirements concern the critic's information, prediction targets, covariance validity, and training coverage. The particular MLP architecture is an implementation choice.

  • Routing-time information: the critic receives the detached layer representation and the allocation at which moments are queried.
  • Task-conditioned moments: it predicts expected expert returns and their covariance. Supervision uses realized returns and raw second moments derived from the language-modeling loss.
  • Valid covariance: a lower-triangular factor CC gives Σ^stat=CC⊤\widehat{\Sigma}_{\mathrm{stat}}=CC^\top, ensuring positive semidefiniteness. A small ridge improves numerical conditioning.
  • Off-route coverage: training-only shadow experts are evaluated at zero allocation, providing observations for unselected experts without changing the forward result. Inverse-propensity weighting corrects coordinate and expert-pair subsampling.
  • Ordered updates: a lagged critic selects the current route; the resulting observations update future predictions. Detached targets avoid higher-order differentiation through the task gradient.

Our implementation uses one MLP per routed layer, with two hidden layers of width 256 and SiLU activations, totaling approximately 3.7M critic parameters at N=16N=16.

Connection between an MLP expert and the gradient. To address the reviewer's explicit question, the vector is the expert's output for the current token:

ei(s)=Ei(x(s))∈Rd.e_i(s)=E_i(x(s))\in\mathbb{R}^{d}.

The task gradient ga=∇yL[a]g^a=\nabla_y L^{[a]} is taken with respect to the layer output and lies in the same dd-dimensional space. Since y(v;s)=y0(s)+∑iviei(s)y(v;s)=y_0(s)+\sum_i v_i e_i(s), the chain rule gives

=-\langle e_i,g^a\rangle =-\left.\frac{\partial L^{[a]}(y(v;s);s,\xi)}{\partial v_i}\right|_{v=a}.$$ A positive return means that increasing the expert's allocation locally decreases the downstream loss. This gives the critic a task-grounded utility target. **Empirical validation.** Allocation-weighted CGR agrees strongly with exact frozen-route deletion effects: the Spearman correlations are **0.995, 0.988, and 0.951** at representative early, middle, and late layers. On held-out anchor queries, the sampled-shadow critic achieves **0.965** Spearman correlation for portfolio variance, a calibration slope of **0.974**, and **0.967** Spearman agreement with oracle pairwise dependence. The latter substantially exceeds raw output cosine similarity, which achieves **0.452**. Removing shadow coverage worsens both calibration and route quality. At inference, the frozen critic predicts the required moments from routing-time inputs. Task gradients, shadow evaluations, and online updates are unnecessary. Therefore, the critic and support solver remain part of routing.
AuthorsOpenReview ↗

Response to Reviewer g6a7 (3/3)

5. Mean-variance sensitivity, the "10.2% versus 10%" example, and super experts

We agree that sensitivity to estimated moments is a legitimate concern. However, a small difference in individual expected returns does not determine the selected portfolio: allocation depends jointly on expected utility, covariance, and feasibility constraints.

A small mean difference does not imply dropping the other expert.

PoE optimizes

\qquad \frac{\partial J}{\partial p_i}=\mu_i-\eta(\Sigma p)_i.$$ An expert with slightly lower expected utility can remain valuable if its returns complement those of the other selected experts. Consequently, the two values "10.2%" and "10%" alone are insufficient to conclude that the second expert receives zero allocation. Accounting for this dependence is precisely the motivation for joint portfolio selection. **The method explicitly addresses estimation sensitivity.** Our feasible set bounds total allocation, individual weights, and active support size. A trust region limits changes from the anchor route, and a covariance ridge improves numerical conditioning. Appendix A.6 also quantifies the consequences of imperfect predictions. Proposition 7 bounds the uniform portfolio-score error by $$\varepsilon_J =B\varepsilon_\mu+ \frac{\eta}{2}B^2 (\varepsilon_{\Sigma,\mathrm{stat}}+\epsilon),$$ and the resulting routing regret by $$J(p^\star)-J(\widehat p) \leq 2\varepsilon_J+\varepsilon_{\mathrm{opt}}.$$ These terms account for mean-estimation error, covariance-estimation error, the ridge, and solver error. The bound concerns regret in the portfolio objective; discrete supports can still change under perturbations. We validate the practical value of covariance through controlled ablations. In Table 10, full covariance improves the **exact-loss certainty-equivalent metric by approximately 71% over diagonal covariance**. The paired improvement is $4.968\times10^{-3}$ nats, with a 95% confidence interval of $[4.042,5.894]\times10^{-3}$, and is positive in **all five seeds**. Permuting expert-pair correlations while preserving marginal information also substantially reduces route quality. These results show that correctly identifying complementary expert pairs provides useful routing information despite estimation error. The mean-variance objective requires finite second moments. Exact equivalence to the CARA certainty equivalent holds under conditional Gaussianity. Appendix A.5 characterizes the approximation remainder for more general distributions under the stated regularity conditions. We will make these assumptions and the sensitivity analysis more prominent. **Super experts are consistent with PoE's motivation.** We thank the reviewer for highlighting the Super Experts study. Its central observation that experts can differ substantially in importance is consistent with our motivation to measure task-relevant expert contributions. PoE explicitly rewards experts according to their expected contribution to reducing downstream loss. Its return $r_i^a=-\langle e_i,g^a\rangle$ measures the local loss reduction associated with increasing expert $i$'s allocation. Experts with larger expected positive contributions are therefore favored by the utility term. Covariance-aware selection additionally considers which expert combinations provide complementary contributions under the activation budget. PoE thus accommodates highly unequal expert importance: **a particularly useful expert can receive substantial routing weight alongside complementary specialists**. Its covariance criterion concerns the joint variability of task contributions and imposes no requirement that experts receive equal weights. The cited study demonstrates the consequences of skipping critical experts in existing pretrained models, whereas our experiments jointly learn expert representations and routing during pretraining. **The presence of super experts does not imply an inherent conflict with PoE. It reinforces the importance of routing based on task-relevant expert contributions.** We appreciate the reviewer's questions and will clarify these points in the revision.
Reviewer_XNuHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Portfolio of Experts introduces a new solution to routing opposed to traditional sparse Mixture of Experts models by utilizing portfolio optimization to estimate and calculate a return of how well experts could potentially work together. Instead of choosing experts that are both useful but have high overlap, they optimize for finding experts that have more complementary behavior. They introduce the concept counterfactual gradient returns, which is just a dot product between output vector of an expert and the negative gradient w.r.t MoE layer output. Measuring this dot product allows them to see whether the expert aligns with the direction of the local minimum to optimize towards or not. They then use a learned router to that proposes potential experts which are judged by the critic on how likely to be compatible they are, which is optimized according to the portfolio objective they define. They then demonstrate that PoE achieved the lowest negative log-likelihood on a held-out test set for each expert count.

Critical strengths

● Counterfactual gradient returns are a new contribution that could be very useful to others in working on future MoE architectures. ● Empirical evidence of improvements over traditional top-k routing shows considering how complementary experts do better together this could be a future direction of how we approach designing MoE architectures.

Critical weaknesses

Limited novelty: The counterfactual gradient-return idea is clever, but it relies on standard gradient calculations. Existing methods: Portfolio optimization is already well established, so the main contribution appears to be combining existing ideas for MoE routing. Computational overhead: The method introduces additional computations through the critic and portfolio solver. Unclear scalability: Experiments only evaluate models with up to 32 experts, leaving uncertainty about performance on larger MoEs. Inference efficiency: The paper does not sufficiently demonstrate how the additional routing computations affect inference speed. Limited generalization: The method is evaluated on one model family, making it unclear whether the improvements transfer to other architectures. Overall assessment: While the results are promising, I am not convinced the paper makes a sufficiently significant methodological contribution for ICLR. I would like stronger evidence of novelty, scalability, and practical efficiency before recommending acceptance.

Questions for authors

● How does the computational overhead for training and inference compare with a traditional Top-k sparse MoE architecture, or the architectures you compared with empirically?

4 comments
AuthorsOpenReview ↗

Response to Reviewer XNuH (1/3)

We thank the reviewer for recognizing the potential usefulness of CGR and the empirical benefits of complementary expert selection. We respond to each weakness and question individually below.

Weakness 1: Limited novelty: The counterfactual gradient-return idea is clever, but it relies on standard gradient calculations.

Response: We agree that gradient calculation itself is standard. Our contribution is to turn loss-sensitive expert contributions into a task-conditioned criterion for joint sparse routing, together with a learning procedure that makes this criterion available before the outcome is known.

For expert output eie_i and layer-output gradient gag^a, CGR is

=-\left.\frac{\partial L^{[a]}}{\partial v_i}\right|_{v=a}.$$ This defines a signed marginal return: increasing an expert's allocation is locally beneficial when its return is positive. We then model the conditional mean and covariance of these returns to evaluate expert groups. Proposition 1 identifies the relevant dependence geometry: $$\Sigma^a(s)=E(s)^\top G^a(s)E(s),$$ where $E(s)$ collects expert outputs and $G^a(s)$ is the conditional covariance of the downstream layer-output gradient. Thus, complementarity depends on how experts contribute to the task loss under downstream uncertainty. Representational similarity alone does not characterize this dependence. The methodological contribution therefore concerns **what expert dependence should be modeled and how it should guide routing**. The standard gradient calculation provides the task-grounded signal. We also validate that signal: allocation-weighted CGR has Spearman correlations of **0.995, 0.988, and 0.951** with exact frozen-route deletion effects at representative early, middle, and late layers. **Weakness 2: Existing methods: Portfolio optimization is already well established, so the main contribution appears to be combining existing ideas for MoE routing.** **Response:** The established portfolio objective provides a useful mathematical foundation. PoE's contribution is the formulation and implementation of **conditional portfolio routing for experts**, replacing selection by per-expert ranking with explicit joint selection and weighting based on predicted task utility and dependence. Making this decision criterion operational requires an allocation-conditioned critic that predicts moments before routing, training-only zero-allocation shadows that provide observations for unselected experts, and constrained support optimization under the activation budget. Appendix A additionally analyzes local approximation, moment-prediction, and solver errors. The distinction from existing group-selection methods is concrete. MP-MoE constructs dependence from co-occurrence statistics of binary expert-selection indicators. PoE predicts the conditional dependence of signed, loss-sensitive expert returns. Controlled ablations substantiate the value of this distinction. In Table 10, full covariance improves the **exact-loss certainty-equivalent metric by approximately 71% over diagonal covariance**. The paired improvement is $4.968\times10^{-3}$ nats, with a 95% confidence interval of $[4.042,5.894]\times10^{-3}$. Full covariance also exceeds the permuted-correlation control by $6.915\times10^{-3}$ nats, with a 95% interval of $[5.464,8.367]\times10^{-3}$. Both contrasts are positive in **all five seeds**. The permutation control preserves marginal information while disrupting expert-pair identities. Its degradation shows that the benefit depends on correctly identifying complementary expert pairs. Together, the learning procedure and mechanism-identifying controls support a substantive methodological contribution to MoE routing.
AuthorsOpenReview ↗

Response to Reviewer XNuH (2/3)

Weakness 3: Computational overhead: The method introduces additional computations through the critic and portfolio solver.

Response: We quantify these additional computations using complete end-to-end training measurements. At fixed N=16N=16, PoE increases step time by approximately 7.6-12.1% relative to Top-k across nominal routing budgets k∈2,4,8k\in\\{2,4,8\\}. The measurement includes backbone computation, anchor routing, critic evaluation and updates, shadow-expert evaluations, and support optimization.

Relative to MP-MoE, PoE adds only 1.69% and 1.76% step time at k=4k=4 and 88, respectively. Thus, the measured additional cost is modest relative to the existing group-selection baseline at these budgets.

The critic uses two hidden layers of width 256 per routed layer, totaling approximately 3.7M parameters at N=16N=16. CGR targets use gradients already supplied by ordinary backpropagation, and detached targets avoid higher-order differentiation. Appendix D.4 and Tables 5-6 provide a component-level breakdown. We give the complete four-method comparison under Question 1 below.

Weakness 4: Unclear scalability: Experiments only evaluate models with up to 32 experts, leaving uncertainty about performance on larger MoEs.

Response: Our experiments provide encouraging evidence that PoE benefits from increasing expert capacity. We vary the routed-expert pool over N∈16,24,32N\in\\{16,24,32\\} per layer while keeping expert width, model depth, and the nominal activation budget k=8k=8 fixed. The N=16N=16 configuration contains approximately 1B parameters, and total capacity grows with NN.

Table 1 reports the following mean downstream accuracies:

Routed expertsTop-kPoERelative gain
1648.1%50.0%4.0%
2450.9%54.3%6.7%
3253.9%59.4%10.2%

PoE achieves the highest mean accuracy and lowest held-out NLL at every evaluated expert-pool size. Its relative improvement over Top-k increases from 4.0% to 6.7% and 10.2%. At N=32N=32, PoE achieves 59.4% mean accuracy compared with 56.0% for MP-MoE, an improvement of 3.4 percentage points.

The widening margin supports our motivation: a larger expert pool offers more possible combinations, increasing the value of selecting high-utility, complementary groups.

We agree that substantially larger expert pools would strengthen the scalability evidence. The current results establish quality improvements within the tested range. Dense covariance prediction has O(N2)O(N^2) output size, and support search adds computational cost. Our systems sweep fixes N=16N=16 while varying kk, so it does not measure cost scaling with NN. We will distinguish the demonstrated capacity-scaling trend from the broader computational-scaling question.

AuthorsOpenReview ↗

Response to Reviewer XNuH (3/3)

Weakness 5: Inference efficiency: The paper does not sufficiently demonstrate how the additional routing computations affect inference speed.

Response: At inference, the anchor proposes an allocation, the frozen critic predicts conditional return moments, and the support solver selects the expert group. The model executes at most kk routed experts per token.

Inference requires no task gradients, shadow-expert evaluations, critic learning, or online parameter updates. The critic and solver remain additional routing computations. The current manuscript does not include dedicated inference-latency or throughput measurements, and we will explicitly distinguish the measured training overhead from inference-time cost.

Figure 3 demonstrates substantially lower joint failure with slightly fewer active experts than Top-k. This establishes more effective use of the activation budget; actual serving speed also depends on critic and solver latency.

Weakness 6: Limited generalization: The method is evaluated on one model family, making it unclear whether the improvements transfer to other architectures.

Response: The current experiments use one model family to compare routing methods under matched initialization, data order, training horizon, and evaluation. Within that family, PoE consistently improves mean performance across three expert-pool sizes and seven downstream tasks.

The formulation applies to MoE layers with weighted additive expert outputs and a differentiable downstream loss. CGR operates on expert outputs and layer-output gradients, so its definition is independent of a particular expert parameterization. Always-active shared experts are explicitly accommodated.

Applying PoE to another architecture or pretrained checkpoint requires fitting the critic to that model's expert contributions and routing distribution. Cross-family evaluation would extend the current empirical evidence, and we will make this scope explicit.

Overall assessment: While the results are promising, I am not convinced the paper makes a sufficiently significant methodological contribution for ICLR. I would like stronger evidence of novelty, scalability, and practical efficiency before recommending acceptance.

Response: We respectfully believe the contribution is supported by complementary theoretical and empirical evidence: a task-grounded formulation of joint expert routing, controlled ablations identifying the importance of correctly modeled pairwise dependence, widening accuracy gains with increasing expert capacity, and measured full training costs.

Question 1: How does the computational overhead for training and inference compare with a traditional Top-k sparse MoE architecture, or the architectures you compared with empirically?

Response: For training, Table 2 compares all four methods on the matched 4×4\timesH200 setup at N=16N=16. Throughput is in thousands of tokens per second:

Routing budgetTop-kReMoEMP-MoEPoE
2335.1332.1322.8304.2
4290.3288.1274.3269.7
8217.9216.6197.8194.4

The corresponding increases in PoE training step time are:

Routing budgetOver Top-kOver MP-MoE
210.14%6.10%
47.64%1.69%
812.08%1.76%

These measurements include the complete PoE training path. Tables 5-6 further show that the support solver takes 0.291, 0.374, and 0.612 seconds per step at k=2,4,8k=2,4,8, accounting for approximately 5.6-7.6% of total PoE step time.

For inference, PoE retains the frozen critic and support solver while removing task gradients, shadow evaluations, and learning updates. Dedicated inference timings are not reported, so the training percentages above do not establish serving-time overhead. We will make this distinction explicit.

Dear Reviewer XNuH, as the author–reviewer discussion closes in approximately one hour, we would be grateful if you could take a moment to review our responses regarding PoE’s methodological contribution, computational overhead, and scalability. We would greatly appreciate hearing whether our explanations and supporting results address your concerns. We remain available to clarify any outstanding points before the deadline. Thank you again for your time and thoughtful feedback.

Reviewer_YmHrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Portfolio of Experts (PoE), which treats sparse MoE routing as a contextual mean-variance (Markowitz) portfolio problem: it selects a group of experts jointly instead of ranking each expert alone. It has four parts: (1) Counterfactual gradient returns (CGRs), r_i = -<e_i, g>, which measure the first-order effect of each expert's allocation on the loss. Training-only "shadow" experts are evaluated at zero weight so that experts outside the route also get targets. (2) A lagged critic that predicts the conditional mean and covariance of CGRs before routing. (3) A support solver (beam search over add/remove/swap moves plus a continuous QP) that maximises a mean-variance score inside a trust region around a softplus anchor router. (4) Route distillation, which trains the anchor toward the solver's output.

The theory shows that the return covariance equals E^T G E, i.e., expert overlap under the downstream gradient covariance. It also motivates the mean-variance objective with a CARA certainty equivalent and bounds the linearisation, moment-prediction and search errors.

Experiments train 12-layer MoE models (about 1B total parameters at N=16) from scratch on 100B tokens with N in {16, 24, 32} and k=8. They compare against Top-k MoE, ReMoE and MP-MoE on seven zero-shot tasks and held-out NLL. PoE reports the best mean accuracy and NLL at every N. Frozen-route ablations over five seeds attribute the gains to marginal variance, pairwise covariance and shadow coverage. Diagnostics report an order-of-magnitude lower "joint failure" than Top-k at a similar number of active experts, and about 10% training step-time overhead.

Critical strengths
  1. A well-motivated idea with a meaningful theoretical lens. Selecting a complementary group of experts, rather than k individually strong but redundant ones, is a natural and under-explored question. Proposition 1 (covariance = E^T G E) gives a clear way to see that "task-relevant overlap" differs from representation similarity. This framing could influence future router design, and it is the main reason I find the paper interesting.

  2. A careful statistical protocol for the mechanism studies. The frozen-route component study (Table 10, Figure 4) uses five paired training seeds, pre-specified contrasts, t-intervals and a permutation placebo. All reported contrasts and percentages are internally consistent with the arm-level values. The appendices are also honest about the limits of the theory (Remark 4, Proposition 10). This level of care is above average.

Critical weaknesses
  1. The claimed mechanism (pairwise covariance improves routing) is not shown end-to-end, and some evidence points the other way.
  • No end-to-end ablation. All component ablations are frozen-route evaluations. There is no Table 1-style result (accuracy and NLL after full training) for eta = 0 (mean-only) or diagonal-only PoE, so it is unknown whether the covariance term contributes to the headline gains at all.
  • The evaluation metric favours the method. The frozen-route metric is a risk-averse CARA certainty equivalent. It structurally rewards variants that penalise variance, so part of the 77% "diagonal vs. mean-only" gain may come from how the metric is defined.
  • Controls without pair information explain most of the joint-failure gain. In Appendix D.3, the permuted-covariance and uniform-covariance controls reach 3.58% and 3.38% joint failure, while Top-k is roughly 12.5% (read from Figure 3b). Most of the reduction therefore remains when pair information is destroyed, which contradicts the introduction's attribution to pairwise dependence. Accuracy and NLL for these controls are not reported.
  • The risk-aversion rationale does not match the training goal. LM training minimises expected loss, and risk aversion trades expected return for lower variance, so at first order it should not improve NLL. Remark 4 itself admits that Eq. 14 is not the certainty equivalent of the loss reduction. This weakens the "risk-sensitive utility" justification listed as a contribution.
  • The trust region leaves little room for the solver. With B = 1, k = 8 and tau = sqrt(2)/k (about 0.177), a uniform anchor allows roughly one equal-weight swap. Much of the difference from Top-k may come from the different anchor router (softplus plus distillation) or from shadow-expert gradients rather than from portfolio selection. No ablation separates these factors.
  • The risk coefficient is not reported. eta_route and eta_eval, the central hyperparameters of the method, do not appear in Table 4 or the text.
  1. The main results raise reliability and practicality concerns.
  • Table 1 is a single run with no variance, although its gains are the paper's central claim.
  • The scores are unusually high. The model has about 0.54B active parameters, a GPT-2-small-sized backbone (12 layers, d = 768) and 100B training tokens, yet PoE reaches 76.3% on ARC-Easy and 59.4% on HellaSwag at N = 32.
  • Accuracy gains do not track NLL. Routing alone improves HellaSwag by 16% relative to Top-k. At N = 32, PoE's NLL is only 0.031 nats lower than MP-MoE's, yet LAMBADA is 6.4 points higher and HellaSwag 5.8 points higher, although both benchmarks normally track language-modelling loss closely.
  • Baseline fairness is unclear: (a) No load-balancing loss is described for Top-k or PoE, and no expert-utilisation statistics are reported. (b) ReMoE activates about 12.4 experts per token in Figure 3b versus 9 for Top-k, which contradicts "matched budgets". Yet its throughput in Table 2 is almost identical to Top-k's (216.6 vs. 217.9 ktok/s). (c) MP-MoE, the strongest group-selection baseline, is missing from Figure 3.
  • The reported solver cost seems too low. Each step has about 1.57M tokens x 12 layers, i.e. about 18.9M routing problems, each with a beam search (width 32, two swap rounds) and QPs at 1e-6 tolerance. The reported solver time is only 0.612 s per step, roughly 32 ns per routing problem. This needs a clear explanation.
  • Inference and scaling are not addressed. No inference latency is reported, although efficiency is the purpose of MoE. The critic output grows as O(N^2) and the search is combinatorial, but only N <= 32 is tested, so the claim of a "scalable routing framework" is not supported.
  • The critic's mean prediction is weak exactly where routing happens: Table 9 reports a mean NRMSE of 0.775 on the anchor-query law, and the text does not discuss this.
Questions for authors

Clarifications (answers could change my assessment):

  1. What values of eta_route and eta_eval were used?
  2. What are the end-to-end accuracy and NLL (as in Table 1) for eta = 0 and for diagonal-only PoE? What are the accuracy and NLL of the two covariance-perturbation controls in D.3?
  3. How many seeds were used for Table 1, and what is the variance? Can you report at least three seeds for N = 16?
  4. Did Top-k MoE and PoE use a load-balancing loss? Please report expert-utilisation statistics for all methods.
  5. How does the solver handle about 18.9M routing problems in 0.6 s? How many QPs are solved per token-layer, and what is the inference latency of PoE compared with Top-k?
  6. What is the performance of (a) the PoE anchor router alone at inference, without the solver, and (b) Top-k with shadow-expert gradients but no critic or solver?
  7. Why does ReMoE in Figure 3b activate about 12.4 experts while its throughput in Table 2 is almost the same as Top-k's? Which N and k does Figure 3b use?
  8. Which exact data mix and 100B-token subset were used? Can you explain the high ARC-Easy and HellaSwag scores for a model of this size?

Suggestions (would not change my score alone):

  • Discuss related work that already uses gradients for experts outside the route, such as SparseMixer, GRIN and dense-gradient "Default MoE" methods. This affects the novelty of CGR.
  • Add MP-MoE and error bars to Figure 3, and state N and k in its caption.
  • Discuss the mean NRMSE of 0.775 in Table 9.
  • Proposition 1 and the introduction say the measure reduces to cosine "only when" G is isotropic, but Appendix A.3 proves only a sufficient condition.
  • The text calls the training data "Dolma" but cites the Olmo 3 report (Ettinger et al., 2025). Please clarify.
  • The formal machinery could be shortened. Propositions 1 and 3 are bilinearity identities, and Propositions 2 and 5-7 are standard Taylor and Cauchy-Schwarz bounds. Moving some of this to the appendix and using the space for end-to-end ablations would improve the paper.
  • Figure 2 adds little, and Figure 1 is crowded.
14 comments
AuthorsOpenReview ↗

Response to Reviewer YmHr (1/9)

We thank the reviewer for recognizing the motivation of task-relevant expert complementarity. We address each concern below.

W1: No end-to-end ablation. All component ablations are frozen-route evaluations. There is no Table 1-style result (accuracy and NLL after full training) for eta = 0 (mean-only) or diagonal-only PoE, so it is unknown whether the covariance term contributes to the headline gains at all.

Response: We trained Mean-only, Diagonal-only, Permuted, Constant-correlation, and Full PoE from scratch at N=16,k=8N=16, k=8 with three paired seeds, 41/42/43. Within each seed, we kept initialization, data order, training duration, and evaluation the same. We evaluated the final checkpoint at step 63,000, after approximately 100B training tokens.

Mean-only uses eta_route=0. Diagonal-only keeps each expert's variance and removes pairwise covariance. Permuted and Constant-correlation remove the association between correlations and specific expert pairs. Each critic-based variant trains its own critic and distills its own solver outputs. All five use the same anchor architecture, shadow protocol, and routing constraints.

Rebuttal Table 1. End-to-end results at N=16,k=8N=16, k=8. Mean Acc is the unweighted average over seven tasks, in percent. Accuracy and NLL are mean ± sample SD across three training seeds. JF is the three-seed mean.

MethodMean Acc (%) ↑Held-out NLL ↓JF (%) ↓
Top-k48.124 ± 0.1663.260 ± 0.004612.733
ReMoE47.733 ± 0.3513.250 ± 0.00531.480
MP-MoE49.076 ± 0.1973.234 ± 0.00364.800
Anchor-only, independently retrained48.395 ± 0.0083.254 ± 0.00108.200
Mean-only48.810 ± 0.1663.248 ± 0.00105.700
Permuted covariance48.948 ± 0.0503.244 ± 0.00103.580
Constant-correlation49.010 ± 0.1013.243 ± 0.00103.380
Diagonal-only49.376 ± 0.1083.237 ± 0.00102.400
Full PoE50.105 ± 0.1003.227 ± 0.00301.210

Full PoE improves accuracy and NLL over all four controls in every paired seed. Compared with Diagonal-only, it improves accuracy by 0.729 percentage points and lowers NLL by 0.010 nats. This comparison measures the contribution of pairwise covariance beyond marginal variance after full training.

W2: The evaluation metric favours the method. The frozen-route metric is a risk-averse CARA certainty equivalent. It structurally rewards variants that penalise variance, so part of the 77% "diagonal vs. mean-only" gain may come from how the metric is defined.

Response: We evaluated the fully trained variants on seven zero-shot tasks and held-out NLL. Neither evaluation metric contains a variance penalty.

As shown in Rebuttal Table 1, Diagonal-only reaches 49.376% accuracy and 3.237 NLL, compared with 48.810% and 3.248 for Mean-only. Full PoE improves these to 50.105% and 3.227. Both improvements hold in every seed.

We will use these accuracy and NLL results as the main evidence for downstream effectiveness, replacing the earlier risk-sensitive percentage comparison.

AuthorsOpenReview ↗

Response to Reviewer YmHr (2/9)

W3: Controls without pair information explain most of the joint-failure gain. In Appendix D.3, the permuted-covariance and uniform-covariance controls reach 3.58% and 3.38% joint failure, while Top-k is roughly 12.5% (read from Figure 3b). Most of the reduction therefore remains when pair information is destroyed, which contradicts the introduction's attribution to pairwise dependence. Accuracy and NLL for these controls are not reported.

Response: We trained the Permuted and Constant-correlation controls and measured their accuracy, NLL, and joint failure. Full PoE improves all three metrics over both controls in every seed.

JF falls from 3.58% for Permuted and 3.38% for Constant-correlation to 1.21% for Full PoE. These are further relative reductions of 66.2% and 64.2%. Accuracy rises from 48.948% and 49.010% to 50.105%. NLL falls from 3.244 and 3.243 to 3.227.

Rebuttal Table 2. End-to-end differences. Accuracy differences are percentage points. NLL differences are nats.

ContrastAccuracy difference ↑NLL difference ↓
Full − Diagonal-only+0.729−0.010
Full − Mean-only+1.295−0.021
Full − Permuted+1.157−0.017
Full − Constant-correlation+1.095−0.016
Diagonal-only − Mean-only+0.567−0.011
Mean-only − retrained Anchor-only+0.414−0.006

Constant-correlation preserves each expert's variance and assigns the same correlation to every off-diagonal pair. We clip the layer-average correlation to the valid range, which makes the matrix PSD without eigenvalue projection. Permuted reassigns correlations to expert identities while preserving each expert's variance. All variance-based variants use the same diagonal ridge.

W4: The risk-aversion rationale does not match the training goal. LM training minimises expected loss, and risk aversion trades expected return for lower variance, so at first order it should not improve NLL. Remark 4 itself admits that Eq. 14 is not the certainty equivalent of the loss reduction. This weakens the "risk-sensitive utility" justification listed as a contribution.

Response: We train the language model with token NLL. The portfolio objective selects experts using learned estimates of their local marginal returns. Mean-only therefore maximizes an estimated local return, which does not determine the final NLL of the trained nonlinear model.

We trained Full, Mean-only, and Diagonal-only with the same training setup. Full PoE reaches 3.227 NLL, compared with 3.248 for Mean-only and 3.237 for Diagonal-only. It also achieves higher accuracy. Both metrics improve in every seed, as shown in Rebuttal Tables 1–2.

Remark 4 still applies. The variance of the final portfolio can differ from the variance of the gain obtained by changing the anchor route. The CARA interpretation concerns the specified linear portfolio payoff under its assumptions. It does not make Eq. 14 an exact certainty equivalent of nonlinear loss reduction. We will state this distinction clearly.

W5: The trust region leaves little room for the solver. With B = 1, k = 8 and tau = sqrt(2)/k (about 0.177), a uniform anchor allows roughly one equal-weight swap. Much of the difference from Top-k may come from the different anchor router (softplus plus distillation) or from shadow-expert gradients rather than from portfolio selection. No ablation separates these factors.

Response: We trained Full, Mean-only, Diagonal-only, Permuted, and Constant-correlation with the same anchor architecture, shadow protocol, auxiliary-training procedures, and routing constraints. Full PoE improves accuracy and NLL over every control in every seed. Its additional gains therefore cannot be explained solely by Softplus routing, shadows, or distillation.

The trust radius 2/k\sqrt{2}/k permits one equal-weight expert replacement. We use it to keep each routing change close to the anchor where CGR is measured. The anchor changes during training. The results above show that portfolio selection adds value within this local search region.

We also trained Anchor-only without the critic, solver, or distillation. This is different from using a trained PoE anchor alone at inference. Shadows have zero allocation and provide detached critic targets, so they do not directly train inactive expert parameters through the task loss.

AuthorsOpenReview ↗

Response to Reviewer YmHr (3/9)

W6: The risk coefficient is not reported. eta_route and eta_eval, the central hyperparameters of the method, do not appear in Table 4 or the text.

Response: We used eta_route=1.0 for Full PoE and the variance-based controls after normalizing CGRs by their per-layer running RMS. Mean-only uses eta_route=0.

The typical recorded RMS is approximately 9.1e-4 nats. At that scale, the normalized coefficient corresponds to approximately 1.1e3 nats⁻¹ in raw-return units. This conversion changes with the running normalization scale.

We used eta_eval=1.0e3 nats⁻¹ in the original frozen-route evaluation. We selected the routing setting on eight held-out validation batches without using downstream test-task scores.

W7: Table 1 is a single run with no variance, although its gains are the paper's central claim.

Response: We ran three independent training seeds at N=16,k=8N=16, k=8 for all nine methods in Rebuttal Table 1. The original Table 1 used one seed per configuration. Within each seed, methods share initialization and data order.

For each run, we first average accuracy over the seven tasks. We then calculate the mean and sample SD across the three training seeds. We report NLL in the same way.

Full PoE achieves 50.105 ± 0.100% accuracy and 3.227 ± 0.0030 NLL. Top-k achieves 48.124 ± 0.166% and 3.260 ± 0.0046. MP-MoE achieves 49.076 ± 0.197% and 3.234 ± 0.0036. Full PoE also improves accuracy and NLL over each covariance control in every seed. We will distinguish these repeated runs from the larger-pool configurations that currently have only the original single-seed result.

W8: The scores are unusually high. The model has about 0.54B active parameters, a GPT-2-small-sized backbone (12 layers, d = 768) and 100B training tokens, yet PoE reaches 76.3% on ARC-Easy and 59.4% on HellaSwag at N = 32.

Response: The 12-layer, width-768 description covers the shared backbone. The full MoE also includes the routed expert pool and shared experts. At N=16N=16, it has approximately 1.02B parameters, including embeddings, shared experts, and the router. This count excludes the critic. The approximately 0.54B active count excludes embeddings. Increasing NN increases total expert capacity. We will state these counting rules and report the critic's parameter count separately.

We evaluated all methods with zero-shot lm-evaluation-harness v0.4.9. ARC-Easy/Challenge, HellaSwag, and PIQA use acc_norm. BoolQ, RACE, and LAMBADA use acc. We used GPT-2 BPE, approximately 99.1B training tokens, and the final checkpoint for every method. We did not select checkpoints using downstream task scores.

The three-seed runs reproduce PoE's advantage under the same training and evaluation setup.

W9: Accuracy gains do not track NLL. Routing alone improves HellaSwag by 16% relative to Top-k. At N = 32, PoE's NLL is only 0.031 nats lower than MP-MoE's, yet LAMBADA is 6.4 points higher and HellaSwag 5.8 points higher, although both benchmarks normally track language-modelling loss closely.

Response: We measured held-out NLL on the training-mixture distribution. LAMBADA measures exact final-word prediction, while HellaSwag ranks answer options by normalized scores. A change in these task accuracies does not have to be proportional to the change in average NLL on another distribution.

We also measured LAMBADA final-word token NLL at N=16N=16. It falls from 2.49 for MP-MoE to 2.26 for PoE. This task-specific improvement is separate from the global held-out NLL. For multiple-choice tasks, a small score change can change the selected answer on examples near a decision boundary.

The added diagnostics are at N=16N=16. Checking the magnitude of the N=32N=32 gains requires the corresponding checkpoints and task-level scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.