- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed PoE which borrowed financial portfolio concept to improve moe model performance and lower the joint failure. While the application of it into llm domain is not very clear working for me. I also think the paper didn't discuss the weakness of the mean variance model, there are few reasons it's not well adopted in real life finance decision, although it's a great invention.
Critical strengths
- Idea is simple and seems effective, it's very interesting to think the problem as an investment problem
- Presentation is clear and easy to understand
Critical weaknesses
We don't really know how efficient this type of methods are for modern moe models e.g. is it useful for bigger models or smaller models? Are the method transferrable? Is it more useful when doing pretraining or post training? Is there any design criteria needs to be satisfied with the critic? Also we already know that mean variance model is very sensitive and there are some criticisms called "error minimization" say if one expert give you 10.2% gain and the other give you 10%, you will be dropping the other one completely but we also know there are super experts in moe arch: https://arxiv.org/abs/2507.23279 which might give you trouble
Questions for authors
Do you have any test results for modern open source models e.g. Qwen, Kimi or Deepseek? Expert is MLP how you make it as vector which can be comparable with the gradient?
7 comments
Response to Reviewer g6a7 (2/3)
3. Is PoE more useful during pretraining or post-training?
Our experiments directly establish effectiveness during pretraining on approximately 100B Dolma tokens, where PoE improves both held-out language-modeling loss and downstream accuracy. Pretraining allows expert representations, routing, and the critic to develop together, and our results demonstrate that this joint learning is effective.
The return definition also applies naturally to gradient-based post-training, such as supervised fine-tuning: expert contributions can be measured using the fine-tuning loss, with the critic adapted to the target distribution. This provides a concrete route to task-specific routing during supervised post-training.
4. What design criteria must the critic satisfy?
The essential requirements concern the critic's information, prediction targets, covariance validity, and training coverage. The particular MLP architecture is an implementation choice.
- Routing-time information: the critic receives the detached layer representation and the allocation at which moments are queried.
- Task-conditioned moments: it predicts expected expert returns and their covariance. Supervision uses realized returns and raw second moments derived from the language-modeling loss.
- Valid covariance: a lower-triangular factor gives , ensuring positive semidefiniteness. A small ridge improves numerical conditioning.
- Off-route coverage: training-only shadow experts are evaluated at zero allocation, providing observations for unselected experts without changing the forward result. Inverse-propensity weighting corrects coordinate and expert-pair subsampling.
- Ordered updates: a lagged critic selects the current route; the resulting observations update future predictions. Detached targets avoid higher-order differentiation through the task gradient.
Our implementation uses one MLP per routed layer, with two hidden layers of width 256 and SiLU activations, totaling approximately 3.7M critic parameters at .
Connection between an MLP expert and the gradient. To address the reviewer's explicit question, the vector is the expert's output for the current token:
The task gradient is taken with respect to the layer output and lies in the same -dimensional space. Since , the chain rule gives
=-\langle e_i,g^a\rangle =-\left.\frac{\partial L^{[a]}(y(v;s);s,\xi)}{\partial v_i}\right|_{v=a}.$$ A positive return means that increasing the expert's allocation locally decreases the downstream loss. This gives the critic a task-grounded utility target. **Empirical validation.** Allocation-weighted CGR agrees strongly with exact frozen-route deletion effects: the Spearman correlations are **0.995, 0.988, and 0.951** at representative early, middle, and late layers. On held-out anchor queries, the sampled-shadow critic achieves **0.965** Spearman correlation for portfolio variance, a calibration slope of **0.974**, and **0.967** Spearman agreement with oracle pairwise dependence. The latter substantially exceeds raw output cosine similarity, which achieves **0.452**. Removing shadow coverage worsens both calibration and route quality. At inference, the frozen critic predicts the required moments from routing-time inputs. Task gradients, shadow evaluations, and online updates are unnecessary. Therefore, the critic and support solver remain part of routing.Response to Reviewer g6a7 (3/3)
5. Mean-variance sensitivity, the "10.2% versus 10%" example, and super experts
We agree that sensitivity to estimated moments is a legitimate concern. However, a small difference in individual expected returns does not determine the selected portfolio: allocation depends jointly on expected utility, covariance, and feasibility constraints.
A small mean difference does not imply dropping the other expert.
PoE optimizes
\qquad \frac{\partial J}{\partial p_i}=\mu_i-\eta(\Sigma p)_i.$$ An expert with slightly lower expected utility can remain valuable if its returns complement those of the other selected experts. Consequently, the two values "10.2%" and "10%" alone are insufficient to conclude that the second expert receives zero allocation. Accounting for this dependence is precisely the motivation for joint portfolio selection. **The method explicitly addresses estimation sensitivity.** Our feasible set bounds total allocation, individual weights, and active support size. A trust region limits changes from the anchor route, and a covariance ridge improves numerical conditioning. Appendix A.6 also quantifies the consequences of imperfect predictions. Proposition 7 bounds the uniform portfolio-score error by $$\varepsilon_J =B\varepsilon_\mu+ \frac{\eta}{2}B^2 (\varepsilon_{\Sigma,\mathrm{stat}}+\epsilon),$$ and the resulting routing regret by $$J(p^\star)-J(\widehat p) \leq 2\varepsilon_J+\varepsilon_{\mathrm{opt}}.$$ These terms account for mean-estimation error, covariance-estimation error, the ridge, and solver error. The bound concerns regret in the portfolio objective; discrete supports can still change under perturbations. We validate the practical value of covariance through controlled ablations. In Table 10, full covariance improves the **exact-loss certainty-equivalent metric by approximately 71% over diagonal covariance**. The paired improvement is $4.968\times10^{-3}$ nats, with a 95% confidence interval of $[4.042,5.894]\times10^{-3}$, and is positive in **all five seeds**. Permuting expert-pair correlations while preserving marginal information also substantially reduces route quality. These results show that correctly identifying complementary expert pairs provides useful routing information despite estimation error. The mean-variance objective requires finite second moments. Exact equivalence to the CARA certainty equivalent holds under conditional Gaussianity. Appendix A.5 characterizes the approximation remainder for more general distributions under the stated regularity conditions. We will make these assumptions and the sensitivity analysis more prominent. **Super experts are consistent with PoE's motivation.** We thank the reviewer for highlighting the Super Experts study. Its central observation that experts can differ substantially in importance is consistent with our motivation to measure task-relevant expert contributions. PoE explicitly rewards experts according to their expected contribution to reducing downstream loss. Its return $r_i^a=-\langle e_i,g^a\rangle$ measures the local loss reduction associated with increasing expert $i$'s allocation. Experts with larger expected positive contributions are therefore favored by the utility term. Covariance-aware selection additionally considers which expert combinations provide complementary contributions under the activation budget. PoE thus accommodates highly unequal expert importance: **a particularly useful expert can receive substantial routing weight alongside complementary specialists**. Its covariance criterion concerns the joint variability of task contributions and imposes no requirement that experts receive equal weights. The cited study demonstrates the consequences of skipping critical experts in existing pretrained models, whereas our experiments jointly learn expert representations and routing during pretraining. **The presence of super experts does not imply an inherent conflict with PoE. It reinforces the importance of routing based on task-relevant expert contributions.** We appreciate the reviewer's questions and will clarify these points in the revision.
Response to Reviewer g6a7 (1/3)
We thank the reviewer for recognizing the simplicity, effectiveness, and clarity of PoE. We address the five concerns below, following the order in the review.
1. Is PoE useful for bigger models or smaller models?
PoE improves performance across all three evaluated expert-pool sizes, and its advantage becomes more pronounced as expert capacity increases.
We vary the number of fixed-width routed experts over N∈16,24,32 while keeping the nominal activation budget at k=8. The models have 12 layers and approximately 1B parameters at N=16, with total capacity increasing with N. Within each configuration, methods share initialization, data order, training horizon, and evaluation.
Table 1 shows that PoE achieves the highest mean downstream accuracy and lowest held-out NLL at every evaluated expert-pool size:
The relative improvement increases from 4.0% to 6.7% and 10.2%, providing encouraging evidence that PoE benefits from additional expert capacity. At N=32, PoE achieves 59.4% mean accuracy compared with 56.0% for the group-selection baseline MP-MoE, an improvement of 3.4 percentage points.
Figure 3 further shows approximately an order-of-magnitude reduction in joint failure relative to Top-k while activating slightly fewer experts. This supports the effectiveness of selecting complementary expert combinations under a limited activation budget.
We also measure complete training cost. At fixed N=16, PoE adds approximately 7.6-12.1% training step time relative to Top-k across k∈2,4,8, including critic updates, shadow evaluations, and support optimization. Its additional step time over MP-MoE is only 1.69% and 1.76% at k=4 and 8. These measurements concern training overhead; inference retains the critic and solver while removing task gradients and shadow evaluations.
2. Is the method transferable?
PoE's formulation is applicable across MoE architectures that combine expert outputs through routing weights and provide a differentiable downstream task loss. The return definition and covariance-aware selection criterion are independent of a particular model family.
The transferable element is the routing formulation: estimate task-conditioned expert utility and dependence, then jointly select and weight experts under the activation budget. Adapting an existing checkpoint requires fitting the critic to that model's expert contributions and routing distribution; the critic is learned for the model and training setting.
Regarding the concern about Qwen, Kimi, and DeepSeek, the current manuscript does not report experiments on those pretrained checkpoints. We use controlled from-scratch training to isolate routing effects under matched conditions. The consistent gains across three expert-pool sizes support the routing principle.