acceptodds
Under review as a conference paper at ICLR 2027

What Must Expert Formation Preserve? Formation–Deployment Compatibility in Channel-Composed Two-Expert Models

Abstract

Mixture-of-experts (MoE) systems often couple two distinct choices: how experts are formed during training and how the resulting experts are combined at inference. We separate these factors in a controlled channel-composed expert model and study how training-time formation affects effective post-training deployment. Matched formation interventions vary assignment information, exact mixing values, low-order statistics, online adaptation, reference dependence, and stochasticity; the resulting frozen checkpoints are then evaluated under multiple deployment rules. Across ESC-50, Speech Commands V2, FSD50K, CIFAR-100, and Tiny ImageNet, no single formation–deployment pair dominates. To test whether learned coefficient-to-channel correspondence matters beyond the coefficient values themselves, we perform a post-hoc exploratory assignment intervention that preserves the exact learned coefficient multiset while permuting only its correspondence with representation channels. The aligned assignment outperforms its permutation mean on ESC-50, Speech Commands, CIFAR-100, and Tiny ImageNet, but not on FSD50K. On ESC-50, the aligned-minus-permuted effect is Robust-B points with a descriptive fold-level 95% interval ; the corresponding learned-channel versus learned-scalar difference is points . Thus channel-assignment sensitivity is itself regime dependent. For the studied linear head, every fixed channel-wise composition compiles exactly into a single affine classifier. A separate reference-free stress test shows that increasing expert count can help or hurt, with no monotonic trend across task, formation, and deployment. Together, these results identify formation–deployment compatibility as a distinct design problem.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Mixture of Experts, expert formation, expert routing, expert deployment, channel-wise composition, static deployment, expert multiplicity, model compilation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_4YJGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper separates training-time expert formation from post-training deployment in a channel-composed linear model. Experiments across five datasets examine how formation rules, deployment strategies, and expert count interact.

Critical strengths

Comparing deployment strategies on the same frozen checkpoint is a useful experimental design that helps separate training effects from post-training composition.

Critical weaknesses

The FixedHalf results need a stronger control. Both experts receive identical task gradients, so their difference is not learned through differentiated task updates. Its gains could therefore reflect initialization and subsequent coefficient fitting rather than task-driven expert differentiation.

Moreover, Appendix E.2 reports that linear probes outperform channel composition by 1.440 points on CIFAR-100 and 0.846 points on Tiny ImageNet, but their weights were not retained. Since both methods deploy a single affine classifier, this comparison directly affects the practical motivation.

Next, the permutation audit has limited explanatory reach. It shows that fitted coefficients depend on their channel correspondence, but testing only MomentBeta does not establish how this sensitivity relates to the formation rule.

Finally, the broader takeaway remains largely descriptive. The study establishes that rankings change across settings, but offers limited guidance on when a deployment strategy should succeed. All results also come from one shallow CNN family with linear expert heads.

Questions for authors

Can you compare FixedHalf with a Dense model initialized at the same effective matrix, ((W(1)+W(2))/2)((W^{(1)}+W^{(2)})/2), under matched effective optimization? How much of the reported improvement survives this control?

Could you rerun and retain the unrestricted probes under matched fitting and selection protocols? Given the reported vision results, when is channel composition preferable to simply refitting the classifier?

Does the aligned-versus-permuted advantage also appear under FixedHalf and Uniform formation? This would help distinguish a formation-dependent effect from a general consequence of fitting channel-specific coefficients.

Beyond the reported functional-contrast statistic, is there a checkpoint property that can explain or predict whether static deployment helps or hurts?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 4YJG

We sincerely thank the reviewer for the specific concerns about FixedHalf optimization, unrestricted classifier fitting, channel assignment, and predictive usefulness. We conducted additional controls in revised Section 5.5 and Appendix I while retaining the original five-dataset results.

1. FixedHalf versus matched Dense (Weakness 1; Q1; Appendix B.1, Appendix I.3, Table 31). We agree that FixedHalf produces identical expert task gradients and cannot establish task-driven expert differentiation. We therefore trained a Dense control matched at the initial effective classifier, with half-scaled classifier gradients and replicated gradient clipping. Across five CIFAR-100 seeds, FixedHalf learned-static minus matched Dense was +0.712 percentage points (95% CI [0.283, 1.141]), whereas neutral-half minus Dense was +0.224 ([−0.316, 0.764]). Crucially, static-minus-neutral within the same FixedHalf checkpoints was +0.488 ([0.273, 0.703]). A separate tied-initialization cohort achieved 51.108% mean accuracy. These results support a contribution from post-training composition, not task-induced specialization or identical optimization trajectories.

2. Retained unrestricted probes (Weakness 2; Q2; Section 5.5; Appendix I.4, Table 32). We refit and retained unrestricted affine probes on frozen MomentBeta representations, selecting the fitted probes using validation data. Relative to recomputed channel-static deployment, probe advantages were +1.440 points on CIFAR-100 (CI [0.967, 1.913]) and +0.992 on Tiny ImageNet ([0.601, 1.383]), both 5/5 positive seeds. Tiny ImageNet uses an independent cohort; its static replay discrepancy of −0.13 to −0.32 points is explicitly reported. Channel composition fits 128 coefficients, versus 12,900/25,800 for the unrestricted probes, although both compile into one affine classifier. We therefore do not claim superior accuracy or fewer deployed parameters for channel composition. Its distinction is constrained fitting while preserving frozen expert weights.

3. Assignment sensitivity across formations (Weakness 3; Q3; Section 5.3, Table 3; Appendix I.2, Table 30). We extended the exact-multiset, post-hoc assignment intervention to Uniform and FixedHalf, without refitting checkpoints. On CIFAR-100, aligned-minus-permuted gains were +0.412/+0.529/+0.908 points for MomentBeta/Uniform/FixedHalf, respectively (5/5 positive seeds each). On FSD50K, all corresponding confidence intervals included zero. Thus, correspondence sensitivity also arises under reference-free formation; it is not evidence that reference-dependent formation information is uniquely necessary.

4. Predictive checkpoint properties (Weakness 4; Q4; Appendix I.4–I.5, Table 33). We investigated a validation-gradient diagnostic for channel versus mean-matched constant-gate deployment. Its directional score agreed with positive held-out advantages in 10/10 evaluated CIFAR-100/Tiny ImageNet checkpoints, with mean advantages +0.302/+0.274 points. However, all evaluated advantages were positive, so this does not validate discrimination of unfavorable deployments or prediction of optimal formation–deployment pairs. Additional 40-run nonlinear/expert-count controls reveal formation-ranking reversals, but do not establish sparse-Transformer generalization.

We sincerely appreciate the reviewer’s constructive questions, which motivated stronger tests of optimization, deployment expressivity, and formation-dependent correspondence.

Reviewer_BdqEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how the way experts are trained affects how they can be combined at inference. The authors consider a controlled architecture with two linear experts and evaluate different formation and deployment strategies across five datasets. They find that no combination consistently performs best and that the assignment of learned mixing coefficients to channels can affect performance.

Critical strengths
  1. I think the motivation is clear, and separating expert formation from deployment is an interesting question.

  2. I appreciate that the authors compare different deployment strategies using the same frozen checkpoints. This allows them to study the effect of deployment without changing the underlying trained model.

  3. The experimental evaluation is quite extensive for the considered setting. In particular, I appreciate the different formation controls and the additional experiments investigating channel assignments and the number of experts.

Critical weaknesses
  1. I am not fully convinced by the considered architecture. The experts are simply two linear heads sharing the same backbone, and any fixed composition is exactly equivalent to a single linear classifier on the learned representation. While I understand that this is a deliberate simplification, I wonder whether the paper is mainly studying different forms of training-time regularization rather than expert formation in the usual MoE sense. I am therefore not sure how much these conclusions extend to more conventional MoE architectures.

  2. I struggle to identify the main scientific insight of the paper. The principal conclusion seems to be that different formation and deployment strategies work better on different tasks. I am not sure what we actually learn beyond the fact that performance depends on the setting.

  3. I am not fully convinced that the experiments identify which information matters during expert formation. For example, GlobalPool and MomentBeta differ not only in the information they preserve but also in the dependence structure of their sampled coefficients. The authors acknowledge this, but it remains difficult to determine which properties are responsible for the observed differences.

  4. I find the channel-assignment experiment interesting, but I am not fully convinced by its significance for the main question. The authors show that permuting coefficients specifically learned for individual channels reduces performance in several datasets. This seems somewhat expected, and I am not sure how it tells us which information needs to be preserved during expert formation, rather than showing that the learned deployment coefficients are channel-specific.

Questions for authors
  1. Could the authors clarify what distinguishes the proposed expert-formation mechanisms from training a single linear head with different forms of regularization? This seems particularly relevant for FixedHalf formation.

  2. Would the channel-assignment experiment give similar results for other formation methods, such as FixedHalf or Uniform? This could help clarify whether the observed effect depends on how the experts were formed.

  3. Is there a way to identify which formation–deployment combinations should work well without evaluating all of them? I think this would make the conclusions more useful.

I find the motivation interesting and the experimental evaluation reasonably careful. However, I am not fully convinced by the significance of the contribution. The architecture is quite restricted, and the main conclusions remain largely descriptive. In particular, I struggle to see what general understanding of expert formation we obtain beyond the observation that different training and deployment strategies behave differently across tasks.

2 comments
AuthorsOpenReview ↗

Response to Reviewer BdqE

We sincerely thank the reviewer for the substantive concerns about architectural scope, mechanistic interpretation, and predictive usefulness. These motivated additional controls in revised Section 5.5 and Appendix I, while preserving the original five-dataset evaluation.

1. Linear-head equivalence and regularization (Weakness 1; Q1; Section 3.3, Appendix B.1/B.8, Appendix I.3, Table 31). We agree that fixed linear composition is exactly affinely compilable and does not establish conventional sparse-MoE behavior. More specifically, FixedHalf has the Dense-equivalent training predictor �0��0�, with identical expert task gradients. We therefore compared it with a Dense control matched at the initial effective classifier, including gradient scaling and clipping. Across five CIFAR-100 seeds, FixedHalf learned-static minus matched Dense was +0.712 pp (95% CI [0.283, 1.141]), whereas neutral-half minus Dense was +0.224 pp ([−0.316, 0.764]). Within FixedHalf checkpoints, learned-static minus neutral was +0.488 pp ([0.273, 0.703]). A separate tied-initialization cohort examined expert symmetry. These results distinguish post-training composition from training parameterization, not task-driven expert specialization or identical optimization trajectories.

2. Formation information and scientific insight (Weaknesses 2–3; Section 5.5, Table 5; Appendix I.1/I.4/I.5, Tables 29, 32–33). We agree that GlobalPool versus MomentBeta alone confounds the gate-value marginal with sampling dependence. We introduced EmpiricalPoolIID, drawing independently with replacement from the current reference pool. Conditional on that pool, it preserves GlobalPool's exact coordinate marginal while removing without-replacement dependence. Across matched cohorts, PoolIID minus GlobalPool on FSD50K was −0.154 mAP points (CI [−0.647, 0.339]), whereas MomentBeta minus PoolIID was +0.612 ([0.229, 0.995]). Thus, sampling-dependence changes alone do not explain MomentBeta's FSD50K advantage. Separately, retained unrestricted affine probes exceeded channel composition by +1.440/+0.992 pp on CIFAR-100/Tiny ImageNet. Nonlinear experts also reversed the MomentBeta–GlobalPool ranking between two and four experts. These controls identify specific sampling, expressivity, and expert-count dependencies rather than merely reporting different winners.

3. Channel-assignment interpretation (Weakness 4; Q2; Section 5.3, Table 3; Appendix I.2, Table 30). We agree that permuting fitted coefficients measures deployment correspondence, not intrinsically necessary formation information. We extended the frozen-coefficient intervention to reference-free formations. On CIFAR-100, aligned-minus-permuted gains were +0.412/+0.529/+0.908 pp for MomentBeta/Uniform/FixedHalf, respectively (all 5/5 seeds positive). FSD50K intervals included zero. Therefore, correspondence sensitivity is neither unique to MomentBeta nor universally present, and does not establish semantic specialization.

4. Predicting favorable combinations (Q3; Appendix I.4). We evaluated a validation-gradient diagnostic for channel versus mean-matched constant-gate deployment. Its directional score agreed with positive held-out advantages in 10/10 CIFAR-100/Tiny ImageNet checkpoints, with mean gains +0.302/+0.274 pp. However, all evaluated advantages were positive; the diagnostic has not demonstrated discrimination of unfavorable cases or selection of optimal formation–deployment pairs on unseen tasks. We report this evidence and limitation explicitly.

We sincerely appreciate the reviewer’s constructive questions and the opportunity to clarify the mechanistic insights, experimental evidence, and scope of our contribution.

I thank the authors for their reply. As far as I see it, there is some contradiction between the goal of the paper and what is inside/studied. The paper is mostly about experimental study using very simple models that would have been fine if studied with a theoretical perspective. In this case, I do not think we can extract meaningful things that could generalize/extend to more practical setting.

I therefore, keep my score.

Reviewer_FVBbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines the compatibility between expert formation during training and expert deployment at inference in Mixture-of-Experts (MoE) models. The authors introduce a simple two-expert framework with linear experts to study these two processes separately. For expert formation, they consider several mechanisms that preserve different levels of information from a reference field. For expert deployment, they consider native and several static expert composition strategies. Experiments on 5 small/medium-scale audio and vision datasets are conducted to evaluate the effectiveness of these mechanisms.

Critical strengths

The paper addresses an interesting research question concerning the relationship between expert formation during training and expert deployment at inference in Mixture-of-Experts (MoE) models.

Critical weaknesses

The considered two-expert framework is confusing and unclear. The input routed to each expert is modified by a gating mechanism, whereas in conventional MoE models, the input is routed directly to the experts. The experts considered are also linear, which makes the framework simple and does not demonstrate whether the findings hold for nonlinear experts in current large-scale language models. The experiments are conducted on small- or medium-scale datasets using a simple architecture for training. It remains unclear whether the findings generalize to larger datasets and large-scale MoE architectures (with more than 100 million parameters). The analysis does not sufficiently investigate whether the different formation mechanisms lead to meaningful expert specialization.

Questions for authors
  1. Can the authors relate the considered two-expert framework to realistic MoE models, such as Switch Transformer [1], DeepSeekMoE [2], or Mixtral [3]?

  2. Can the authors provide empirical evidence from training realistic, large-scale Transformer-based MoE models (with more than 100 million parameters) on real-world language or vision tasks?

  3. Does the paper provide any insights or potential solutions to common challenges in MoE models, such as expert specialization and expert collapse?

  4. What suggestions do the authors have for improving training and inference in large-scale MoE models?

Reference

[1] W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.

[2] D. Dai, C. Deng, C. Zhao, et al. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 1280–1297, 2024.

[3] A. Q. Jiang et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.

2 comments
AuthorsOpenReview ↗

Response to Reviewer FVBb

We sincerely thank the reviewer for raising important questions about architectural relevance, scalability, and expert specialization. We agree these are essential for understanding the scope of our findings. The revision adds nonlinear-expert controls and frozen-checkpoint diagnostics while clarifying what our experiments establish.

1. Relation to conventional MoEs (Weakness; Q1; Section 2, Section 3.1, Figure 1). Our model deliberately isolates a different mechanism from the sparse token dispatch used in Switch Transformer, DeepSeekMoE, and Mixtral. The gate scales representation channels before the expert matrices; for linear experts, this is algebraically equivalent to mixing the corresponding parameter columns (Eqs. 1–2). The construction is therefore closer to conditional parameter mixtures and channel gating, with precedents in CondConv and DeepMoE (Section 2). Its purpose is not to reproduce sparse routing, but to independently intervene on training-time formation and frozen-checkpoint deployment. Importantly, every fixed composition compiles exactly into one affine classifier (Appendix B.8), making deployment compatibility directly measurable without an inference router.

2. Nonlinear experts and scalability (Weakness; Q2; Section 3.4, Table 4; Appendix I.5, Table 33). To test whether the findings are confined to two linear experts, we completed 40 CIFAR-100 runs crossing GlobalPool/MomentBeta formation, two/four experts, linear/nonlinear heads, and five seeds under a common backbone. For nonlinear heads, the MomentBeta-minus-GlobalPool learned-static difference reverses from +0.492 points with two experts to −1.474 with four. Learned-static also improves over neutral deployment in all eight tested configurations. Thus, the formation–deployment interaction persists with nonlinear heads, but its direction changes with expert count. Parameter counts are reported explicitly (324,452–533,348). These input-gated heads are not sparse Transformer MoEs. We did not evaluate Transformer MoEs exceeding 100 million parameters, and therefore cannot establish generalization to that setting.

3. Expert specialization and collapse (Weakness; Q3; Appendix B.1; Appendix I.2–I.4, Tables 30–31). We distinguish measurable expert contrast from semantic specialization. Under FixedHalf, both experts receive identical task gradients, ruling out task-gradient-driven differentiation under that formation. Frozen weight-contrast and logit-disagreement diagnostics characterize differences between experts but cannot establish semantic specialization or prevent collapse. Separately, aligned coefficient assignments outperform permutations under MomentBeta, Uniform, and FixedHalf on CIFAR-100 (+0.412/+0.529/+0.908 points), while FSD50K shows no clear advantage (Table 30). This demonstrates deployment correspondence sensitivity without requiring reference-dependent formation; it is not presented as a solution to sparse-MoE collapse.

4. Implications for large-scale training and inference (Q4; Sections 5.2–5.5, Tables 2–3, 5). A transferable experimental principle is to evaluate formation and deployment separately rather than assuming the training router must be retained. For example, learned-static versus native deployment improves Speech Commands by +0.660 points but reduces FSD50K mAP by 6.916 points for LearnedMoE (Table 2). Our sampling and assignment controls further show how to test which properties matter before changing inference rules. For large sparse MoEs, we propose this as a controlled evaluation strategy—not a validated training prescription—requiring additional measurements of routing load, specialization, accuracy, and computational cost.

We sincerely appreciate the reviewer’s critical questions, which helped us sharpen the architectural distinction, provide nonlinear evidence, and state the boundaries of applicability precisely.

I would like to thank the authors for their efforts and the additional clarifications.

However, I will maintain my score, as I still believe that the scope of the work is very narrow. The empirical evaluation is conducted using a simple architecture that is not commonly used in either medium- or large-scale applications. Furthermore, the datasets used are relatively small.

Therefore, I do not believe that the work makes a sufficiently significant contribution to the scientific community to warrant publication at this venue.

Reviewer_frpaOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the way experts are combined during training affects their deployment after training. Using a shared backbone and a two-expert linear classification head, the authors compare several training-time gating rules and evaluate different deployment rules after freezing the model. Results on five datasets support the view that formation and deployment should be considered jointly. They also show that the learned correspondence between mixing coefficients and feature channels affects performance on some datasets.

Critical strengths

Comparing deployment rules on the same frozen model helps separate the effects of training and deployment. The channel-permutation experiment keeps the learned mixing coefficients fixed and changes only their assignment to channels. The authors also report that this intervention shows no clear benefit on FSD50K.

Critical weaknesses
  1. Some comparisons do not identify the cause of the performance difference. For example, GlobalPool and MomentBeta differ both in the gate-value information they retain and in their sampling procedures. Their performance difference therefore cannot be attributed to either factor alone. The authors acknowledge this limitation, but an intermediate control is still needed.
  2. The core experiments cover a limited range of expert models. The full formation–deployment comparison uses a two-expert linear classification head. The four- and eight-expert experiments use two different, reference-free formation rules and do not repeat the core information-retention comparison. Nor is that comparison repeated with nonlinear experts. The current evidence therefore does not show whether the central findings extend to broader MoE designs.
Questions for authors

Weakness 1: Could the authors add an intermediate control that samples each gate value independently with replacement from the same reference-value pool used by GlobalPool? Comparing this control with GlobalPool and MomentBeta under the same datasets, random seeds, and deployment rules would help separate the effects of the sampling procedure and the gate-value distribution. Weakness 2: Could the authors run a 2-versus-4 experts × linear-versus-nonlinear experts comparison with the same dataset and backbone, repeating at least one core information-retention comparison and its corresponding deployment comparison? Please also report the parameter count of each model.

2 comments
AuthorsOpenReview ↗

Response to Reviewer frpa

We sincerely thank the reviewer for identifying two important limitations and proposing specific experiments to address them. We completed both requested controls, reported in the revised Section 5.5 and Appendix I, Tables 29 and 33. The original five-dataset evaluation remains unchanged.

1. Separating sampling dependence from retained gate-value information (Weakness 1; Q1; Section 5.5, Table 5; Appendix B.5 and I.1, Table 29). We agree that the original GlobalPool–MomentBeta comparison changes both the sampled marginal and dependence structure, preventing attribution to either factor alone. Following the reviewer's suggestion, we introduced EmpiricalPoolIID, which independently samples each gate entry with replacement from the same current minibatch reference-value pool used by GlobalPool. Conditional on that pool, both methods have exactly the same coordinate marginal, mean, and variance; GlobalPool instead samples without replacement, preserving the finite multiset and inducing covariance �0��0� between distinct entries. MomentBeta independently samples from a fitted Beta distribution. Thus, GlobalPool versus PoolIID isolates sampling dependence, while PoolIID versus MomentBeta compares different conditional marginals under independent sampling.

We evaluated all three formations across five datasets, using five matched scientific seeds per dataset and all five official ESC-50 folds, with common training and static-deployment protocols within each new cohort. On FSD50K, PoolIID minus GlobalPool was −0.154 mAP points (95% CI [−0.647, 0.339]), whereas MomentBeta exceeded PoolIID by +0.612 ([0.229, 0.995]). Speech Commands showed a different ordering: PoolIID 89.441%, GlobalPool 89.132%, and MomentBeta 88.827%. These controls resolve the specific experimental confound while showing that neither dependence nor marginal choice provides a universal performance explanation. The new cohorts are analyzed separately from the principal experiments.

2. Extending the core comparison across expert count and nonlinearity (Weakness 2; Q2; Section 3.4, Appendix I.5, Table 33). We completed the requested 2 × 2 × 2 × 5 factorial experiment on CIFAR-100: GlobalPool versus MomentBeta, two versus four experts, linear versus nonlinear heads, and five seeds (40 runs). All configurations share the same three-block CNN backbone. Total trainable parameter counts are 324,452/350,052 for two/four linear experts and 416,100/533,348 for two/four nonlinear experts, under either formation.

For nonlinear experts, the MomentBeta-minus-GlobalPool learned-static accuracy difference reverses from +0.492 points at two experts to −1.474 at four. Learned-static also exceeds neutral deployment in all eight formation–count–head configurations, with gains ranging from +0.416 to +0.730 points. These results establish that formation–deployment interactions persist beyond the original two-linear-expert construction, but their direction depends on expert count and head type. The nonlinear heads use input-gated channel-weighted features rather than sparse token routing; parameter counts also increase with expert number. We therefore do not claim generalization to large-scale sparse Transformer MoEs.

We sincerely appreciate the reviewer’s precise experimental suggestions, which enabled us to strengthen the causal interpretation and clarify the scope of our findings.

Thank you for the additional experiments. The CIFAR-100 results show that the ranking of training rules can change between two and four nonlinear experts. I am keeping my rating at 2 because it remains unclear how this finding would help choose a deployment rule for nonlinear experts in other settings.

Reviewer_s6KPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A major proposal in this paper is separating the effect between 'expert formation' and ‘expert deployment’. Control experiments and interventions are done separately on formation and deployment levels. On the formation level, this is done via changes on mixing values, statistics, adaptation, channel assignments randomized and etc. Further, during the inference time process, the model is varied on frozen checkpoints using varying deployment rules, such as, learned static. The control utilized 2 linear expert matrices used with a shared representation. In summary, this research investigates the manner in which the experts are trained affecting the combination of experts after training.

The results are comprehensively presented and include detailed control experiments on each of the sub-factors mentioned above. A major result obtained is that there is no single strategy utilized which performs the best across all settings.

Critical strengths

In my view, a strength of this submission is that this is a useful research direction towards the understanding of the relation between expert training and inference. MoEs are one of the most important models for scalability and inference time efficiency. And, reading this paper, I am convinced that noting the differences of effects between varying strategies in expert formation and deployment are important. A result that authors found (in section 5.1) is that there is no single strategy that performed the best across all datasets. If this is well understood in future, it could help researchers better understand and select the right MoE training + inference-time strategies for sub-fields (audio, and image classification, etc.) for a better performance.

Further, the experimentation w.r.t. breadth is noteworthy. Several standard datasets were covered, such as, ESC-50, FSD50K, CIFAR, Speech Commands, and, Tiny ImageNet-200. Further, tables 6,7,8,9, and 10 showcases evaluation results for each of the 5 datasets.

Additionally, we see the use of confidence intervals and multi seed experiments to demonstrate uncertainties of test accuracies for the presented strategies. The control experiment of permuting the assignments of fixed parameters (on whether it affects performance) is also important and interesting. These include both adaptive or data dependent strategies and data independent strategies (as per table 5).

Critical weaknesses

(1) In my view, the submission is difficult to read. Vocabulary involves the use of several non-standard terms. Further, these terms were used without explicit definitions in order, which made it harder for me to completely comprehend the ideas and the results. For instance, it wasn't immediately clear what robust B-points meant in the abstract. In the introduction, it wasn't immediately clear what "one fixed bounded two-view reference field" implied.

(2) I am not convinced that the controlled interventions experiments can completely explain why these differences in performances happen, using the tested strategies. For example, section 5.4 explains the numbers from the Dirichlet formulation and Fixed Equal formulations from table 4, however, doesn't postulate an explanation. Further, a natural extension or prediction on whether these effects would be expected in other settings and datasets, is also important.

(3) While the experiments and the pipeline for measuring the effects are extensive, I am not entirely convinced here that these results provide sufficiently new understanding of the sub-field. It is also not entirely clear on the usage of these ideas and results in MoE applications.

(4) Scalability and Generalizability is also a concern. All experiments focus on a small MoE model with linear expert heads and consisting of a representation width of 128. It is unclear if these results and outcomes remain or can generalized when scaling to larger MoE architectures.

(5) Lemmas and theorems trivially follow. For instance, theorem 1 (Exact static compilation) is just a direct substitution of the definition. Additionally, lemma 1 is a substitution of one equation.

Questions for authors

(1) Can authors conduct more experiments on scale to demonstrate if their findings remain consistent?

(2) Authors have identified certain patterns based on datasets in their results (for instance, the claim of regime dependent pattern for assignment intervention in table 3). Do some of these patterns have strong usability for current MoE research? For instance, one broad example can be on expectation (or average), the settings (X) is useful for the audio classification tasks, and settings (Y) is more useful for the image classification tasks. And, is there significance to the outcomes predicted? In short, is it possible to make any predictions of the optimal settings for newer datasets, given that all necessary characteristics of the datasets are known?

(3) Based on their results, can authors clearly explain the underlying reasons why certain improvements and reductions appear in their results based on different interventions conducted?

2 comments
AuthorsOpenReview ↗

Response to Reviewer s6KP

We sincerely thank the reviewer for recognizing the importance of separating expert formation from deployment and for identifying gaps in clarity, mechanistic interpretation, and generalizability. We address these concerns through the revised manuscript and additional controls.

1. Definitions and presentation (Weakness 1; Sections 3.2, 4.1; Table 1; Appendix A). Robust-B denotes ESC-50 accuracy averaged over 20 predefined corruption conditions (five families × four severities), not a separate statistical measure. The bounded two-view reference field RR is a detached, channel-wise statistic constructed from paired augmentations using feature-map directional agreement and RMS-based stability (Appendix A, Eqs. 5–12). It is a fixed experimental reference, not a semantic label or learned router. Figure 1 and Table 6 clarify the formation controls and retained information.

2. Mechanistic interpretation (Weakness 2; Q3; Sections 5.4–5.5; Tables 4–5; Appendix I, Tables 29–31). We agree that expert-count rankings alone do not establish their causes. On Tiny ImageNet, the N=8N=8 minus N=2N=2 channel-deployment effect reverses from −0.740 points under Dirichlet formation to +0.712 under FixedEqual (Appendix H, Table 28). On FSD50K, effective expert count rises from 1.85 to 6.04 while mAP falls from 34.412 to 33.739. Additional controls separate specific effects: EmpiricalPoolIID preserves GlobalPool's conditional coordinate marginal while removing without-replacement dependence; MomentBeta exceeds PoolIID by +0.612 FSD50K mAP points (95% CI [0.229, 0.995]). FixedHalf has identical expert task gradients, yet learned-static exceeds neutral deployment by +0.488 CIFAR-100 points (CI [0.273, 0.703]). These experiments isolate meaningful contrasts without fully explaining every performance reversal.

3. Novelty, practical relevance, and prediction (Weakness 3; Q2; Sections 3.3, 5.2–5.3; Tables 2–3; Appendix I.4). Our contribution is the controlled formation–deployment factorization, not a new MoE architecture or universal audio-versus-vision recommendation. For the same LearnedMoE formation, static-minus-native deployment is +0.660 points on Speech Commands but −6.916 mAP points on FSD50K. Thus even audio tasks do not support one deployment rule. Fixed composition also permits exact single-classifier inference without an input-dependent router. We investigated a frozen validation-gradient diagnostic, but its favorable directional results on evaluated vision checkpoints do not validate optimal-setting prediction for unseen datasets.

4. Scalability and generalizability (Weakness 4; Q1; Section 3.4, Table 4; Appendices H and I.5, Table 33). We evaluated reference-free N∈2,4,8N \in \\{2,4,8\\} experts across five datasets and completed 40 additional CIFAR-100 runs varying formation, expert count, and linear/nonlinear input-gated heads. For nonlinear experts, MomentBeta-minus-GlobalPool reverses from +0.492 points at two experts to −1.474 at four. This extends the observed interactions beyond linear heads, but training parameter counts differ, and sparse Transformer-scale generalization remains untested.

5. Mathematical novelty (Weakness 5; Appendix B.2/B.8). We agree that the Beta-moment lemma and exact static-compilation theorem follow by algebraic substitution. Their purpose is to specify exact distributional and inference-equivalence properties, not to claim novel mathematical results.

We sincerely appreciate the reviewer’s constructive questions and the opportunity to clarify the contributions, experimental evidence, and limitations.

Thank you for your response.

I would like to start from the positives. The authors have made edits to clarify points according to the weakness's points. For instance, you have clarified the terminology about Robust-B and have made revisions to explain unclear points. The experimental results added are useful. For instance, the EmpiricalPoolIID experiments and the 5 seed FixedHalf experiments add further context for the main story.

However, my major concern of this approach being some impact to training and applications of current large-scale MoE models or new understanding towards a sub-field remains. I believe that to be the most important weakness. Even for the thorough experimental results that the paper presents, a strong understanding of why these effects happen or predictability for observing these effects independently for large-scale models is salient. And further, can these effects lead to better directions to improve strategies or selections for MoE training and inference in new datasets (which are non-standard). Secondly, scalability is an essential factor for MoE applications. Hence, more experiments should be demonstrated on whether these effects can be identified and can continue on scale to large MoE models. Yes, there is an attempt made with 40 additional CIFAR-100 runs, however, your response also specifies the transformer level generalization is 'untested'.

Thanks,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.