acceptodds
Under review as a conference paper at ICLR 2027

Auditing Expert Architecture Regret in Mixture-of-Experts Language Models

Abstract

Sparse Mixture-of-Experts models fix their expert architecture before training, while adaptive methods reorganize it during training when an observational statistic fires. Both rest on one unmeasured premise, that the changes such statistics suggest actually improve the model. We measure it. Expert Architecture Regret is the largest validation-loss improvement available from an audited local operation (retire, merge, or a budget-neutral composite), estimated by paired interventional probes against a null of no-op pairs. Each probe adapts two branches identically except for the operation. Across 407 probe results spanning two fine-grained families, seven checkpoints, random and heuristic candidates and two data distributions, six candidates cross the band. None survives re-probing on all control streams, where function-preserving placebos score the same. The instrument is not blind, since an amplitude control puts a known defect above the band and the audit finds it. But its resolution turned out to be a protocol choice. The adaptation rate moves the un-mutated model ten to fifty times further than any single expert can. Lowering it tenfold cuts the band from 0.0041 to 0.0025 nats while the control still fires, and 101 further operations give a largest effect of +0.0003. A sixteen-stream re-test confirms two of them, near-dead-expert retires, as gains of 1–2×10⁻⁴ nats that the pooled rule cannot flag. These gains hold on held-out WikiText text and reverse on OLMoE's own training mixture, where the two experts are not near-dead. Destroying a layer's busiest expert costs 0.0018–0.0100, so a removal gain of that size reaches 80% power only where experts are large. We state this scale argument and give a counterexample class to it. A coarse-grained model (JetMoE-8B, 8 experts, top-2) confirms the account. In it, a layer's busiest expert is worth 11–26× its band, and eight of nine audited edits lose, six of them by 2.5–13× that band. The ninth, which only a random candidate proposed, retires a busiest expert that is harmful on WikiText. It clears the band on all six streams of the validation slice, vanishes on the held-out test split (−0.0001, 95% CI −0.0004 to +0.0002) and reverses on RefinedWeb, JetMoE's main training source (−0.0036, every stream negative). Every positive audited LM edit we find reverses on data its model was trained on. Half a null draw's variance separates repeat runs of one configuration rather than one stream from another. More streams shrink it and 16× the evaluation tokens do not, so buy control streams, not evaluation data. We release the harness, the raw per-run records, and the scripts that recompute the statistics quoted here.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, sparse MoE, expert pruning, expert merging, structural adaptation, interventional evaluation, null calibration, statistical power, language models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_LgaiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether dynamic restructuring of routed experts in small (<10B parameter) MoE architectures via pruning, merging, or a composite of these two operations can improve validation loss after a small number of recovery training steps compared to the unadulterated baseline.

Critical strengths

None

Critical weaknesses
  • The paper’s core motivation is poorly aligned with the empirical evidence presented. The authors cite works such as DynMoE, EPnG, and LAEP and claim that these works have an “implicit quality premise” of improving loss. However, in each of these three cases, the motivation was largely related to pre-training or PEFT efficiency gains, not necessarily improved validation loss. The efficiency gains of these methods are not considered in this work. Further, many of the cited methods are intended to be applied during training end-to-end training, rather than local interventions coupled with only local recovery. So the claims refuting the value of these adaptive MoE methods fail to consider their intended use-case of supporting pre-training or fine-tuning where the majority of parameters are unfrozen. The authors present expert merging and pruning as MoE adaptation techniques; however, these are first and foremost model compression techniques. Any intervention that removes parameters is likely to increase validation loss, especially with very few optimizer steps and a limited hyperparameter sweep as described here. The claims made here with respect to the utility of MoE restructuring are not well supported by the empirical results.
  • The paper’s writing is poor. Results are presented in a verbose manner with excessive enumeration of results without a coherent structure. Unspecific and colloquial language is used in many locations., eg.: “Destroying a layer’s busiest expert…”, “near-dead experts…”, “trigger fires the hardest…”, “Buy control streams, not evaluation data”, “Common random numbers cancel the data draw but not the trajectory.” Introduction of terms such as “Structural Utility” which obfuscate simple concepts rather than elucidating. Terms are swapped in various sections “prune vs. retire experts”.
Questions for authors

None

Reviewer_yRSBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper defines Expert Architecture Regret and makes edits on Experts in the MoE Model and records the loss differences.

Critical strengths

None

Critical weaknesses

The paper is extremely difficult to follow due to its dense writing, excessive numerical details, unexplained terminology, abruptly introduced concepts, and heavy use of jargon. Even the abstract and conclusion, which should clearly communicate the paper's main contributions and findings, are unnecessarily complicated and difficult to understand. The main text relies heavily on dense paragraphs rather than clear visualizations or structured presentations of the results, making it difficult to identify the key findings.

From what I could understand, the paper primarily reports negative results: a few architectural modifications initially appeared to yield improvements, but these improvements largely disappeared upon repeated evaluation. However, this central finding is buried under excessive technical detail, making it difficult to assess the paper's actual contributions and significance.

Questions for authors

Abstract:

  • What is "observational statistic" in the abstract
  • What are these "audited local operation (retire, merge, or a budget-neutral composite)"? What are they applied to? How they work?
  • "paired interventional probes against a null of no-op pairs" What are these probes? how they work? what they measure? what is no-op pairs?
  • "Each probe adapts two branches identically" What is a branch in this context?
  • " None survives re-probing on all control streams" What is re-probing? What is the conclusion of this?
  • "sixteen-stream re-test" What is a stream in this context?

Intro

  • Never abbreviated EAR (Expert Architecture Regret) and suddenly used it in line 056!
  • What is the point of introducing "structural utility SU(o)" in the intro?
  • L70: "recovers SU = +0.031 nats at gain 16": What does "gain" mean?
Reviewer_YCedOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1(−1)
Read review
Paper summary

This paper proposes Expert Architecture Regret (EAR), which uses paired interventions and an empirical null to evaluate local structural changes in MoE models. The changes include removing an expert, merging two experts, and splitting an expert under a fixed parameter budget. Experiments on OLMoE, Qwen1.5-MoE, and JetMoE find no structural change that reliably improves performance on the models’ training distributions.

Critical strengths
  1. Expert activation frequency is commonly used to judge expert importance. This paper shows the limits of this approach, and also shows that large run-to-run noise strongly affects the evaluation results.

  2. The experimental design is fairly thorough. The paper uses paired interventions, empirical null calibration, and other controls, and provides a systematic framework for evaluating structural changes in MoE models.

Critical weaknesses
  1. Gap between the evaluation setting and real adaptive methods. The evaluation mainly focuses on a single layer. After each change, the model is trained for only a short time with a newly initialized optimizer. In contrast, existing adaptive methods change the structure during continued training. It is therefore unclear whether these negative results apply to real dynamic MoE training.

  2. Limited statistical conclusions about EAR. EAR is defined as the maximum SU over a limited candidate set, but the paper gives no confidence bound on this maximum. The paper also notes that in some fine-grained models, the effect of a single expert is close to or below the detection limit. This weakens the negative results and makes it hard to rule out a few small positive gains.

  3. The organization and writing need major improvement. The paper is not well structured. Important results and analyses are spread across the main text and a long appendix, which makes the core contributions hard to follow. I suggest moving key method figures, such as Figures 3 and 4, into the main text and using clearer vector graphics.

Questions for authors
  1. Does short-term, single-layer SU predict the gain after longer, full-parameter training? Could the sign flip with longer training?
  2. Why is the Adam state re-initialized at each intervention? Would keeping the original optimizer state change SU much?
  3. Can the authors give a confidence bound on the maximum SU over the audited candidates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.