acceptodds
Under review as a conference paper at ICLR 2027

What Does Expert Sparsity Buy? A Two-Sided Matched-Budget Study of Mixture-of-Experts

Abstract

Expert sparsity exchanges active computation for stored capacity, but its value depends on which resource a comparison holds fixed. We introduce MoE-Tax, a two-sided evaluation framework that compares the same sparse model with both a parameter-matched and an active-compute-matched dense reference. Controlled language-model experiments reveal a clear exchange: at approximately 96M parameters, the sparse configurations in our main sweep incur a 3.8–38.2% perplexity increase at matched parameters, whereas matching active FFN compute yields 2.2–6.1% lower perplexity at a 24.6–42.2% parameter premium. A protocol-controlled scale comparison preserves the parameter-matched gap from 96M to 203M. Expert-count, activation, routing, and balancing-loss ablations identify where the tax arises, and replicate seeds reproduce the configuration ordering. We further isolate a batch-size effect large enough to reverse a compute-matched comparison. Device profiling completes the accounting by measuring latency and peak memory separately from FLOPs. Together, the configuration generator, comparison audit, and released run records make the resource exchange explicit, reproducible, and useful for architecture selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-experts, sparse expert models, load balancing, matched-budget comparison, inference efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QXv9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the trade-off between model capacity and active computation in sparse MoE models. The authors compare each MoE configuration with two dense baselines, one matched by total parameter count and the other by active FFN compute. On WikiText-103, the MoE models generally perform worse than parameter-matched dense models in terms of perplexity, but some achieve better results when active FFN compute is matched, at the expense of additional parameters. The paper also investigates how expert count, routing, load balancing, and batch size affect these comparisons, alongside measurements of inference latency and memory usage.

Critical strengths

1.The two-sided evaluation provides a clear account of the trade-off between stored capacity and active computation. The matched comparisons and explicit accounting make the reported results easier to interpret. 2.The experimental controls are useful. The batch-size analysis demonstrates how protocol differences can reverse a comparison, and the latency measurements prevent nominal FLOP savings from being confused with actual runtime improvements.

Critical weaknesses

1.The conceptual novelty is limited. Separating parameter count from active computation is well established in the MoE literature, including Clark et al. (2022) and Artetxe et al. (2022). The proposed framework mainly organizes existing comparison principles. The paper does not sufficiently establish what new scientific understanding follows from this organization.

2.The experimental evidence is too narrow for the broader claims. Controlled quality comparisons cover only WikiText-103 at approximately 96M and 203M parameters, with a fixed training-token budget. Most comparisons rely on one seed, and the replicate study excludes the dense references. Furthermore, active-FFN matching does not establish equal end-to-end training cost. The unusual latency differences across model scales also require clarification.

Questions for authors

I have three questions listed below.

1.Beyond combining two established matching criteria, what conclusions does MoE-TAX enable that cannot be obtained from existing routed-model scaling and compute-efficiency studies?

2.Do the parameter-matched and compute-matched conclusions persist under longer training budgets and additional datasets? Can the authors provide uncertainty estimates for the dense references?

3.What explains the substantial decrease in measured MoE latency from S to M scale under the reported profiling setup? Were precision, kernel implementation, and measurement conditions identical?

3 comments
AuthorsOpenReview ↗

Response to Reviewer QXv9

Thank you for recognizing the two-sided accounting and controls. Your questions prompted further analysis of the training records and a new controlled latency benchmark.

Q1. What scientific understanding does the framework add?

Separating parameters from computation is established in prior work. We empirically characterize how quality, resource premiums and protocol choices interact for the same sparse model. Matching equations alone do not determine these observations:

  • A larger AF gain need not identify the better-quality model. Increasing EE from 4 to 8 enlarges the AF gain from 2.2% to 6.1%, while absolute PPL worsens from 25.20 to 25.84 and the parameter premium rises from 24.6% to 42.2%. Reading both references together distinguishes improvement over a narrower baseline from improvement in absolute quality.
  • Load balance does not select the best quality in this sweep. Reducing λ\lambda from 0.01 to 0.001 improves PPL from 25.84 to 25.70 while load CV increases from 0.24 to 0.27. These observations motivate evaluating balancing strength jointly with quality.
  • Protocol variation can outweigh the architecture difference. A batch mismatch changes the four-expert AF comparison from a 2.2% gain to a 3.0% penalty despite a fixed token budget. The executable comparison audit makes this failure mode directly checkable.

These findings complement routed-model scaling studies. We will distinguish the reusable evaluation procedure from empirical magnitudes that depend on dataset, budget and implementation.

Q2. Longer budgets, additional datasets and uncertainty.

We additionally examined all 44 common logged checkpoints of the S-scale, batch-32, seed-0 runs. Both top-2 MoEs have higher PPL than TP dense throughout; from step 1000 through 21555, both outperform their respective AF references at all 43 common checkpoints. The two-sided ordering therefore persists across the recorded trajectory after its emergence, rather than appearing only at the selected best checkpoint. The batch-16 scale check also retains the four-expert TP gap at 96M and 203M: 4.9% and 5.1%, respectively.

Completed paired longer-budget or additional-dataset experiments are not available. The six-epoch dense result (23.27 to 21.10 PPL) demonstrates remaining optimization headroom, but cannot determine the extended-budget ranking without corresponding MoE and AF runs. We will explicitly scope the numerical quality conclusions to WikiText-103 and the stated training budget.

Table 7 repeats five MoE configurations with seeds 0 and 1; their ordering is unchanged, with a maximum reported absolute PPL change of 0.23. Dense-reference repeats are unavailable, so we cannot provide their training-seed uncertainty estimates. We will clarify that neither these MoE-only repeats nor checkpoints along one trajectory establish statistical significance of the smaller MoE-dense gaps.

Q3. Why did latency decrease sharply from S to M? Were conditions identical?

We addressed this concern with a new controlled S/M benchmark. The archived profiles came from separate runs and lack sufficient device/backend metadata to establish the cause of the discrepancy. The source-code audit found the same explicit expert loop and no explicit mixed-precision switch; it cannot verify identical historical runtime conditions.

The new measurements use one idle RTX 5090, FP32 with TF32 disabled, PyTorch 2.13.0/CUDA 13.0, one CPU thread, batch 32 and sequence length 512. We keep the same model implementation and use fixed synthetic token IDs with newly initialized models, as an architecture-level forward benchmark. Each model receives 3 warmups followed by two blocks of 10 forwards, with CUDA synchronization around each timed block.

ModelS latency (ms)M latency (ms)
TP dense93.02184.18
MoE, E=4E=495.26176.47
AF dense, E=4E=479.52153.52
MoE, E=8E=8101.62180.57
AF dense, E=8E=873.00138.03

Values are mean milliseconds per forward over both timed blocks. The large absolute S-to-M latency decrease is not reproduced. We will replace the affected S/M measurements and avoid treating the archived cross-run difference as an established scale effect; the unremeasured L profile will be identified separately.

The controlled results retain the relevant systems finding: MoE/AF latency ratios range from 1.151.15 to 1.391.39 across all four pairs, despite matched active FFN arithmetic. Thus, active-FFN matching does not establish equal runtime. This correction concerns the systems profiles; the separately trained TP/AF quality comparisons are unchanged.

Reviewer_QXv9OpenReview ↗

Reply to the author response

Thank you for the detailed response and for the new controlled latency benchmark. I appreciate the transparent correction. However, the response does not resolve my main concerns, and I keep my rating.

Thank you for the follow-up. We have now completed nine longer-budget runs covering both dense references and three seeds, an official-test evaluation, and a first matched training experiment on C4. These add experimental evidence beyond our previous response.

Q1. What scientific understanding does MOE-TAX add?

Our contribution is a controlled empirical characterization of when the two resource comparisons support different conclusions. The new results identify three concrete boundaries:

  • The AF gain changes substantially during training. Within the same six-epoch trajectories, its mean decreases from 1.690% at the halfway checkpoint to 0.645% at completion, while the TP deficit and AF advantage both persist across all three seeds on the training-tail validation split. Thus, the ordering can persist while the size of the benefit contracts.
  • The two conclusions have different robustness. On the official test split, MoE remains worse than TP-dense in all three seeds, whereas its smaller AF advantage reverses in one seed. The completed C4 triplet recovers both sides of the trade-off on a separately trained corpus. Reporting these outcomes together distinguishes a consistent TP deficit from an AF benefit that depends on the evaluation setting.
  • Matching forward arithmetic does not precisely match training arithmetic. An executed operation audit finds only 0.030% extra full-model forward matrix FLOPs for E4 versus AF4, but 3.376% extra forward-plus-backward matrix FLOPs with our gradient-checkpoint configuration. The trace identifies different recomputation work. These counts include attention and the output head, but exclude non-matrix operations and optimizer work.

Prior work establishes the distinction between stored parameters and active computation. Our empirical contribution quantifies how the quality comparison responds to training progress, evaluation and execution choices, within the tested scope.

Q2. Longer training, additional datasets and dense-reference uncertainty.

The new six-epoch study uses 43,110 updates and 706.3M sampled sequence tokens. TP-dense, E4-MoE and AF4-dense share the training recipe, batch size and minibatch indices within each seed, using an independent data RNG. All nine runs reached the fixed final endpoint.

Completed experimentSeedsTP PPLE4-MoE PPLAF4 PPL
Six-epoch WikiText, training-tail validation321.892 ± 0.08522.443 ± 0.02022.589 ± 0.030
Same final checkpoints, official WikiText test322.388 ± 0.04722.900 ± 0.08322.944 ± 0.096
New C4 training, official-validation prefix152.43754.57855.735

Uncertainty is sample SD across seeds 0/1/2, not a significance claim. At the six-epoch validation endpoint, the paired TP penalty is 2.519% and AF gain is 0.645%. On the official test, these become 2.287% and 0.190%; individual AF gains are +0.444%, +0.416% and -0.290%. We report the reversal explicitly. Test evaluation uses fixed final checkpoints, training-compatible tokenization and no test-based checkpoint selection. These are GPT-2 subword block perplexities.

The C4 experiment trains all three models from scratch for 353.2M sampled sequence tokens, using a fixed 117.7M-token training prefix and a separate 204,800-token official-validation prefix. Its MoE has 4.08% higher PPL than TP-dense and 2.08% lower PPL than AF-dense. This extends the observed ordering beyond WikiText in one completed seed; the remaining C4 repeats are running and are not included in this claim.

We also completed historical-style three-epoch repeats including the dense references: TP/MoE/AF PPLs are 24.297 ± 0.025 / 25.224 ± 0.038 / 25.813 ± 0.035, with a paired AF improvement of 0.588 ± 0.022 PPL. This row combines archived and new runs under the original global-RNG convention and best-recorded endpoint; its variability can include execution differences. We keep it separate from the new fixed-final study. The six-epoch halfway checkpoint is not an independent three-epoch experiment.

We will clarify that historical validation held out the last 400 cached training-stream blocks, sampling used replacement, and AdamW beta2 was 0.999. Our conclusions concern finite-budget training.

Q3. The S-to-M latency anomaly.

The controlled rerun does not reproduce the anomaly: E4 latency increases from 95.26 to 176.47 ms and E8 from 101.62 to 180.57 ms. All configurations use one idle RTX 5090, FP32 with TF32 disabled, the same expert-loop implementation, batch 32 and length 512. We will replace the affected S/M values. The old separate-session profiles lack sufficient metadata to establish the cause. The new MoE/AF latency ratios remain 1.15-1.39, so active-FFN matching still does not imply equal runtime.

We would appreciate your reassessment in light of these new completed experiments.

Reviewer_dn5jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MOE-TAX, an experimental accounting protocol for evaluating a sparse Mixture-of-Experts (MoE) model against two distinct dense references: one with approximately equal total parameters (TP), and one with equal active FFN FLOPs (AF). Under these controlled comparisons, the authors report that, at around 96M total parameters, sparse configurations have 3.8–38.2% higher perplexity than parameter-matched dense models, yet the four- and eight-expert MoEs achieve 2.2–6.1% lower perplexity than their AF-matched dense references while storing 24.6–42.2% more parameters. The paper further studies expert count, top-k activation, routing choices, auxiliary balancing, batch-size confounding, and forward latency/memory under a specific implementation.

Critical strengths
  1. A useful comparison discipline for the literature. Presenting TP-dense, MoE, and AF-dense side by side prevents readers from confusing a parameter-efficiency claim with a compute-efficiency claim. The parameter and active-FFN matching equations are transparent, and the manuscript explicitly distinguishes nominal FLOPs from measured latency and memory.

  2. A clean, pedagogical example of protocol confounding. The batch-16 versus batch-32 experiment shows that batch size changes results enough to reverse an apparent compute-matched ranking at fixed tokens. This is useful methodological guidance and is documented as a deliberately invalid mixed-batch comparison rather than promoted as a valid result.

  3. Several informative architectural controls. In the explored small-scale setting, expert count, top-1 vs top-2 activation, hash vs learned routing, and loss coefficient affect perplexity differently. The all-experts-active E=2,k=2 control is particularly useful in separating the gated-expert architecture itself from sparse activation.

Critical weaknesses
  1. The central 'two-sided' resource-accounting insight substantially overlaps established MoE scaling work; the empirical novelty needs sharper positioning. The manuscript cites Clark et al., Unified Scaling Laws for Routed Language Models (ICML 2022), which already models parameters and computation as independent axes over a large spectrum of routed-model sizes. Ludziejewski et al., Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024), analyze expert granularity, training tokens, model size, and compute-optimal choices. Artetxe et al. (EMNLP 2022) already conduct compute-matched dense/MoE comparisons. MOE-TAX packages two dense comparisons around the same sparse checkpoint and adds protocol auditing and a batch-size diagnostic, which is sensible but not itself a new architectural principle or scaling law. At ICLR, a careful empirical paper need not introduce a new algorithm; however, it should establish a sufficiently surprising, generalizable finding beyond what follows almost by construction when total stored capacity exceeds active capacity. I would appreciate a precise table of what the prior matched-budget literature measures versus what MOE-TAX uniquely establishes, ideally with a stronger analysis that predicts where the sign or magnitude of the trade changes.

  2. The reported 'compute match' is only an active-FFN-FLOP match, not a total training-compute or actual inference-cost match. The AF reference sets the dense FFN width to the selected total expert width (Eq. 2), but it does not equalize attention, routing, dispatch, expert batching, wall-clock cost, or total training FLOPs. To the paper's credit, this limitation is stated explicitly. Nevertheless, labeling the results 'compute-matched' without always carrying the 'active-FFN' qualifier risks a stronger conclusion than the experiment supports. The TP comparison likewise fixes stored parameters while allowing the sparse model to perform much less work per token. The study therefore does not determine which architecture is best at a fixed total pretraining FLOP budget, GPU-hour budget, inference latency, or deployment memory limit. These distinctions are especially important because the authors' actual runtime results can have the opposite ranking from nominal FLOPs. At minimum, report full-model measured forward FLOPs, training FLOPs/step, and achieved tokens/s for the paired configurations; carefully delimit the decision rules this study can support.

  3. The quality study is too small and undertrained for broad claims about what MoE sparsity buys. The primary controlled quality sweep uses ~96M-parameter models and three epochs (~353M tokens) of WikiText-103; the scale check extends only to ~203M, while ~531M is profiled for systems performance without quality training. The 96M/203M runs see only about 3.7/1.7 training tokens per parameter, respectively. These are legitimate controlled short-budget experiments, but the relative quality of dense and MoE architectures can change with both training duration and scaling; MoE specialization may be substantially underdeveloped after such limited pretraining. Indeed, Appendix G shows that simply doubling the dense training budget from three to six epochs materially improves its perplexity. The paper should present the conclusion as limited to short-budget, small-scale experiments and avoid suggesting an MoE quality-tax law. A modest, paired longer-budget sweep at the same model scale (not a new multi-billion-parameter project) would be highly informative.

  4. The systems measurements are dominated by a deliberately unfused expert-loop implementation. Figure 6 reports 505.9 ms for four experts and 856.9 ms for eight experts versus 183.8 ms for dense at the 96M scale, despite lower active FFN FLOPs. The paper clearly acknowledges that its per-expert loop is not representative of optimized MoE execution. However, deployment decisions typically involve token-grouped kernels, block-sparse expert computation, expert parallelism, communication, and varying batch/concurrency; MegaBlocks, Tutel, and DeepSpeed-MoE are relevant precisely because they change these costs. Hence the observed 2.8–4.7× latency penalty at the small scale is not evidence about the intrinsic MoE-versus-dense runtime trade. If systems evidence is promoted as a contribution, one optimized grouped expert implementation at a realistic utilization point would add much more value than profiling additional sizes with the same inefficient loop. Otherwise, limit conclusions to the included implementation.

  5. The architectural ablations do not identify a general mechanism or quality frontier. Increasing expert count E at matched parameters and fixed k simultaneously reduces expert width, active FFN capacity, and routes per expert; the perplexity change cannot uniquely be attributed to 'expert sparsity.' The E=k=2 all-experts-active experiment still underperforms dense, demonstrating a gated-multi-expert architectural penalty even without sparse selection. This is a useful observation, but it also complicates describing the TP difference as the 'sparsity tax' alone. Experiments that hold active expert width or active FFN FLOPs fixed while changing expert count, and a normalized inference/training compute frontier, would help disentangle routing, expert fragmentation, and effective capacity. The public Qwen3-30B-A3B versus Qwen3-8B comparison is explicitly labeled non-matched and should remain illustrative rather than evidence for the causal claim.

Questions for authors
  1. What genuinely new empirical regularity does MOE-TAX reveal beyond prior parameter/compute scaling studies, and can it predict a break-even architecture from the measured capacity premium and quality gap?

  2. How does the TP and AF ranking evolve in a longer-budget same-model experiment, with all other settings held fixed?

  3. Can the authors report total training and inference FLOPs and wall-clock measurements for matched pairs, rather than only active FFN work? Where does the AF comparison land when actual end-to-end runtime is equalized?

  4. Which conclusions remain under a modern grouped/block-sparse MoE kernel, or alternatively, how should a practitioner interpret results that depend strongly on the explicit expert-loop implementation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dn5j

Thank you for recognizing the comparison protocol and controls. We provide additional analysis of saved curves and model arithmetic below.

Q1. Empirical contribution and break-even interpretation.

Our contribution complements prior work:

WorkMain emphasis
Clark et al. (2022)Parameter/compute scaling laws for routed models.
Ludziejewski et al. (2024)Scaling with tokens, model size and expert granularity; compute-optimal allocation.
Artetxe et al. (2022)Compute-matched dense/MoE quality and generalization.
MOE-TAXSame-MoE TP/AF triplets, explicit parameter premiums, protocol audits and implementation-specific systems measurements.

The equations determine resource budgets, not quality. Increasing the expert count from 4 to 8 increases the AF gain from 2.2% to 6.1%, yet worsens absolute PPL from 25.20 to 25.84 and raises the parameter premium from 24.6% to 42.2%. A larger AF gain need not identify the better-quality model. Moreover, the 4.4% dense batch-size effect exceeds the four-expert AF gain and can reverse the comparison's sign. Neither observation follows from stored versus active capacity alone.

MoE beats AF dense when its penalty relative to TP dense is smaller than the penalty from narrowing dense to the AF width: 3.8% versus 6.1% for four experts, and 6.4% versus 13.4% for eight experts. This explains the observed break-even condition; parameter premiums alone cannot predict unseen architectures. We contribute a reproducible empirical account of this exchange, not a universal scaling law.

Q2. Evolution with training budget.

We inspected all 44 common checkpoints of the S-scale, batch-32, seed-0 runs. Both top-2 MoEs have higher PPL than TP dense at every checkpoint. At the first checkpoint (step 500), neither yet beats AF dense; from step 1000 through 21555, both outperform their respective AF references at all 43 common checkpoints. Thus, the two-sided ordering emerges during training and persists throughout the remaining recorded trajectory.

This is trajectory evidence within the three-epoch schedule, not a completed longer-budget comparison. The available six-epoch result is dense-only (23.27 to 21.10 PPL); it cannot establish the extended-budget TP/AF ranking. We will attach the numerical quality findings explicitly to the 353M-token, 96M-203M regime and use "tax" for the measured TP gap, not a universal law or an isolated causal effect of sparse selection.

Q3. Whole-model arithmetic, wall-clock cost and equal runtime.

AF pairs share width, depth, sequence length, vocabulary and backbone, so attention and vocabulary-projection arithmetic are also identical. Extending accounting beyond FFNs gives the following analytical leading matrix-operation counts (two FLOPs per multiply-add):

S-scale modelForward MFLOPs/tokenFwd+bwd TFLOPs/step (estimate)Unified rerun forward ms
TP dense203.0249.95893.02
MoE E4165.3248.10995.26
AF dense E4165.2758.10779.52
MoE E8146.4997.185101.62
AF dense E8146.4017.18173.00

Forward counts use length 512. Training estimates use batch 32, input length 511 and approximately three times the forward matrix FLOPs at that length; checkpoint recomputation is excluded. These analytical counts include attention, FFN, router and output projections; they exclude non-matrix operations and optimizer work and are not profiler-measured total costs. Latency is from a new unified forward-only rerun: one idle RTX 5090, FP32 with TF32 disabled, batch 32, length 512, the same expert loop, fixed synthetic inputs and new model initialization, 3 warmups and two synchronized blocks of 10 forwards.

AF pairs are close in leading arithmetic but differ substantially in runtime. We will retain the "active-FFN" qualifier consistently. No equal-runtime quality comparison is available; AF gains do not establish a quality advantage at fixed GPU-hours or latency.

Q4. Conclusions under optimized kernels.

For a fixed checkpoint, a numerically equivalent grouped/block-sparse implementation preserves the model function and TP/AF accounting. The measured latency and peak-memory ratios must be re-evaluated for that implementation. The unified S/M rerun did not reproduce the earlier 2.8 to 4.7 times S-scale penalty relative to TP dense. MoE/AF latency ratios are 1.15 to 1.39 across the four new pairs. These are implementation-specific measurements; the unremeasured L profile will be identified separately.

Practitioners should combine the paired quality/resource accounting with profiling of their actual kernel and workload. We will emphasize this distinction between the reusable protocol and implementation-specific systems results.

Reviewer_KHNeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares each MoE model with two dense references: one matched in total parameters and another matched in active FFN compute. Controlled language-model experiments quantify the quality cost at matched parameters and the quality gain at matched active compute, together with the additional parameters required. Ablations, batch-size controls, and device profiling examine the factors behind this trade-off.

Critical strengths
  1. Comparing the same MoE with both dense references makes the resource trade-off explicit. The paper clearly distinguishes active FFN compute from measured latency and memory.

  2. The training-protocol controls are useful, particularly the example showing that a batch-size mismatch can reverse the apparent result. The paper also separates controlled experiments from public-model comparisons and implementation-specific profiling.

Critical weaknesses
  1. The top-1 routing implementation would benefit from clarification. Section 3.2 describes applying softmax to the selected top-k logits, which gives a constant weight of one when k=1. Under ordinary hard selection, this would remove the language-model gradient to the router through the gating weight. It is therefore unclear whether the activation ablation also changes how the router learns, in addition to reducing active capacity.

  2. The seed check in Table 7 provides useful evidence that the MoE configuration ordering is stable. Since the dense references are not included in this check, the stability of the smaller MoE-versus-dense differences, such as the four-expert AF gain of 0.58 perplexity, remains less clear.

Questions for authors
  1. Could you clarify how the top-1 router receives the language-model training signal, and whether this affects the interpretation of the activation comparison?

  2. If repeat-seed results for the dense references are available, could you summarize them? Otherwise, a brief note on the scope of the current seed check would help contextualize the smaller gains.

1 comment

Thank you for your careful review and for recognizing the value of the two-sided comparisons and protocol controls. We checked the implementation and experiment records to address your questions.

Q1. How is the top-1 router trained, and what does the activation comparison show?

The top-1 router receives a training gradient from the auxiliary load-balancing loss. You are correct that it receives no direct LM gradient through the gating operation: our implementation normalizes the selected logits, so the top-1 weight is identically one, and hard selection does not provide a differentiable path.

The balancing loss uses probabilities from a separate softmax over ALL experts:

Laux=λE∑efePe,L_{\mathrm{aux}}=\lambda E\sum_e f_e P_e,

where PeP_e is the mean full-softmax probability and fef_e is the argmax assignment fraction. Gradients flow through PeP_e. A small CPU autograd check using the actual model confirmed zero LM gradient and a nonzero auxiliary-loss gradient to the top-1 router; the top-2 router received a nonzero LM gradient. Thus, the top-1 router is trained for balancing, while the experts and shared backbone receive the LM training signal.

We agree that the top-1/top-2 ablation therefore measures the combined effect of changing active capacity and the router's access to task-dependent gradients. We will make this interpretation explicit in Sections 3.2, 4.5, and 4.6, including the two-expert control, and remove attribution of the full gap solely to active capacity.

The main paired-budget finding is supported independently by the learned top-2 configurations in Tables 1-3, which retain task-dependent gating gradients. The same four-expert top-2 model achieves PPL 25.20 versus 24.29 for parameter-matched dense and 25.78 for active-FFN-compute-matched dense. The eight-expert top-2 model exhibits the same two-sided pattern (+6.4% PPL against TP dense; -6.1% against AF dense). These central comparisons do not use the top-1 ablation. The clarification therefore narrows the causal interpretation of that ablation while preserving the evidence for the resource exchange.

Q2. Are dense-reference repeat seeds available, and what does the current seed check establish?

We checked the available records and found no completed repeat-seed runs for the dense references. Table 7 repeats five MoE configurations with seeds 0 and 1. All five retain the same ordering, with a maximum reported absolute change of 0.23 PPL; the four-expert top-2 model changes from 25.20 to 25.27.

As you suggest, we will explicitly state the scope of this check in Section 4.6 and the Table 7 caption: it documents consistency of the tested MoE ordering across two seeds, but does not provide a paired-seed uncertainty estimate for MoE-dense differences. We will also clarify in Section 4.2 that the four-expert AF improvement of 0.58 PPL (25.78 versus 25.20, or 2.2%) is an observed matched-protocol result, whose statistical significance is not established by the MoE-only seed check.

Thank you for these specific suggestions. They help us make the ablation interpretation and the strength of the supporting evidence more precise.

Comments on the paper

1 comment
Area_Chair_wcKSOpenReview ↗

Please respond to the rebuttal

Dear Reviewers,

The authors have posted their rebuttal. Please read it and let them know whether it addresses your concerns, updating your review and score if appropriate, so that they have a chance to address any remaining ones.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.