- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper empirically studies overfitting of Mixture-of-Experts (MoE) models under repeated-data training. Comparing dense and MoE models at 80M, 200M, and 1B activated parameters, with repetition rates up to 1024 and several data domains and mixtures, the authors find that MoEs overfit earlier and more severely than dense models matched by activated parameters, increasingly so with higher sparsity, and argue that the effect is governed by total rather than active parameters. Mixing in non-repeated data can partially mitigate this, depending on the domains. Among regularization methods, dropout and output masking help, whereas weight decay, gradient clipping, and router jitter do not, although none recovers all-unique-data performance. Finally, routing-stability and expert-knockout analyses associate repetition with earlier routing stabilization and stronger expert specialization.
Critical strengths
-
Timely and relevant question. The paper systematically studies how MoE sparsity, expert count, and granularity interact with data repetition, whereas prior work examined only a single MoE configuration.
-
Broad experiments with consistent findings. Results span three scales, several domains, and mixing and filtering settings, with consistent trends and gaps well beyond seed variance.
-
Useful regularization comparison. The paper shows that dropout and output masking mitigate overfitting while weight decay, gradient clipping, and router jitter do not, and acknowledges that no method matches all-unique data.
Critical weaknesses
-
Confounded MoE–dense comparison. Models are matched by active parameters, so MoEs have substantially more total parameters, which the paper itself identifies as a potential driver of the observed effect. Li et al. (Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource, ICLR 2026), not discussed in the paper, find that MoEs can still outperform total-parameter-matched dense models under several epochs of data repetition. The observed sensitivity may therefore reflect differences in total model capacity rather than sparsity itself. Dense baselines matched by total parameters across different repetition rates are needed to disentangle these factors.
-
Limited regularization tuning. Weight decay is swept only up to 0.4 with a fixed learning rate. Kim et al. (Pre-training under Infinite Compute, ICLR 2026), not discussed in the paper, find that under data constraints the optimal weight decay for over-parameterized dense models is 0.8–3.2, i.e., 8–32$\times$ the standard value of 0.1. Although their setting differs, this raises the possibility that stronger regularization could mitigate the observed overfitting. It therefore remains unclear whether the sensitivity of MoEs is intrinsic to sparsity or partly a consequence of insufficient regularization, which also weakens the paper’s mechanistic conclusions.
Questions for authors
-
Total-parameter-matched baselines. How do dense models matched to the MoEs by total parameter count perform across repetition rates? If they exhibit similar degradation, to what extent would this change the interpretation of the observed effect and the paper's practical recommendations regarding MoE sparsity?
-
Stronger regularization. Does the performance gap between MoE and dense models persist with substantially stronger weight decay (e.g., ) and separately tuned learning rates? Given that decoupled weight decay may also update expert parameters that receive no tokens, depending on the optimizer implementation, could the authors clarify how weight decay is applied to inactive experts and whether this affects MoEs differently from dense models?
-
Mechanistic interpretation. Could the authors reconcile the discussion in Section 5.1, which suggests that dropout reduces routing ossification, with the caption of Figure 23, which reports increased routing stability under stronger dropout? More broadly, do the authors consider routing ossification a causal contributor to overfitting or primarily a correlated phenomenon?
3 comments
W2 and Q2: Stronger regularization. We thank the reviewer for raising this concern, which we agree is important. Kim, et. al., indeed find much larger optimal weight decay values than typically used. They also find, in line with prior work, that the optimal WD setting depends on model size. Referring to their Figure 3 and the paired table, they find that a 150M parameter model in their R=16 setting with 200M data tokens has an optimal WD of 0.8. Following their power law trend yields an optimal WD around 0.4 for an 80M parameter model. However, upon a closer reading, we were able to find in the appendix that their batch size was only 64 sequences. As a result, they train with ~8x our step count, applying weight decay 8x as many times. Thus, we have run additional experiments with WD 1.6, 3.2, 6.4, 12.8 and 25.6. We already had WD=0.8 settings in our 80M experiments but did not include it in the paper text:
| Model | WD | R=1 (Valid Loss) | R=32 (Valid Loss) |
|---|---|---|---|
| Dense | 0.05 | 4.919 | 5.306 |
| Dense | 0.1 | 4.901 | 5.247 |
| Dense | 0.2 | 4.923 | 5.279 |
| Dense | 0.4 | 4.934 | 5.268 |
| Dense | 0.8 | 4.999 | 5.280 |
| Dense | 1.6 | 4.882 | 5.194 |
| Dense | 3.2 | 4.846 | 5.096 |
| Dense | 6.4 | 4.847 | 5.023 |
| Dense | 12.8 | 5.028 | 5.132 |
| Dense | 25.6 | 5.508 | 5.544 |
| MoE (64 x 1/4) | 0.05 | 4.608 | 6.071 |
| MoE (64 x 1/4) | 0.1 | 4.605 | 6.063 |
| MoE (64 x 1/4) | 0.2 | 4.611 | 6.009 |
| MoE (64 x 1/4) | 0.4 | 4.620 | 5.965 |
| MoE (64 x 1/4) | 0.8 | 4.646 | 5.924 |
| MoE (64 x 1/4) | 1.6 | 4.546 | 6.127 |
| MoE (64 x 1/4) | 3.2 | 4.523 | 6.127 |
| MoE (64 x 1/4) | 6.4 | 4.556 | 5.968 |
| MoE (64 x 1/4) | 12.8 | 4.828 | 5.340 |
| MoE (64 x 1/4) | 25.6 | 5.371 | 5.433 |
(* Note that there is a slight change in the pattern at WD 0.8 -> 1.6, as we originally used the same weight decay for norm gains, but at WD 1.6 - 25.6, fixed norm gains decay to 0.1, which is a best practice based on prior work like nanoGPT for cases where WD is higher. This has only a small impact at WD 0.8, but we wanted to note this for full context.)
WD 3.2, 6.4, or 12.8 achieve best performance on all 4 (Dense, MoE) x (R=1, R=32) settings. With optimal WD, the gap between R=1 and R=32 settings for both dense and MoE models is reduced but not closed. It does not appear that weight decay alone is sufficient to mitigate data repetition damage, even at much higher weight decay settings, but we hope that future work will consider more complex combinations of weight decay with other regularizers, which may close the gap between dense and sparse models
As for whether the sensitivity is intrinsic to sparsity, we found that dropout successfully reduces data repetition effects in MoEs, as shown in the S4 and Figure 8(a). We do not find any single regularization method which entirely removes the increased sensitivity of sparse models to data repetition. A full grid search of all possible hyperparameter combinations is intractable, but we hope that future work will further investigate combinations of regularization methods.
Regarding how weight decay is applied: we use PyTorch AdamW, so decay is decoupled and scaled by the current learning rate. All experts of a layer are stored in one tensor, so every expert is decayed at every step whether or not it received tokens. In practice no expert goes without tokens. A batch has about 1M tokens, so the average expert in MoE (64 x 1/4) receives about 65,000 per step, and in our routing analysis of 71 runs no expert had zero tokens. Decay therefore acts the same way on dense and MoE weights. Though the gradient differs, since each expert receives a gradient from k/n of the tokens.
We tuned the learning rate early in our experimental process, and found that the true optimum varies by setting, but chose lr=4e-4, which was near the optimum for all settings we were able to sweep at the time, and retained the overall pattern of the trends we observed. We have added a new sweep for the finalized settings in the paper:
| Model | LR | R=1 (Valid Loss) | R=32 (Valid Loss) |
|---|---|---|---|
| Dense | 2e-4 | 5.377 | 5.542 |
| Dense | 4e-4 | 4.901 | 5.247 |
| Dense | 8e-4 | 4.573 | 5.283 |
| MoE (64 x 1/4) | 2e-4 | 5.095 | 5.501 |
| MoE (64 x 1/4) | 4e-4 | 4.605 | 6.063 |
| MoE (64 x 1/4) | 8e-4 | 4.282 | 7.318 |
We have additional learning rates running as well, and will add text to the paper to make this variance of optimal learning rate clear, but we emphasize that tuning the learning rate does not appear to close the gap between dense and sparse model responses to data repetition, it shifts the absolute values of the losses.
Q3 (Section 5.1 and Figure 23). Thank you for catching this discrepancy. The caption is correct, and we will correct the text in Section 5.1 accordingly. Dropout makes routing slightly more stable. For example, with repetition count R=64, stability is 0.875 without dropout, 0.903 with dropout 0.1 and 0.913 with dropout 0.4. Raising dropout from 0.1 to 0.4 increases stability at every R by 0.010 to 0.028. There was a discrepancy in our plot itself, which we will correct; the revised plots show that dropout always increases stability in our experiments.
To clarify, we consider routing ossification a correlate and not a cause. Dropout substantially reduces overfitting while also making routing slightly more stable, so routing stability cannot explain the overfitting. We will revise the wording in the paper to clearly emphasize our evidence supports correlation rather than causation.
We thank reviewer HB1G for the positive review pointing out the breadth, consistency, and relevance of our experiments, and for the two references. We address the reviewer's questions and weaknesses below:
W1 and Q1: Active parameters. We will revise the wording in the paper to make our settings and claims clearer: Our work shows that at fixed active parameters and fixed data, the damage from repetition follows total parameters. Our original submission does not include experiments designed to vary active parameters at a fixed total. This is in large part because, unlike total parameters, varying active parameters by definition also alters the compute budget at fixed data and vice versa. Thus, there is no clear candidate setting for a fair comparison when changing active parameters while holding the total fixed. In S3.1 there is currently this text: "we compare the 200M dense models to the 80M MoE (32 x 1/4) and MoE (64 x 1/4) models, which have 158M and 244M total parameters, respectively. In Figure 2, the data repetition effects of the 200M model appear to lie between those of the MoE (32 x 1/4) and MoE (64 x 1/4) models at 80M active parameters."
We agree with the reviewer that it would strengthen our work to make this comparison more direct. We have added a new experiment. We compare the 80M MoE (32 x 1/4) and MoE (64 x 1/4) with 2 new dense models, based on the dense 80M models, but where each feed-forward layer is 8 or 16 times wider, respectively, thus matching MoE (32 x 1/4) and MoE (64 x 1/4) total parameters. During this short response period, we run R=1, 32 settings. We will add all other R values to the camera ready.
At 158M total parameters, both dense (158M active) and MoE (80M active) models see an identical R=1 -> R=32 loss increase of +0.83. Active parameters alone do not correlate with data repetition effects in this setting. Similarly, at 245M total parameters, the dense (245M active) and MoE (80M active) models see a R=1 -> R=32 loss degradation of +1.39 and +1.46, respectively. In other words, at 80M active parameters, increasing total parameters by 55% correlates with loss delta increase by 67-75%. At 158M or 245M total parameters, increasing active parameters by 97-210% correlates with only loss delta decrease by 0-5% We believe that this result provides some more direct evidence suggesting that data repetition effects may be more directly correlated with total than with active parameters. We will add the full final results with additional R values to the camera ready with appropriate discussion.
We will also add more explicit practical takeaways to our camera ready: in a data-constrained setting requiring high repetition, adding more total parameters to MoE may not result in improved performance, and it may be instead beneficial to train a dense model to the same number of data repetitions, or to train a larger dense model on fewer repetitions of the data. Further, the practice of using "scaling ladders," may be especially misleading under data constraints if insufficient data points are collected, as data repetition effects may increase when small-scale MoEs are scaled up.
Re Li et al. (ICLR 2026): We thank the reviewer for the reference, which we will add. Their setup differs from ours in an important way. Their dense models never see repeated data. At equal compute the MoE is cheaper per token, so it sees the dense model's data 1.7x to 8.3x.. They also note that more than two epochs hurt at 7B, and they report MMLU falling from 32.9 to 24.6 under reuse, below the dense model's 31.3. This is in line with our low-R results, where the 80M MoEs outperform dense models up to R=8.