acceptodds
Under review as a conference paper at ICLR 2027

Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

Abstract

As the supply of human-written text is exhausted, it has become standard practice to repeat language model training data. Prior work has studied data repetition for densely activated Transformers, but the effects of data repetition remains largely unexplored for recently dominant sparse architectures such as Mixture-of-Experts (MoE), despite their increased compute efficiency. We vary data repetition rates across single- and multi-domain data mixes, and across MoE settings, including expert count and granularity. We consistently find, for models ranging from 80M to 1B active (8.5B total) parameters, that MoEs degrade more rapidly under data repetition. This effect increases with sparsity, dictated by total rather than active parameters. While 80M dense models can repeat data over 8× with minimal degradation, MoEs (active-parameter-matched) instead begin to suffer at 4 repetitions, and deteriorate rapidly, ceding their performance benefits in all-unique data settings to dramatically underperform dense models after 32 repetitions. We experiment with existing regularization methods as a potential remedy. We find that some methods, such as dropout, can mitigate overfitting. In particular, with strong masking-based regularization, MoEs are able to outperform dense models even when data is repeated more than 64 times. However, no method fully matches the performance of all-unique training data with matched total data and compute. Finally, we analyze internal mechanisms correlated with MoE overfitting in high repetition regimes, and find that MoE routing universally stabilizes early in training, and that expert specialization correlates with overfitting to repeated data. In sum, our work addresses the adverse interactions between sparsity and data repetition: we present evidence for the core mechanisms of overfitting and its potential remediation, and suggest promising avenues for future methods to reduce over-specialization in model parameters by disrupting memorization patterns.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data, data repetition, MoE, MoEs, sparsity, scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HB1GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper empirically studies overfitting of Mixture-of-Experts (MoE) models under repeated-data training. Comparing dense and MoE models at 80M, 200M, and 1B activated parameters, with repetition rates up to 1024 and several data domains and mixtures, the authors find that MoEs overfit earlier and more severely than dense models matched by activated parameters, increasingly so with higher sparsity, and argue that the effect is governed by total rather than active parameters. Mixing in non-repeated data can partially mitigate this, depending on the domains. Among regularization methods, dropout and output masking help, whereas weight decay, gradient clipping, and router jitter do not, although none recovers all-unique-data performance. Finally, routing-stability and expert-knockout analyses associate repetition with earlier routing stabilization and stronger expert specialization.

Critical strengths
  1. Timely and relevant question. The paper systematically studies how MoE sparsity, expert count, and granularity interact with data repetition, whereas prior work examined only a single MoE configuration.

  2. Broad experiments with consistent findings. Results span three scales, several domains, and mixing and filtering settings, with consistent trends and gaps well beyond seed variance.

  3. Useful regularization comparison. The paper shows that dropout and output masking mitigate overfitting while weight decay, gradient clipping, and router jitter do not, and acknowledges that no method matches all-unique data.

Critical weaknesses
  1. Confounded MoE–dense comparison. Models are matched by active parameters, so MoEs have substantially more total parameters, which the paper itself identifies as a potential driver of the observed effect. Li et al. (Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource, ICLR 2026), not discussed in the paper, find that MoEs can still outperform total-parameter-matched dense models under several epochs of data repetition. The observed sensitivity may therefore reflect differences in total model capacity rather than sparsity itself. Dense baselines matched by total parameters across different repetition rates are needed to disentangle these factors.

  2. Limited regularization tuning. Weight decay is swept only up to 0.4 with a fixed learning rate. Kim et al. (Pre-training under Infinite Compute, ICLR 2026), not discussed in the paper, find that under data constraints the optimal weight decay for over-parameterized dense models is 0.8–3.2, i.e., 8–32$\times$ the standard value of 0.1. Although their setting differs, this raises the possibility that stronger regularization could mitigate the observed overfitting. It therefore remains unclear whether the sensitivity of MoEs is intrinsic to sparsity or partly a consequence of insufficient regularization, which also weakens the paper’s mechanistic conclusions.

Questions for authors
  1. Total-parameter-matched baselines. How do dense models matched to the MoEs by total parameter count perform across repetition rates? If they exhibit similar degradation, to what extent would this change the interpretation of the observed effect and the paper's practical recommendations regarding MoE sparsity?

  2. Stronger regularization. Does the performance gap between MoE and dense models persist with substantially stronger weight decay (e.g., λ≥1\lambda \geq 1) and separately tuned learning rates? Given that decoupled weight decay may also update expert parameters that receive no tokens, depending on the optimizer implementation, could the authors clarify how weight decay is applied to inactive experts and whether this affects MoEs differently from dense models?

  3. Mechanistic interpretation. Could the authors reconcile the discussion in Section 5.1, which suggests that dropout reduces routing ossification, with the caption of Figure 23, which reports increased routing stability under stronger dropout? More broadly, do the authors consider routing ossification a causal contributor to overfitting or primarily a correlated phenomenon?

3 comments

We thank reviewer HB1G for the positive review pointing out the breadth, consistency, and relevance of our experiments, and for the two references. We address the reviewer's questions and weaknesses below:

W1 and Q1: Active parameters. We will revise the wording in the paper to make our settings and claims clearer: Our work shows that at fixed active parameters and fixed data, the damage from repetition follows total parameters. Our original submission does not include experiments designed to vary active parameters at a fixed total. This is in large part because, unlike total parameters, varying active parameters by definition also alters the compute budget at fixed data and vice versa. Thus, there is no clear candidate setting for a fair comparison when changing active parameters while holding the total fixed. In S3.1 there is currently this text: "we compare the 200M dense models to the 80M MoE (32 x 1/4) and MoE (64 x 1/4) models, which have 158M and 244M total parameters, respectively. In Figure 2, the data repetition effects of the 200M model appear to lie between those of the MoE (32 x 1/4) and MoE (64 x 1/4) models at 80M active parameters."

We agree with the reviewer that it would strengthen our work to make this comparison more direct. We have added a new experiment. We compare the 80M MoE (32 x 1/4) and MoE (64 x 1/4) with 2 new dense models, based on the dense 80M models, but where each feed-forward layer is 8 or 16 times wider, respectively, thus matching MoE (32 x 1/4) and MoE (64 x 1/4) total parameters. During this short response period, we run R=1, 32 settings. We will add all other R values to the camera ready.

ModelTotalActiveR=1 (Valid Loss)R=32 (Valid Loss)Loss Increase
Dense81.9M81.9M4.9015.247+0.345
Dense, FFN 8x157.8M157.8M4.6465.480+0.834
MoE (32 x 1/4)157.7M80M4.6705.503+0.833
Dense, FFN 16x244.5M244.5M4.5395.927+1.388
MoE (64 x 1/4)244.6M80M4.6056.063+1.457

At 158M total parameters, both dense (158M active) and MoE (80M active) models see an identical R=1 -> R=32 loss increase of +0.83. Active parameters alone do not correlate with data repetition effects in this setting. Similarly, at 245M total parameters, the dense (245M active) and MoE (80M active) models see a R=1 -> R=32 loss degradation of +1.39 and +1.46, respectively. In other words, at 80M active parameters, increasing total parameters by 55% correlates with loss delta increase by 67-75%. At 158M or 245M total parameters, increasing active parameters by 97-210% correlates with only loss delta decrease by 0-5% We believe that this result provides some more direct evidence suggesting that data repetition effects may be more directly correlated with total than with active parameters. We will add the full final results with additional R values to the camera ready with appropriate discussion.

We will also add more explicit practical takeaways to our camera ready: in a data-constrained setting requiring high repetition, adding more total parameters to MoE may not result in improved performance, and it may be instead beneficial to train a dense model to the same number of data repetitions, or to train a larger dense model on fewer repetitions of the data. Further, the practice of using "scaling ladders," may be especially misleading under data constraints if insufficient data points are collected, as data repetition effects may increase when small-scale MoEs are scaled up.

Re Li et al. (ICLR 2026): We thank the reviewer for the reference, which we will add. Their setup differs from ours in an important way. Their dense models never see repeated data. At equal compute the MoE is cheaper per token, so it sees the dense model's data 1.7x to 8.3x.. They also note that more than two epochs hurt at 7B, and they report MMLU falling from 32.9 to 24.6 under reuse, below the dense model's 31.3. This is in line with our low-R results, where the 80M MoEs outperform dense models up to R=8.

W2 and Q2: Stronger regularization. We thank the reviewer for raising this concern, which we agree is important. Kim, et. al., indeed find much larger optimal weight decay values than typically used. They also find, in line with prior work, that the optimal WD setting depends on model size. Referring to their Figure 3 and the paired table, they find that a 150M parameter model in their R=16 setting with 200M data tokens has an optimal WD of 0.8. Following their power law trend yields an optimal WD around 0.4 for an 80M parameter model. However, upon a closer reading, we were able to find in the appendix that their batch size was only 64 sequences. As a result, they train with ~8x our step count, applying weight decay 8x as many times. Thus, we have run additional experiments with WD 1.6, 3.2, 6.4, 12.8 and 25.6. We already had WD=0.8 settings in our 80M experiments but did not include it in the paper text:

ModelWDR=1 (Valid Loss)R=32 (Valid Loss)
Dense0.054.9195.306
Dense0.14.9015.247
Dense0.24.9235.279
Dense0.44.9345.268
Dense0.84.9995.280
Dense1.64.8825.194
Dense3.24.8465.096
Dense6.44.8475.023
Dense12.85.0285.132
Dense25.65.5085.544
MoE (64 x 1/4)0.054.6086.071
MoE (64 x 1/4)0.14.6056.063
MoE (64 x 1/4)0.24.6116.009
MoE (64 x 1/4)0.44.6205.965
MoE (64 x 1/4)0.84.6465.924
MoE (64 x 1/4)1.64.5466.127
MoE (64 x 1/4)3.24.5236.127
MoE (64 x 1/4)6.44.5565.968
MoE (64 x 1/4)12.84.8285.340
MoE (64 x 1/4)25.65.3715.433

(* Note that there is a slight change in the pattern at WD 0.8 -> 1.6, as we originally used the same weight decay for norm gains, but at WD 1.6 - 25.6, fixed norm gains decay to 0.1, which is a best practice based on prior work like nanoGPT for cases where WD is higher. This has only a small impact at WD 0.8, but we wanted to note this for full context.)

WD 3.2, 6.4, or 12.8 achieve best performance on all 4 (Dense, MoE) x (R=1, R=32) settings. With optimal WD, the gap between R=1 and R=32 settings for both dense and MoE models is reduced but not closed. It does not appear that weight decay alone is sufficient to mitigate data repetition damage, even at much higher weight decay settings, but we hope that future work will consider more complex combinations of weight decay with other regularizers, which may close the gap between dense and sparse models

As for whether the sensitivity is intrinsic to sparsity, we found that dropout successfully reduces data repetition effects in MoEs, as shown in the S4 and Figure 8(a). We do not find any single regularization method which entirely removes the increased sensitivity of sparse models to data repetition. A full grid search of all possible hyperparameter combinations is intractable, but we hope that future work will further investigate combinations of regularization methods.

Regarding how weight decay is applied: we use PyTorch AdamW, so decay is decoupled and scaled by the current learning rate. All experts of a layer are stored in one tensor, so every expert is decayed at every step whether or not it received tokens. In practice no expert goes without tokens. A batch has about 1M tokens, so the average expert in MoE (64 x 1/4) receives about 65,000 per step, and in our routing analysis of 71 runs no expert had zero tokens. Decay therefore acts the same way on dense and MoE weights. Though the gradient differs, since each expert receives a gradient from k/n of the tokens.

We tuned the learning rate early in our experimental process, and found that the true optimum varies by setting, but chose lr=4e-4, which was near the optimum for all settings we were able to sweep at the time, and retained the overall pattern of the trends we observed. We have added a new sweep for the finalized settings in the paper:

ModelLRR=1 (Valid Loss)R=32 (Valid Loss)
Dense2e-45.3775.542
Dense4e-44.9015.247
Dense8e-44.5735.283
MoE (64 x 1/4)2e-45.0955.501
MoE (64 x 1/4)4e-44.6056.063
MoE (64 x 1/4)8e-44.2827.318

We have additional learning rates running as well, and will add text to the paper to make this variance of optimal learning rate clear, but we emphasize that tuning the learning rate does not appear to close the gap between dense and sparse model responses to data repetition, it shifts the absolute values of the losses.

Q3 (Section 5.1 and Figure 23). Thank you for catching this discrepancy. The caption is correct, and we will correct the text in Section 5.1 accordingly. Dropout makes routing slightly more stable. For example, with repetition count R=64, stability is 0.875 without dropout, 0.903 with dropout 0.1 and 0.913 with dropout 0.4. Raising dropout from 0.1 to 0.4 increases stability at every R by 0.010 to 0.028. There was a discrepancy in our plot itself, which we will correct; the revised plots show that dropout always increases stability in our experiments.

To clarify, we consider routing ossification a correlate and not a cause. Dropout substantially reduces overfitting while also making routing slightly more stable, so routing stability cannot explain the overfitting. We will revise the wording in the paper to clearly emphasize our evidence supports correlation rather than causation.

Reviewer_ro78OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically investigates how data repetition affects the generalization of sparse Mixture-of-Experts (MoE) language models. The authors find that MoE models generally suffer more severe degradation under moderate to high data repetition rates, with the degradation increasing with sparsity. The paper further evaluates several regularization techniques, showing that dropout and output masking can substantially mitigate overfitting.

Critical strengths
  1. The authors conduct extensive experiments across multiple model scales, MoE configurations, repetition rates, data domains, and data quality settings. The consistent trends across these settings provide empirical evidence for the main observations.
  2. The authors systematically evaluate several regularization techniques and demonstrate that dropout and output masking can substantially reduce the degradation caused by data repetition.
Critical weaknesses

The authors conclude that induced repetition degradation primarily depends on total rather than active parameters. While the experiments provide supporting evidence, the effects of total parameters and active parameters are not fully disentangled.

Questions for authors
  1. Could the authors provide additional quantitative evidence showing that total parameter count is a stronger predictor of repetition-induced degradation than active parameter count? 
  2. Could the authors clarify whether increased expert knockout cost necessarily indicates stronger expert specialization, rather than greater dependence on individual experts?
1 comment

We thank reviewer ro78 for the positive review and for noting the extensiveness of our experiments. We address the reviewer's questions and comments below:

Weakness and Q1:Total and active parameters. We will revise the wording in the paper to make our settings and claims clearer: Our work shows that at fixed active parameters and fixed data, the damage from repetition follows total parameters. Our original submission does not include experiments designed to vary active parameters at a fixed total. This is in large part because, unlike total parameters, varying active parameters by definition also alters the compute budget at fixed data and vice versa. Thus, there is no clear candidate setting for a fair comparison when changing active parameters while holding the total fixed. In S3.1 there is currently this text: "we compare the 200M dense models to the 80M MoE (32 x 1/4) and MoE (64 x 1/4) models, which have 158M and 244M total parameters, respectively. In Figure 2, the data repetition effects of the 200M model appear to lie between those of the MoE (32 x 1/4) and MoE (64 x 1/4) models at 80M active parameters."

We agree with the reviewer that it would strengthen our work to make this comparison more direct. We have added a new experiment. We compare the 80M MoE (32 x 1/4) and MoE (64 x 1/4) with 2 dense models, based on the dense 80M models, but where each feed-forward layer is 8 or 16 times wider, respectively, thus matching total parameters. During this short response period, we run R=1, 32 settings. We will add all other R values to the camera ready.

ModelTotalActiveR=1 (Valid Loss)R=32 (Valid Loss)Loss Increase
Dense81.9M81.9M4.9015.247+0.345
Dense, FFN 8x157.8M157.8M4.6465.480+0.834
MoE (32 x 1/4)157.7M80M4.6705.503+0.833
Dense, FFN 16x244.5M244.5M4.5395.927+1.388
MoE (64 x 1/4)244.6M80M4.6056.063+1.457

At 158M total parameters, both dense (158M active) and MoE (80M active) models see an identical R=1 -> R=32 loss degradation of +0.83. Active parameters alone do not correlate with data repetition effects in this setting. Similarly, at 245M total parameters, the dense (245M active) and MoE (80M active) models see a R=1 -> R=32 loss degradation of +1.39 and +1.46, respectively. In other words, at 80M active parameters, increasing total parameters by 55% correlates with loss delta increase by 67-75%. At 158M or 245M total parameters, increasing active parameters by 97-210% correlates with only loss delta decrease by 0-5%. We believe that this result provides some more direct evidence suggesting that data repetition effects may be more directly correlated with total than with active parameters. We will add the full final results with additional R values to the camera ready with appropriate discussion.

Q2: Knockout cost and specialization. We thank the reviewer for raising this question, and would be happy to add discussion to the paper to clarify this, but we are a little unsure of the distinction that the reviewer is making between "dependence" and "specialization". We will define specialization as an expert expressing functions that are not expressed by any other expert. It would follow that an MoE model would have greater dependence on such an expert, and that the two are entangled by design of MoE models. Can the reviewer explain the difference between the concepts? One factor we believe may be relevant is token share. An expert in a 16-expert model handles 8 times more tokens than one in a 128-expert model that is otherwise identical in architecture. To disentangle token share from specialization, we divide knockout by the number of routed tokens to measure the median increase in cross-entropy per token routed to the removed expert. At 80M active parameters:

ModelR=1 (KO/ Tok)R=32 (KO / Tok)Ratio
MoE (16 x 1/4)0.0500.0621.2
MoE (32 x 1/4)0.0400.0581.4
MoE (64 x 1/4)0.0310.0501.6
MoE (128 x 1/4)0.0260.0602.4

The results indicate that knockout divided by token share still increases, though less dramatically. We will update the paper with these metrics. We are happy to add more analyses to disentangle specialization from dependence.

Reviewer_b4mkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how data repetition during LLM pretraining interacts with various aspects of architecture (dense vs MoE of different sparsity, different model size), data mixture (dataset used, whether the whole dataset repeated or a fraction), and training techniques (training duration, dropout, weight decay, gradient clipping, etc). There is also analysis of interaction between MoE expert specialization and repetition.

For some of the studied aspects the interaction with memorization (as measured by test set cross entropy) was observed to be substantial, for example total number of parameters and dropout. While for other aspects the interaction was absent or weak, including router ossification and expert specialization studied in section 5.

The experiments were performed on OLMoE architecture with active parameter sizes 80M, 200M, 1B across the experiments (not every experiment used all 3 sizes), and chinchilla optimal training tokens (20x of active parameters). When repeating data RR times, the total training budget was kept fixed resulting into number of unique tokens scaling with repetitions as R−1R^{-1}. In particular, this resulted in a quite small unique token counts at the onset of repetition related test loss degradation, from 0.1 to 1.125 billion tokens across model sizes at characteristic value R=16R=16, putting the experiments in easy to memorize regime.

Critical strengths

The paper has a broad experimental scope. While it can potentially be a strength in other circumstances, it is rather a weakness for the paper in its current form.

Critical weaknesses

Before going into the weaknesses, I need to describe the structural composition of the paper, as it will be important for the both points below. The paper is organized as a collection of many experiments, each typically equipped with one or a couple plots and one or a couple paragraphs with authors conclusions/interpretations of the experiment. These paragraphs and figures takes majority of main paper space, while appendix is mostly additional figures which are extensively referenced from the respective paragraphs in the main paper. This results into a big volume of research directions/questions which are (i) short (i.e. developed other its respective figure and couple paragraphs) and (ii) rather isolated (i.e. not so much interaction or connection between the threads). In short, the paper looks a lot like an experimental journal/notes. Now to the weaknesses which follow from the above structure.

Narrative and contextualization. It is hard to get any memorable or interesting take away from the paper, as the authors do not connect all their experiments into a coherent investigation united by some question or a problem. It does not mean that the paper does not touch on interesting and important problems - in fact it touches on many - but none of them is developed sufficiently.

A related thing is that the results are not properly contextualized. Both externally with the related literature - there are some discussion in the introduction and related work, but they are not connected to the experiments and conclusions of this work described in sections 3-5 (those section does not have a single sentence comparing or discussing the paper findings with respect to the literature).

Breadth and Depth. Also internal contextualization is missing, in a sense that whenever the authors make observation/conclusion it feels somehow abandoned half way, without proper reflection on it. This results in some conclusions that under at least a little further thought reveal more nuance. Here are some examples I was able to notice.

  • the absence of weight decay impact on memorization (Figure 17) may simply follow from too short training duration for which weight decay cannot mechanically have any impact. The weight decay require the number of steps t≳1LR∗WDt \gtrsim \frac{1}{LR * WD} to have an impact on the weights, which is around 10410^4 steps for the highest used weight decay value WD=0.4WD=0.4 and learning rate (and its schedule) reported in the table 2. Only the largest 1B model has 2×1042\times 10^4 steps (converted from tokens via the reported batch size) for weight decay to have an impact, while smaller models are trained for 40004000 and 16001600 steps that are clearly insufficient. The authors did not report either the direct duration of this experiment of the model size used, while this information is cruicial to correct interpret the results of the weight decay experiment.
  • The last paragraph of section 3.1 claims that the total data budget TT (which is proportional to unique tokens under the paper settings) has weak impact on the overfitting by comparing the shapes of test loss vs. repetition curves for the original and 4x data budgets. When such conclusion is made unconditionally, it raises immediate questions. As the data budget TT (and hence unique tokens) is increased, it is a reasonable to expect the dependence on the repetitions RR will disappear in the limit T→∞T\to\infty, i.e. a 80M model cannot possibly memorize 1 trillion tokens of data even if said data is repeated thousands of times. Such asymptotic regime seems to appear even at practical scales of model training: for example Falcon-H1-tiny-R (0.6B) was trained by repeating around 30 times the same reasoning dataset of around 30 billion tokens without signs of overfitting (see https://huggingface.co/spaces/tiiuae/tiny-h1-blogpost). As mentioned in the paper summary, the presented experiment seem to be in the opposite regime of small unique tokens for which model has the capacity to memorize. Although the precise relation between memorization and unique tokens size is a somewhat open question, making unconditional claims that total or unique tokens do not matter for overfitting seems careless for a paper focused on memorization.
Questions for authors

Which conclusion and its supporting experiments you would single out as the most interesting or consequential from your paper? And what are your thoughts on how you investigation of this aspect can be continued to deeper the understanding or obtain practically useful training recipe?

3 comments

We thank reviewer b4mk for the detailed review, for pointing out that we study interesting and important problems and for raising critical discussion points. We address them below:

1. Weight Decay (WD) and Figure 17. We thank the reviewer for raising this point. We based our design of the WD investigation on related works, but we agree on review that our WD range is too limited to draw firm conclusions from. We have extended our sweep to include weight decay 1.6, 3.2, 6.4, 12.8, and 25.6. WD=3.2 corresponds to roughly 1 decay timescale in our main setting, training 80M active parameter models for 1,526 steps on 1.6B tokens. We previously collected results with WD 0.8 which were not included in the submission. The full WD results for 80M R=1 and R=32:

ModelWDR=1 (Valid Loss)R=32 (Valid Loss)
Dense0.054.9195.306
Dense0.14.9015.247
Dense0.24.9235.279
Dense0.44.9345.268
Dense0.84.9995.280
Dense1.64.8825.194
Dense3.24.8465.096
Dense6.44.8475.023
Dense12.85.0285.132
Dense25.65.5085.544
MoE (64 x 1/4)0.054.6086.071
MoE (64 x 1/4)0.14.6056.063
MoE (64 x 1/4)0.24.6116.009
MoE (64 x 1/4)0.44.6205.965
MoE (64 x 1/4)0.84.6465.924
MoE (64 x 1/4)1.64.5466.127
MoE (64 x 1/4)3.24.5236.127
MoE (64 x 1/4)6.44.5565.968
MoE (64 x 1/4)12.84.8285.340
MoE (64 x 1/4)25.65.3715.433

(* Note that there is a slight change in the pattern at WD 0.8 -> 1.6, as we originally used the same weight decay for norm gains, but at WD 1.6 - 25.6, fixed norm gains decay to 0.1, which is a best practice based on prior work like nanoGPT for cases where WD is higher. This has only a small impact at WD 0.8, but we wanted to note this for full context.)

WD 3.2, 6.4, or 12.8 achieve best performance on all 4 (Dense, MoE) x (R=1, R=32) settings. With optimal WD, the gap between R=1 and R=32 settings for both dense and MoE models is reduced but not closed.

With the best WD for each model, the MoE is 0.32 nats ahead of the dense model at R=1 (4.523 against 4.846) and 0.32 behind at R=32 (5.340 against 5.023), so the gap persists.

2: Data budget and scope of claims (Section 3.1). We agree that the dependence on R would weaken as the unique token set grows relative to the model. Our experimental setup is based on Chinchilla-optimality and comparable to other works like Kim et al. (ICLR 2026). To cover a wider range of data-per-token settings, we have trained 80M scale models on 4x as much data, resulting in a 6.4B tokens or a data-per-token ratio of 80. We show the delta increase in loss over R=1 for 1.6B tokens or 6.4B tokens as appropriate:

RDense, 1.6BDense, 6.4BMoE64, 1.6BMoE64, 6.4B
8+0.026+0.034+0.062+0.067
16+0.081+0.061+0.255+0.218
32+0.345+0.174+1.457+0.927
64+1.668+0.889+5.185+3.937

Indeed, at higher R values, the 4x larger data budget shows less data repetition degradation. We will add language to explicitly define the constraints of our claims to the neighborhood of the data budgets we consider, and will also add this table of results as well as discussion of high data-token-per-parameter regimes.

Please also see point 4 below for additional experiments about fixing total parameters and data budget to vary active parameters.

3: Narrative and Comparison with prior work. We agree that the presentation of our findings would be best contextualized against prior work. We will revise the paper, adding comparisons where available. For example:

  • Muennighoff et al. (2023) find about 4 epochs are safe for dense models. We find about 8 for the 80M dense model, and show evidence that this number may increase with the unique token budget.
  • Xue et al. (2023) attribute multi-epoch damage to parameter count and not FLOPs, from one MoE configuration. Our findings on many more configurations extend this finding.
  • Kazdan et al. (2026) report that damage grows with model scale, which matches our results in Figure 2.
  • Lovelace et al. (2026) report that strong weight decay reduces repetition damage. Our new results agree. Strong weight decay reduces the damage from repetition but does not remove it.

With regards to the reviewer's statement that "It is hard to get any memorable or interesting take away from the paper, as the authors do not connect all their experiments into a coherent investigation united by some question or a problem", our narrative is essentially as follows:

What do we find? – In the popular Chinchilla-optimal regime of data-to-active-parameters, MoEs suffer more from data repetition as they become more sparse. Specifically, under fixed active parameters and data, data repetition damage rises with total parameters, across all diverse data domains and mixtures that we consider.

Is there a remedy? – There is no singular regularization method that appears to entirely remove the correlation between sparsity and data repetition damage, but some methods reduce the damage.

How does data repetition damage performance, and how do the mitigating regularizers work? – Routing mechanisms stabilize early in MoE training, so experts are primarily trained on the same small slice of the unique tokens. Expert specialization or dependence is higher under high data repetition. These are correlated with data repetition damage, not necessarily causal. Our evidence suggests that treating these factors as if they are causal and directly disrupting them is not a complete solution.

Our central issue of data repetition in sparse models has not been thoroughly studied before, so our paper aims to establish a map of the most fundamental questions, setting some groundwork for more thorough study, hence the broad coverage. It seems that our narrative is not coming through clearly, and we would be grateful for any specific actionable suggestions that the reviewer might have, or if the reviewer's stance is that no paper with such broad coverage should be attempted, given the space constraints. We appreciate the opportunity to improve the presentation of our findings.

4: Most important result. We summarized our overall narrative above, so we will not summarize all findings again, but instead isolate one. The individual finding we might point out as most impactful is that the data repetition damage appears to be more strongly correlated with total parameters than with the composition of training data, or with the active parameter count in all settings we study (near Chinchilla tokens-per-parameter, etc.)

Specifically, we cover a wide range of data domains and mixes with very similar results, and even a small slice of heavily repeated data, mixed into a larger set of fully unique data, can still result in significant data repetition damage. To more directly study the impact of active parameters alone, we have added 2 dense models, based on the dense 80M models, but where each feed-forward layer is 8 or 16 times wider, respectively, thus matching total parameters and data to 80M 32x1/4 and 64x1/4.

ModelTotalActiveR=1 (Valid Loss)R=32 (Valid Loss)Loss Increase
Dense81.9M81.9M4.9015.247+0.345
Dense, FFN 8x157.8M157.8M4.6465.480+0.834
MoE (32 x 1/4)157.7M80M4.6705.503+0.833
Dense, FFN 16x244.5M244.5M4.5395.927+1.388
MoE (64 x 1/4)244.6M80M4.6056.063+1.457

Active parameters alone do not correlate with data repetition effects in this 158M total parameter setting. At 245M total parameters, the dense (245M active) and MoE (80M active) models see a R=1 -> R=32 loss degradation of +1.39 and +1.46, respectively. In other words, at 80M active parameters, increasing total parameters by 55% correlates with loss delta increase by 67-75%. At 158M or 245M total parameters, increasing active parameters by 97-210% correlates with only loss delta decrease by 0-5%

Regarding practical guidance, the primary recommendations we would like to highlight: 1) In a setting where data is constrained, even if the limited data forms only a slice of the total mix, more total parameters can be detrimental, even though adding sparsity is typically beneficial. 2) This specifically requires consideration for scaling ladders and extrapolations, in which one or more small scale experiments are used to predict larger scale settings. To build scaling ladders/predictions in settings with any constrained data, the ratio between (unique, total) tokens and (active, total) parameters must all be considered. For example, if the number of unique tokens per total parameters changes between small scale and target settings, the predictions may not hold. 3) The optimal regularizer settings must be specifically tuned for data repetition and sparsity. As an example, the below table illustrates the best settings we find in our 80M scale for each R value.

RBest settingValidation LossSame MoE, no regularizerDense, no regularizer
up to 8MoE, no regularizer
16MoE64 + expert output masking 0.24.7264.8614.982
32MoE64 + expert output masking 0.44.9526.0635.247
64MoE64 + dropout 0.45.6329.7906.570

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.