- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents LinMix, an interpretable, nearly hyperparameter-free, linear approach to optimizing data mixtures using small proxy models. LinMix is shown to have a strong theoretical underpinning as well as to empirically outperform other techniques such as RegMix and OlMix. The resulting mixtures are also shown empirically to be quite interpretable.
Critical strengths
- The paper is very well-written and provides convincing theoretical and empirical support for the efficacy of the technique.
- The technique is very readily adoptable, with very minimal hyperparameter selection needed.
Critical weaknesses
None
Questions for authors
You mention that you set the cap such that “no sequence repeats at the target scale.” Have you done any studies on the effects of higher caps? In practice, in data-constrained regimes, it is commonplace to repeat sequences more than once (e.g., small specialized datasets for high-priority topics or very large models trained for large token horizons, etc.). Can LinMix give new insights into the diminishing returns of repetitions in such settings and how to balance vs. more unique data from other sources?
2 comments
Thank you for your replies.
We sincerely thank the reviewer for their appreciation of our work!
We have not studied the effects of higher caps, which is a natural direction for future work. Instead of treating the cap as another hyperparameter of the setup, we followed the principled choice of Wettig et al. (2025) not to allow data repetition at the target scale. More broadly, prior work on scaling data-constrained language models shows that repeating pretraining data for a few epochs could yield negligible performance degradation (Muennighoff et al., 2025), but its influence on data mixing has not been comprehensively verified. The sensitivity of higher caps has been briefly studied in Olmix (their RQ6 in Section 3.3.4; Chen et al., 2026), which shows differences in the optimized data mixture. In concurrent work, Sedova et al. (2026) find that additional target-domain passes can be optimal when mixing with non-repeated generic data.
We have extended a discussion of this topic in Section 2 (in blue).
We analyze the diminishing returns of sampling tokens from a particular domain, but not for repeating tokens from a particular domain.
We do this by transforming the utility of domain weights π with a root π or a logarithm logπ. This is mathematically represented as curvature r in Equation (5), which models π with r=0.5 and lnπ with r=1. In brief, any value r>0 relaxes domain weights to include all those with positive coefficients of the linear model (Proposition 1), where how much the weights spread increases with r (Corollary 1).
For a concrete example, we visualize the optimized domain weights in App. Figure 10, based on which, we transform the utility on 24 domains:
In particular, there are no such significant differences observed for the more fine-grained 288-domain setup (Fig. 10, right), or the more unique 40-domain setup (Fig. 11), where we expect less diminishing returns and do not need to relax the linear utility π.
Wettig et al. Organize the Web: Constructing Domains Enhances Pre-Training Data Curation. ICML 2025.
Muennighoff et al. Scaling Data-Constrained Language Models. JMLR, 2025.
Chen et al. Olmix: A Framework for Data Mixing Throughout LM Development. ICML 2026.
Sedova et al. Scaling Laws for Mixture Pretraining Under Data Constraints. arXiv:2605.12715, 2026.