acceptodds
Under review as a conference paper at ICLR 2027

When do data mixtures improve scaling laws? Insights from high-dimensional regression.

Abstract

Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an extensive literature on data mixing and reweighting, existing work is largely empirical and it remains unclear when auxiliary data genuinely improves scaling laws rather than merely providing more samples. To gain insight into this question, we study a high-dimensional mixed-data regression model with a shared regression function, heterogeneous covariances and noise levels, and dataset sizes that may grow at different rates. we establish the minimax risk under an ellipsoidal parameter constraint for the general covariances structure and derive deterministic equivalents for the test error of ridge regression under commutative covariances. We then specialize to a target domain and an auxiliary domain with aligned power-law covariance spectra, where the theory yields explicit scaling laws in terms of spectral decay, target regularity, and the relative growth of the two datasets. These laws identify regimes in which combining data mixtures provably yields a faster scaling rate than using either dataset alone. In particular, improving the scaling law requires a specific interplay between spectra and relative sample sizes of the domains. Our numerical experiments on language models exhibit the same qualitative phenomenon: appropriate data mixtures yield a faster decrease in target-domain test loss than training on either domain alone.

Primary area:
learning theory
Keywords:
Date mixtures, High-dimensional regression, Minimax risk, Deterministic equivalence, Scaling laws

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fgJgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when multi-source data mixtures can improve scaling laws, rather than merely increasing the effective sample size. The authors develop a high-dimensional mixed-data regression framework, where different domains have different covariance structures, noise levels, and sample growth rates but share a common regression function. The paper first characterizes the minimax risk under ellipsoidal parameter constraints and further derives deterministic equivalents for mixed ridge regression under commuting covariance structures. In the specific setting with power-law covariance spectra, the authors analyze when data mixtures can improve the scaling exponent and show that, under appropriate spectral conditions and sample growth regimes, mixture data can achieve faster scaling rates than either individual source. Finally, the authors conduct synthetic experiments, semi-synthetic regression experiments on pretrained visual features, and small-scale language model experiments to demonstrate phenomena that are qualitatively consistent with the theoretical predictions.

Critical strengths

1.This paper analyses when auxiliary data can genuinely improve the scaling law rather than merely add samples. The authors develop a high-dimensional regression framework for heterogeneous data sources, characterizing the minimax risk under data mixing and deriving an explicit scaling law under power-law covariance spectra. They show that mixing can achieve a better scaling exponent than either source alone when the auxiliary data has a heavier-tailed spectrum and its sample size grows within an appropriate regime relative to the target data—offering a useful theoretical perspective on data synergy in large-scale pretraining. 2.The paper derives a deterministic equivalent for mixed ridge regression under commuting covariances, using it to analyze specific scaling regimes. Experiments on synthetic high-dimensional regression, linear regression on real visual features, and language model training illustrate the predicted phenomena. Notably, the language model experiments offer only qualitative rather than direct validation of the theory, but they enhance the results' heuristic relevance to broader settings.

Critical weaknesses
  1. A substantial gap remains between the theoretical regression setting and modern neural network training. The explicit scaling-law results use a shared regression function and aligned power-law covariance spectra, while the ridge analysis concerns a specific estimator. Neural representation learning, optimization dynamics, and changes in the regression function across domains are outside this analysis. The language model experiments therefore provide qualitative evidence of analogous data-mixing phenomena, rather than direct validation of the proposed spectral mechanism.
  2. The scope of the claimed “minimax optimality” of ridge regression requires clarification. Theorem 3 establishes minimax rates over a parameter class, whereas Theorem 4 analyzes ridge regression for a fixed power-law signal. Corollary 1 compares these rates, but it is unclear whether the result establishes a uniform minimax guarantee over the parameter class or only agreement between the fixed-signal ridge exponent and the class minimax exponent. These claims have different theoretical implications.
Questions for authors
  1. Does Corollary 1 establish that ridge regression achieves the minimax rate uniformly over the parameter class in Theorem 3, or only that its rate for the fixed signal in Theorem 4 matches the class minimax exponent? Please also explain how this comparison accounts for the boundary-membership distinction noted in Section 5. 2.The transformer experiments are presented as qualitative agreement with theory. Could the authors clarify which aspects of the theoretical assumptions (e.g., covariance spectrum, heavy-tailed auxiliary domain) are empirically measured or only used as an analogy? In particular, token-frequency distributions and covariance spectra appear related but are not mathematically equivalent. 3.The LM experiments fit NLL as a linear function of log dataset size over a finite range. Could the authors clarify how this empirical slope should be interpreted relative to the asymptotic scaling exponent derived in the theoretical analysis?
2 comments

We thank the reviewer for the useful comments and for the positive evaluation of our work. We respond to all points below and we have uploaded a revision marking the new text in red.

Weakness 1. Gap between theoretical regression setting and modern neural network training.

We agree with the reviewer that there is a gap between our paper and modern neural network training, and we hope to narrow the gap in future work. However, we also emphasize that theoretical results on linear models can still capture key properties of modern neural networks. In particular, many data-related properties tend to be agnostic to the specific model structure. For example, [1] derives a formulation of data mixing laws on a linear model, finding that the law fits the scaling law of LLM training well. In addition, it is common to train a linear head on top of features learned from a pre-trained network, and it turns out that intuitions from linear models well capture properties of modern networks, see e.g. [2].

[1] Rui Dai, Shuran Zheng, Explaining Data Mixing Scaling Laws. ICML 26.

[2] Parham Rezaei, Filip Kovacevic, Francesco Locatello, Marco Mondelli. High-dimensional Analysis of Synthetic Data Selection. ICLR 26.

Weakness 2 and Question 1. Fixed power-law signal.

This is a good point and we addressed it in our reply to Reviewer 9JZn (see Weakness 1 and Question 1. Worst case risk bound on the ellipsoid). For this reviewer’s convenience, we report our answer below as well.

As stated, Corollary 1 compares the ridge rate for the fixed signal θ\*[k]=k−β\mathbf{\theta}_\*[k] = k^{-\beta} with the minimax rate over the class. We note that this comparison does give a uniform statement since θ\*\mathbf{\theta}_\* reflects the worst case for the ellipsoid. We now explain why this is the case and we have added this discussion in l. 368-395 of the revision.

For any θ∈Θ(R),\mathbf{\theta} \in \mathbf{\Theta}(R), letting z=C11/2θ\mathbf{z}=\mathbf{C}_1^{1/2} \mathbf{\theta} and z∗,k=C1/2θ\*,\mathbf{z}_{*,k}=\mathbf{C}^{1/2}\mathbf{\theta}_\*, we have

\asymp M^{-2s} \asymp \sup_{\mathbf{\theta}\in\mathbf{\Theta}(R)} \sum_{k=M}^{2M-1}\mathbf{z}_k^2.$$ Thus, on each spectral band $M\le k<2M$, $\mathbf{z}_\*$ has the same energy order as the largest allowed by the ellipsoid. The bias term in the test error sums the contribution of each band and is governed by the dominant band, up to logarithmic factors. Hence, the worst-case bias is obtained by picking a signal in the ellipsoid that concentrates its energy on the worst band, yielding the same bias exponent as $\mathbf{\theta}_\*.$ As the variance does not depend on $\mathbf{\theta}_\*,$ this gives the same polynomial scaling exponent for the test error. As a consequence, the scaling law of the risk w.r.t. to the worst signal on the ellipsoid is the same as in Theorem 3.

Question 2. Qualitative agreement of language experiments with theory.

Indeed, our language experiments provide a qualitative agreement with the theory, rather than a quantitative illustration. In particular, the tasks (generation v.s. regression), learning scheme (self-supervised learning v.s. supervised learning) and the data structures (sequential data v.s. i.i.d. generated data) are different, so we do not expect the theoretical results to hold quantitatively on the language experiments.

The token frequency is not equivalent to the covariates spectrum but, at a high level, we see that it has a similar impact on the scaling laws. In fact, on the one hand, for the regression model, the strength of each component in the spectrum of the covariates of the training distribution captures how much we can learn about the signal in that direction; on the other hand, the frequency of each token captures how much we can learn about that token from the training set. We note that the frequency of tokens does not reflect all the properties that could affect the scaling law, such as the causal relation between tokens, and we leave a more sophisticated modeling to future work.

Question 3. Interpretation of fitting NLL w.r.t. log⁡n\log n.

Let us clarify that the NLL is the log⁡\log of the perplexity. Thus, our experiments identify the power-law behaviour of the perplexity as a function of nn, just as our theory identifies the power-law behaviour of the test error as a function of nn. We clarify this point in l. 429 of the revision.

Reviewer_Q9JxOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper answers when training on an auximilary domain (or a mix of an auxiliarly and target domain) can improve performance on a target domain. The paper takes a theoretical approach, using high-dimensional linear regression, and, under concentration inequalities and a toy model, provides closed form asymptotic behaviors of:

  1. the minimax risk - for general covariance structure, and
  2. the risk of a ridge estimator - for commutative covariance structure From these risk estimators, the paper derives scaling laws for the mimimax and ridge, and show that each law has three regoins of behavior. The size of the regions and the behavior of the risk within them is related to the power law spectra of the toy model, and the relative size of the target and auxiliary domains. This allows the paper to make statments about when single data training can be outperformed by an auxiliary data mix. Specifically, a) when the mixed dataset minimax rate improves on the rate of either dataset, ridge regression is minimax optimal, which happens when (i) the auxiliary spectrum is heavytailed (meaning target and auxiliary datasets cover complementary parts of the spectrum), and (ii) there are sufficiently many auxiliary samples.

The analytical results are supported by experiments on langauge modeling.

Critical strengths

1. The paper does a clear analytical treatment of an important problem in large scale model training that until this work (as far as I am aware) has remained primarily a mix of careful empiricism and heuristics.

This paper lays the groundwork for greater understanding for why one mixture affects the performance of another. Many questions of this type are practical inacessible for empirical study, for example, the case of a large number of mixture components with its exponentially large space of mixings.

Critical weaknesses

1. Shared optimal theta

On line 124 the paper assumes the mixture components share an optimal θ∗\theta^*. We know this isn't true in general - a language model trained on only code will learn a different model to a language model trained on only text. It is not necessary to solve this problem in the paper - it just needs stating, for example, in a limitations section, or preferably up front.

Questions for authors

1. What would happen if you had a different optimal theta for each mixture component?

See weaknesses.

2 comments

We thank the reviewer for the very positive review and the helpful comments. We respond below to the point about having a different θ∗\mathbf \theta_* for each mixture component. We have uploaded a revision marking the new text in red.

In our work, we indeed focus purely on a covariate shift, assuming a shared optimal θ∗\mathbf \theta_*. We note that the general deterministic equivalent results in Theorem 2 can likely be extended to model shift (i.e., different θ∗1,…,θ∗K\mathbf \theta_*^1, \dots, \mathbf \theta_*^K for each mixture component) using a similar approach, at the cost of additional technical complications. This problem is in fact studied in [1, 2] in the regime in which sample sizes and feature dimension grow proportionally. We also expect that the minimax error in Theorem 1 can be extended under further assumptions on θ∗1,…,θ∗K\mathbf \theta_*^1, \dots, \mathbf \theta_*^K, such as e.g. a bound on the maximal difference between the signals. In that regard, when K=2,K=2, [3] characterizes the minimax error for linear estimators when the signals are in a ball and the differences are bounded, under additional assumptions on the empirical covariates. As concerns the scaling law results, it is less clear how to model the discrepancy in θ∗1,…,θ∗K\mathbf \theta_*^1, \dots, \mathbf \theta_*^K so that the model remains mathematically tractable while leading to power laws. We leave this investigation to future work.

We have included this discussion in l. 252-263 (Remark 4.2) of the revision.

[1] Yanke Song, Kenneth Gu, Sohom Bhattacharya, Pragya Sur, Generalization error of min-norm interpolators in transfer learning. https://arxiv.org/abs/2406.13944

[2] Fan Yang, Hongyang R. Zhang, Sen Wu, Christopher Re, Weijie J. Su, Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers. JMLR 2025.

[3] Qi Lei, Wei Hu, Jason D. Lee, Near-Optimal Linear Regression under Distribution Shift. ICML 2021

Thank you for the response and the revision.

Remark 4.2 is what I was looking for: a clear statement of the shared optimal theta assumption up front, and a discussion of extensions for when it may not hold. This fully addresses my weakness and question.

My assessment is unchanged, and I am keeping my Rating at 4. Thank you for a clear and interesting paper.

Reviewer_9JZnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when adding auxiliary data makes target-domain error decay faster, rather than merely lowering it at a given sample size. Its main result is an exact criterion for high-dimensional regression with a shared regression function and aligned power-law covariance spectra. With nn target samples and nγn^\gamma auxiliary samples, the mixture beats both single-domain minimax exponents precisely when δ>1\delta>1 and 1−δ/(1+2s)<γ<11-\delta/(1+2s)<\gamma<1, where δ\delta is the gap between the spectral decay exponents and ss measures target regularity.

Critical strengths

The strongest contribution is that the paper explains both boundaries of this regime. Equation (7) shows that the information available in direction jj scales as n+nγjδn+n^\gamma j^\delta. Too few auxiliary samples leave the target-only rate unchanged; when auxiliary samples are abundant enough, auxiliary-only training already achieves the mixed rate. Between these extremes, the target samples determine the leading directions while the auxiliary samples resolve the tail. Figure 1 illustrates all three cases, rather than showing only the favorable one.

Critical weaknesses
  1. The ridge optimality claim needs a sharper qualification. Theorem 3 controls worst-case risk over an ellipsoid, whereas Theorem 4 analyzes one fixed signal, θj=j−β\theta_j=j^{-\beta}. As the paper notes, this signal is not in the ellipsoid at the boundary regularity ss. Matching the minimax exponent for this signal is informative, but it is not itself a uniform risk bound over the parameter class. Corollary 1’s title obscures that distinction.

  2. The language-model evidence supports a narrower conclusion than the regression theory. Figure 3(a) shows a steeper decline in target loss at γ=1.25\gamma=1.25, outside the theorem’s γ<1\gamma<1 regime. Moreover, Table 1 fits NLL against log⁡n\log n; this measures a finite-range slope, not a power-law exponent. The close-distribution control also deserves clarification: Appendix G.2 says the lack of improvement persists across growth rates, yet at γ=1\gamma=1 the table reports slope gains of 23.5% and 22.4% over the two baselines.

Questions for authors
  1. Can you give a ridge risk bound uniform over the ellipsoid in Theorem 3, or state Corollary 1 explicitly as a fixed-signal exponent comparison?
  2. Does the Figure 3(a) advantage persist when all runs process the same total number of tokens? Reporting uncertainty across seeds for the difference in slopes would also help distinguish a mixing benefit from fitting variability.
2 comments

We thank the reviewer for the insightful comments and for the positive evaluation of our work. We respond to all points below and we have uploaded a revision marking the new text in red.

Weakness 1 and Question 1. Worst case risk bound on the ellipsoid.

Thank you for pointing this out. As stated, Corollary 1 compares the ridge rate for the fixed signal θ\*[k]=k−β\mathbf{\theta}_\*[k] = k^{-\beta} with the minimax rate over the class. We note that this comparison does give a uniform statement since θ\*\mathbf{\theta}_\* reflects the worst case for the ellipsoid. We now explain why this is the case and we have added this discussion in l. 368-395 of the revision.

For any θ∈Θ(R),\mathbf{\theta} \in \mathbf{\Theta}(R), letting z=C11/2θ\mathbf{z}=\mathbf{C}_1^{1/2} \mathbf{\theta} and z\*=C1/2θ\*,\mathbf{z}_\*=\mathbf{C}^{1/2}\mathbf{\theta}_\*, we have

\asymp M^{-2s} \asymp \sup_{\mathbf{\theta}\in\mathbf{\Theta}(R)} \sum_{k=M}^{2M-1}\mathbf{z}_k^2.$$ Thus, on each spectral band $M\le k<2M$, $\mathbf{z}_\*$ has the same energy order as the largest allowed by the ellipsoid. The bias term in the test error sums the contribution of each band and is governed by the dominant band, up to logarithmic factors. Hence, the worst-case bias is obtained by picking a signal in the ellipsoid that concentrates its energy on the worst band, yielding the same bias exponent as $\mathbf{\theta}_\*.$ As the variance does not depend on $\mathbf{\theta}_\*,$ this gives the same polynomial scaling exponent for the test error. As a consequence, the scaling law of the risk w.r.t. to the worst signal on the ellipsoid is the same as in Theorem 3. **Weakness 2. $\gamma = 1.25$ is outside the theorem’s $\gamma < 1$ regime.** We note that the language experiments are meant as a qualitative illustration of the theoretical results, and we do not expect the regime of $\gamma$ where an improvement is possible to match exactly with the theory. We clarify this point in l. 451-452 in the revision. **Weakness 2. Fitting NLL w.r.t. $\log n$.** Let us clarify that the NLL is the $\log$ of the perplexity, so our experiments identify the power-law behaviour of the perplexity as a function of $n$. We mention this explicitly in l. 429 of the revision. **Weakness 2. Inconsistency in Appendix G.2, $\gamma=1$.** Thank you for pointing this out. We have revisited our experiments under this setup for $\gamma_2=1.$ The estimated improvement of the mixture over target-only and auxiliary-only data decreases from $23.5\\%$ and $22.4\\%$ over $n=4,6,8,12,16,20$ Mi to $8.9\%$ and $10.2\%$ over $n=14,16,18,20,22,24$ Mi. For this particular $\gamma_2=1$ configuration, the reduced gains in the fitting window of larger dataset size suggest that finite-size effects may contribute to the larger advantage observed in the original window. We therefore do not regard the original slope estimate for this case as conclusive evidence of a persistent scaling advantage. We have clarified this point in l.3760-3766 of the revision.

Question 2. Persistence of advantage when all runs process the same total number of tokens, and uncertainty on slopes.

Thank you for these two questions. We address both of them below.

For the uncertainty on the slopes, we report the mean and standard deviation across three seeds of the scaling slopes estimated by ordinary least squares (OLS) for the curves in Figure 3(a)-(c). For γ2=0.75\gamma_2=0.75 and γ2=1.75\gamma_2=1.75, the mixture yields only modest increases in the magnitude of the mean slope over the better-performing single-domain baseline: 2.92.9\\% over target-only and 3.13.1\\% over auxiliary-only, respectively. The corresponding absolute slope differences are small relative to the observed seed-to-seed variability, and we conservatively do not regard them as clear evidence of improved scaling over both baselines. In contrast, for γ2=1.25\gamma_2=1.25, the mixture achieves substantially larger gains of 34.534.5\\% and 38.438.4\\% over target-only and auxiliary-only, respectively. Across all settings, the standard deviation of the fitted slopes is less than 77\\% of their mean magnitude, indicating limited variation across the three training seeds.

For the advantage when all runs process the same total number of tokens, we have evaluated the scaling law with respect to the total number of tokens in the setting of Figure 3. For γ2=1.25\gamma_2=1.25, the mixture retains its advantage over both single-domain baselines, with increases in mean slope magnitude of 18.218.2\\% over target-only and 52.252.2\\% over auxiliary-only. The mixture also exhibits steeper mean slopes than both baselines for γ2=0.75\gamma_2=0.75 and γ2=1.75\gamma_2=1.75. The corresponding gains over target-only and auxiliary-only are 15.915.9\\% and 67.867.8\\% for γ2=0.75\gamma_2=0.75, and 13.313.3\\% and 22.922.9\\% for γ2=1.75\gamma_2=1.75, respectively. This difference from the comparison based on nn reflects the rescaling of slopes induced by expressing the learning curves in terms of total training tokens, which depends on the relative sizes and growth rates of nn and n2n_2. Thus, the qualitative finding that mixing can improve scaling over both baselines is preserved over the evaluated ranges, although the regime of γ2\gamma_2 exhibiting this advantage is expected to be different between the two parametrizations.

We have added the above discussion in Appendix G.2 of the revision.

Comments on the paper

1 comment

Dear Reviewers,

The rebuttal period ends in less than two days. I kindly ask you to acknowledge the authors’ responses, and indicate whether your main concerns have been adequately addressed or further clarification is needed.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.