- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when multi-source data mixtures can improve scaling laws, rather than merely increasing the effective sample size. The authors develop a high-dimensional mixed-data regression framework, where different domains have different covariance structures, noise levels, and sample growth rates but share a common regression function. The paper first characterizes the minimax risk under ellipsoidal parameter constraints and further derives deterministic equivalents for mixed ridge regression under commuting covariance structures. In the specific setting with power-law covariance spectra, the authors analyze when data mixtures can improve the scaling exponent and show that, under appropriate spectral conditions and sample growth regimes, mixture data can achieve faster scaling rates than either individual source. Finally, the authors conduct synthetic experiments, semi-synthetic regression experiments on pretrained visual features, and small-scale language model experiments to demonstrate phenomena that are qualitatively consistent with the theoretical predictions.
Critical strengths
1.This paper analyses when auxiliary data can genuinely improve the scaling law rather than merely add samples. The authors develop a high-dimensional regression framework for heterogeneous data sources, characterizing the minimax risk under data mixing and deriving an explicit scaling law under power-law covariance spectra. They show that mixing can achieve a better scaling exponent than either source alone when the auxiliary data has a heavier-tailed spectrum and its sample size grows within an appropriate regime relative to the target data—offering a useful theoretical perspective on data synergy in large-scale pretraining. 2.The paper derives a deterministic equivalent for mixed ridge regression under commuting covariances, using it to analyze specific scaling regimes. Experiments on synthetic high-dimensional regression, linear regression on real visual features, and language model training illustrate the predicted phenomena. Notably, the language model experiments offer only qualitative rather than direct validation of the theory, but they enhance the results' heuristic relevance to broader settings.
Critical weaknesses
- A substantial gap remains between the theoretical regression setting and modern neural network training. The explicit scaling-law results use a shared regression function and aligned power-law covariance spectra, while the ridge analysis concerns a specific estimator. Neural representation learning, optimization dynamics, and changes in the regression function across domains are outside this analysis. The language model experiments therefore provide qualitative evidence of analogous data-mixing phenomena, rather than direct validation of the proposed spectral mechanism.
- The scope of the claimed “minimax optimality” of ridge regression requires clarification. Theorem 3 establishes minimax rates over a parameter class, whereas Theorem 4 analyzes ridge regression for a fixed power-law signal. Corollary 1 compares these rates, but it is unclear whether the result establishes a uniform minimax guarantee over the parameter class or only agreement between the fixed-signal ridge exponent and the class minimax exponent. These claims have different theoretical implications.
Questions for authors
- Does Corollary 1 establish that ridge regression achieves the minimax rate uniformly over the parameter class in Theorem 3, or only that its rate for the fixed signal in Theorem 4 matches the class minimax exponent? Please also explain how this comparison accounts for the boundary-membership distinction noted in Section 5. 2.The transformer experiments are presented as qualitative agreement with theory. Could the authors clarify which aspects of the theoretical assumptions (e.g., covariance spectrum, heavy-tailed auxiliary domain) are empirically measured or only used as an analogy? In particular, token-frequency distributions and covariance spectra appear related but are not mathematically equivalent. 3.The LM experiments fit NLL as a linear function of log dataset size over a finite range. Could the authors clarify how this empirical slope should be interpreted relative to the asymptotic scaling exponent derived in the theoretical analysis?
2 comments
Question 2. Qualitative agreement of language experiments with theory.
Indeed, our language experiments provide a qualitative agreement with the theory, rather than a quantitative illustration. In particular, the tasks (generation v.s. regression), learning scheme (self-supervised learning v.s. supervised learning) and the data structures (sequential data v.s. i.i.d. generated data) are different, so we do not expect the theoretical results to hold quantitatively on the language experiments.
The token frequency is not equivalent to the covariates spectrum but, at a high level, we see that it has a similar impact on the scaling laws. In fact, on the one hand, for the regression model, the strength of each component in the spectrum of the covariates of the training distribution captures how much we can learn about the signal in that direction; on the other hand, the frequency of each token captures how much we can learn about that token from the training set. We note that the frequency of tokens does not reflect all the properties that could affect the scaling law, such as the causal relation between tokens, and we leave a more sophisticated modeling to future work.
Question 3. Interpretation of fitting NLL w.r.t. .
Let us clarify that the NLL is the of the perplexity. Thus, our experiments identify the power-law behaviour of the perplexity as a function of , just as our theory identifies the power-law behaviour of the test error as a function of . We clarify this point in l. 429 of the revision.
We thank the reviewer for the useful comments and for the positive evaluation of our work. We respond to all points below and we have uploaded a revision marking the new text in red.
Weakness 1. Gap between theoretical regression setting and modern neural network training.
We agree with the reviewer that there is a gap between our paper and modern neural network training, and we hope to narrow the gap in future work. However, we also emphasize that theoretical results on linear models can still capture key properties of modern neural networks. In particular, many data-related properties tend to be agnostic to the specific model structure. For example, [1] derives a formulation of data mixing laws on a linear model, finding that the law fits the scaling law of LLM training well. In addition, it is common to train a linear head on top of features learned from a pre-trained network, and it turns out that intuitions from linear models well capture properties of modern networks, see e.g. [2].
[1] Rui Dai, Shuran Zheng, Explaining Data Mixing Scaling Laws. ICML 26.
[2] Parham Rezaei, Filip Kovacevic, Francesco Locatello, Marco Mondelli. High-dimensional Analysis of Synthetic Data Selection. ICLR 26.
Weakness 2 and Question 1. Fixed power-law signal.
This is a good point and we addressed it in our reply to Reviewer 9JZn (see Weakness 1 and Question 1. Worst case risk bound on the ellipsoid). For this reviewer’s convenience, we report our answer below as well.
As stated, Corollary 1 compares the ridge rate for the fixed signal θ\*[k]=k−β with the minimax rate over the class. We note that this comparison does give a uniform statement since θ\* reflects the worst case for the ellipsoid. We now explain why this is the case and we have added this discussion in l. 368-395 of the revision.
For any θ∈Θ(R), letting z=C11/2θ and z∗,k=C1/2θ\*, we have
\asymp M^{-2s} \asymp \sup_{\mathbf{\theta}\in\mathbf{\Theta}(R)} \sum_{k=M}^{2M-1}\mathbf{z}_k^2.$$ Thus, on each spectral band $M\le k<2M$, $\mathbf{z}_\*$ has the same energy order as the largest allowed by the ellipsoid. The bias term in the test error sums the contribution of each band and is governed by the dominant band, up to logarithmic factors. Hence, the worst-case bias is obtained by picking a signal in the ellipsoid that concentrates its energy on the worst band, yielding the same bias exponent as $\mathbf{\theta}_\*.$ As the variance does not depend on $\mathbf{\theta}_\*,$ this gives the same polynomial scaling exponent for the test error. As a consequence, the scaling law of the risk w.r.t. to the worst signal on the ellipsoid is the same as in Theorem 3.