acceptodds
Under review as a conference paper at ICLR 2027

LinMix: Data Mixing as Linear Domain Attribution

Abstract

Language model performance depends strongly on the mixture of domains in the pretraining data corpus. Popular data mixing methods select these domain weights by training many small proxy models on random mixes and then fitting a regression model to predict performance from the mix. We recognize this practice as a domain attribution problem, where the regression measures each domain's contribution to the target metric. Through this lens, the standard Dirichlet sampling of proxy data mixes is a poor regression design that requires tuning and cannot support a linear fit for hundreds of domains. Building on classical optimal experimental design, we introduce LinMix, which samples mixes at the vertices of the capped simplex so that each domain is at its cap or absent. This hyperparameter-free vertex sampling decorrelates domain effects, enabling an interpretable linear model whose coefficients are the domain attributions. The optimal mix given by LinMix provably keeps only domains with positive attribution values and requires no search over candidate mixes. We further establish the optimality of vertex sampling, and prove that Dirichlet sampling inflates the attribution error when its mixes concentrate around the prior data mix. Pretraining experiments with 1B-parameter OLMo2 and 4B Qwen3.5 language models show that LinMix outperforms Dirichlet-based data mixing across reasoning, code, and math benchmarks, with relative gains of 46% on MBPP and 14% on GSM8K. At 288 fine-grained domains, only vertex sampling supports a linear fit, and LinMix improves benchmark accuracy by 11% on average over the best nonlinear baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model pretraining, data mixture, domain weights, attribution

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_a1FjOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents LinMix, an interpretable, nearly hyperparameter-free, linear approach to optimizing data mixtures using small proxy models. LinMix is shown to have a strong theoretical underpinning as well as to empirically outperform other techniques such as RegMix and OlMix. The resulting mixtures are also shown empirically to be quite interpretable.

Critical strengths
  1. The paper is very well-written and provides convincing theoretical and empirical support for the efficacy of the technique.
  2. The technique is very readily adoptable, with very minimal hyperparameter selection needed.
Critical weaknesses

None

Questions for authors

You mention that you set the cap such that “no sequence repeats at the target scale.” Have you done any studies on the effects of higher caps? In practice, in data-constrained regimes, it is commonplace to repeat sequences more than once (e.g., small specialized datasets for high-priority topics or very large models trained for large token horizons, etc.). Can LinMix give new insights into the diminishing returns of repetitions in such settings and how to balance vs. more unique data from other sources?

2 comments

We sincerely thank the reviewer for their appreciation of our work!

Q1. Have you done any studies on the effects of higher caps?

We have not studied the effects of higher caps, which is a natural direction for future work. Instead of treating the cap as another hyperparameter of the setup, we followed the principled choice of Wettig et al. (2025) not to allow data repetition at the target scale. More broadly, prior work on scaling data-constrained language models shows that repeating pretraining data for a few epochs could yield negligible performance degradation (Muennighoff et al., 2025), but its influence on data mixing has not been comprehensively verified. The sensitivity of higher caps has been briefly studied in Olmix (their RQ6 in Section 3.3.4; Chen et al., 2026), which shows differences in the optimized data mixture. In concurrent work, Sedova et al. (2026) find that additional target-domain passes can be optimal when mixing with non-repeated generic data.

We have extended a discussion of this topic in Section 2 (in blue).

Q2. Can LinMix give new insights into the diminishing returns of repetitions in such settings and how to balance vs. more unique data from other sources?

We analyze the diminishing returns of sampling tokens from a particular domain, but not for repeating tokens from a particular domain.

We do this by transforming the utility of domain weights π\pi with a root π\sqrt{\pi} or a logarithm log⁡π\log\pi. This is mathematically represented as curvature rr in Equation (5), which models π\sqrt{\pi} with r=0.5r=0.5 and ln⁡π\ln\pi with r=1r=1. In brief, any value r>0r>0 relaxes domain weights to include all those with positive coefficients of the linear model (Proposition 1), where how much the weights spread increases with rr (Corollary 1).

For a concrete example, we visualize the optimized domain weights in App. Figure 10, based on which, we transform the utility on 24 domains:

  1. with ln⁡π\ln\pi for HellaSwag, which adds Social Life, Fashion & Beauty, Games, and Industrial to the four domains at their caps (Health, Home & Hobbies, Food & Dining, Adult);
  2. and with π\sqrt{\pi} for CPile, which adds a sixth Hardware domain to the data mix.

In particular, there are no such significant differences observed for the more fine-grained 288-domain setup (Fig. 10, right), or the more unique 40-domain setup (Fig. 11), where we expect less diminishing returns and do not need to relax the linear utility π\pi.


Wettig et al. Organize the Web: Constructing Domains Enhances Pre-Training Data Curation. ICML 2025.

Muennighoff et al. Scaling Data-Constrained Language Models. JMLR, 2025.

Chen et al. Olmix: A Framework for Data Mixing Throughout LM Development. ICML 2026.

Sedova et al. Scaling Laws for Mixture Pretraining Under Data Constraints. arXiv:2605.12715, 2026.

Thank you for your replies.

Reviewer_xkk2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues the sub-optimality of the Dirichlet prior choice that is commonly used in data mixture experiments in prior methods, such as RegMix or Olmix. Building on canonical experiment design literature for linear regression, the authors propose to use vertex sampling on the truncated data mixture simplex to train proxy models. Experiments for pretraining Olmo2-1B model and Qwen3.5-4B model show that the resulting optimized mixture consistently enables 0.9%-13.8% relative gains in performance for both training set BPB and held-out benchmark accuracy. Qualitative experiments show that the proposed modification to mixture sampling enabled mixture that are more tailored to different downstream tasks. The paper also theoretically relate mixture modeling to datamodel and prove analytical relationship between estimation error of linear domain attribution and the smallest Eigen value of the design matrix. This proposition then imply optimality of Vertex sampling on the capped simplex, as well as the suboptimality of Dirichlet sampling, under the linear data model assumption.

Critical strengths
  • The paper identified mixture sampling as an important yet often overlooked design choice in prior data mixture modeling via proxy models.
  • The proposed vertex sampling is theoretically sound based on literature on optimal experiment design for linear regression, and is effective in practical pretraining experiments.
  • The comparison baseline methods are representative and state-of-the-art to the best of my knowledge, including Regmix and Olmix. The experiment settings mimics practice involving different pretrained model types and model scales.
Critical weaknesses
  • [Sensitivity to the upsampling factor (\kappa) and domain imbalance] The vertex-sampling procedure in Eq. (3) randomly permutes the per-domain sampling caps and greedily allocates mixture weights until they sum to one. The diversity of the resulting mixtures appears sensitive to the choice of the upsampling factor (\kappa), which controls the feasible mixture space. In particular, when the sum of the normalized per-domain caps is close to one, the feasible region becomes highly constrained, potentially producing very similar sampled mixtures. A similar issue may arise when the data distribution is highly imbalanced, with one domain dominating the corpus. These scenarios could limit the diversity of the sampled mixtures and potentially affect the quality of the fitted surrogate model. While the method performs well under the evaluated configurations, it would be helpful to discuss its sensitivity to (\kappa) and domain imbalance, or ideally provide an ablation examining their effects on mixture diversity and downstream optimization performance.

  • [Theory-practice gap] Despite the empirical success, the linear datamodel assumption appeared to be necessary in establishing the intuitive and theoretical optimality of vertex-based experiment design. The paper devoted little efforts in validating whether and when it hold, especially given that the experiment is non-linear pretraining from scratch. Any discussion is valuable to understand potential gap, or room to further improve on the proposed method.

  • [Mixture domain sensitivity] The paper solely experimented topic partition and topic-format partition of the WebOrganizer corpus. It is thus unclear whether the empirical benefits are likely to generalize on other ways of domain partitioning. Intuitively, the similarity among domains should also affect how well the linear datamodel assumption hold and thus it would be good to have a discussion on potential other ways of splitting domains, such as Source-based or Programming-language-based partitions experimented in Olmix.

Questions for authors
  1. How sensitive are the downstream surrogate-model fit, mixture optimization, and overall performance to the choice of κ\kappa and the potential domain imbalance?

  2. Does the method's success rely on the linear datamodel assumption? If so, when does this assumption approximately hold versus not in practice?

4 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

We sincerely thank the reviewer for their appreciation of our work and the constructive feedback.

On the sensitivity to the upsampling factor (\kappa) and domain imbalance (W1).

In particular, when the sum of the normalized per-domain caps is close to one, the feasible region becomes highly constrained, potentially producing very similar sampled mixtures.

We agree with this geometric observation. Because the sum of the normalized caps equals κ\kappa, as κ→1\kappa \to 1, the feasible region collapses to a single point (the prior mix). Consider that in this limit, no sampler can produce diverse mixtures and no optimizer can deviate from the prior. Rather than being a limitation specific to vertex sampling, this is an inherent property of the feasible set shared by all constrained mixture samplers. In practical pretraining scenarios, such as our three experimental setups where κ∈3.40,5.63,5.98\kappa \in \\{3.40, 5.63, 5.98\\} across 40/288/24 domains, the feasible region remains sufficiently large for vertex sampling to provide substantial diversity gains, larger than Dirichlet.

A similar issue may arise when the data distribution is highly imbalanced, with one domain dominating the corpus. These scenarios could limit the diversity of the sampled mixtures and potentially affect the quality of the fitted surrogate model.

This observation aligns directly with one of the central motivations and contributions of our paper. Specifically, we demonstrate that when the data distribution is highly imbalanced (Fig. 6), standard Dirichlet-sampled mixtures suffer from low diversity due to increasing γ\gamma (Figs. 2 & 9), which degrades the quality of the fitted surrogate model (Figs. 3 & 8). We formalize this phenomenon by modeling the domain distribution as a Zipfian distribution with exponent ss (Figs. 5 & 13), and characterizing how ss drives the failure modes of Dirichlet sampling (Fig. 12). We introduce vertex sampling specifically to overcome this bottleneck, maximizing sample diversity across the feasible region to ensure a high-quality surrogate fit even under severe domain imbalance.

Q1. How sensitive are the downstream surrogate-model fit, mixture optimization, and overall performance to the choice of (\kappa) and the potential domain imbalance?

Great question! We extended a discussion of κ\kappa in Section 2 to address it.

Principled choice of κ\kappa and prior work. Instead of treating κ\kappa as a hyperparameter, it is determined by the ratio of available data tokens to the target model training budget, following WebOrganizer (Wettig et al., 2025), so that no data repeats at the target scale. More broadly, prior work on scaling data-constrained language models shows that repeating pretraining data for a few epochs could yield negligible performance degradation (Muennighoff et al., 2025), but its influence on data mixing has not been comprehensively verified. The sensitivity of κ\kappa has been briefly studied in Olmix (their RQ6 in Sec. 3.3.4; Chen et al., 2026), which shows differences in the optimized data mixture. In concurrent work, Sedova et al. (2026) find that additional target-domain passes can be optimal when mixing with non-repeated generic data. For vertex sampling, increasing κ\kappa increases sparsity of the data mix by containing fewer domains with higher budgets, while decreasing κ\kappa restricts deviations from the prior. As we show in our paper, sparse designs often yield a better downstream surrogate model fit.

Empirical evaluation across varying (κ,s)(\kappa, s) setups. While a comprehensive grid over κ\kappa is computationally prohibitive at pretraining scale, our three setups span a realistic range of up-sampling factors (κ∈[3.40,5.98]\kappa \in [3.40, 5.98]) and Zipfian imbalance exponents (s∈[0.58,0.91]s \in [0.58, 0.91]), which relate to domain imbalances; each with 3–4 target benchmarks:

  1. WebOrganizer topics (195B tokens; 24 domains) exhibit moderate imbalance, s=0.58s = 0.58. Pretraining a 1B model on 32B tokens gives κ=5.98\kappa = 5.98.
  2. WebOrganizer topics ×\times formats (183B; 288 domains) exhibit high imbalance, s=0.86s = 0.86. Pretraining the same model on 32B tokens gives κ=5.63\kappa = 5.63.
  3. A data corpus consisting of WebOrganizer, Nemotron-CC-Math, and Stack-Edu (416B; 40 domains) exhibits the strongest imbalance, s=0.91s = 0.91. We pretrain a 4B model on 80B tokens with a tighter constraint κ=3.40\kappa = 3.40 (initially derived for a 120B-token budget).

Ultimately, across all three setups, from moderate to strong domain imbalance and from looser to tighter κ\kappa constraints, LinMix with vertex sampling yields surrogate fits and downstream performance on par with or better than the Dirichlet-based baselines.


Wettig et al. Organize the Web. ICML 2025.

Muennighoff et al. Scaling Data-Constrained Language Models. JMLR, 2025.

Chen et al. Olmix. ICML 2026.

Sedova et al. Scaling Laws for Mixture Pretraining Under Data Constraints. arXiv:2605.12715, 2026.

AuthorsOpenReview ↗

[2/2] Response by Authors

On the theory-practice gap (W2).

Despite the empirical success, the linear datamodel assumption appeared to be necessary in establishing the intuitive and theoretical optimality of vertex-based experiment design. [...] Any discussion is valuable to understand potential gap, or room to further improve on the proposed method.

Thank you for your valuable feedback, which allowed us to strengthen the discussion of this point in the paper. We agree that our optimality results for vertex sampling (Prop. 3 and Thm. 2) rely on the linear model assumption. Two points narrow the gap:

  1. The theory needs linearity only at the domain level, i.e. E[y∣π]=⟨βˉ,π⟩\mathbb{E}[y \mid \boldsymbol\pi] = \langle \bar{\boldsymbol\beta}, \boldsymbol\pi \rangle (Eq. 7). The sequence-level datamodel of Eq. (6), which prior work supports empirically (Ilyas et al., 2022), is a sufficient condition for it (Prop. 2), not a necessary one.
  2. Domain-level linearity is testable, and we test it. Under vertex sampling, the linear model fits held-out proxy runs on par with nonlinear surrogates at 24, 288, and 40 domains (Figs. 3, 7 & 8), and the linear fit on 30M models ranks the outcomes of 60M and 100M models at ρ≥0.95\rho \geq 0.95 (Fig. 9).

The remaining gap is that in a vertex mix, every domain is either absent or at its cap, so the design cannot measure curvature between these two levels. Optimal designs for second-order mixture models with domain interactions, y=∑kβkπk+∑j<kβjkπjπky = \sum_k \beta_k \pi_k + \sum_{j<k} \beta_{jk} \pi_j \pi_k, add midpoints between pairs of vertices for this purpose (Kiefer, 1961), which is a potential direction for improving LinMix in future work.

Q2. Does the method's success rely on the linear datamodel assumption? If so, when does this assumption approximately hold versus not in practice?

No. While the sample-level linear datamodel assumption provides a theoretical bridge connecting data attribution to domain attribution, allowing us to prove the optimality of vertex-based experimental design, our method’s practical success does not strictly depend on it. Vertex sampling operates at the macro-level of domains, not at the level of individual data points, and can also be used successfully to fit nonlinear surrogates.

Testing the linear datamodel assumption in LLM pretraining is computationally intractable, as it would require fitting attribution models over millions or billions of training sequences using only thousands of proxy runs. However, we could infer its plausibility through three lenses:

  1. Prior empirical support in smaller-scale regimes. Prior work on datamodels (Ilyas et al., 2022 with its subsequent papers) has extensively validated that sample-level linearity holds to a strong approximation in smaller-scale vision and language tasks, in which larger proxy swarms are feasible.
  2. Our empirical evidence from domain-level linearity. We have tested the linear domain model hypothesis in Fig. 9, where a linear domain surrogate achieves good rank correlation across scales on a vertex-sampled design.
  3. Our theoretical contribution. We establish that in the limit as the number of partitions d→nd\to n, the domain model converges to the sample-level datamodel (Cor. 2). Given that the linear domain model is already a strong empirical approximation at d∈24,40,288d\in \\{24,40,288\\}, it is plausible that the underlying datamodel assumption would hold locally as well.

On the mixture domain sensitivity (W3).

The paper solely experimented topic partition and topic-format partition of the WebOrganizer corpus. [...] Intuitively, the similarity among domains should also affect how well the linear datamodel assumption hold and thus it would be good to have a discussion on potential other ways of splitting domains, such as Source-based or Programming-language-based partitions experimented in Olmix.

Kindly note that a programming-language partition is already evaluated in Section 4.4. Beyond the topic (24 domain) and topic-format (288 domain) WebOrganizer setups, our scaling experimental setup explicitly extends the corpus with 15 programming languages sourced from Stack-Edu, alongside the math corpus sourced from Nemotron-CC-Math, for 40 domains in total. We pretrain ten 4B Qwen3.5 language models on 80B tokens targeting math and coding capabilities, where Dirichlet-based data mixing baselines often provide no gains (Tab. 2). These results demonstrate that the benefits of vertex sampling generalize to various partitions better than prior work.

We agree that the similarity among domains can affect how well the linear datamodel assumption holds, which is also related to the potential of domain interactions (cf. our answer to Q2). We added this point to the paper’s discussion.


Thank you again for finding our paper generally understandable, informative, and sound. We would greatly appreciate an increase in rating if you feel we adequately addressed your remaining concerns.

Thanks for the clarifications. Most of my concerns are addressed. However, I have remaining doubts about the sensitivity to the upsampling factor, particularly

"Consider that in this limit, no sampler can produce diverse mixtures and no optimizer can deviate from the prior."

I'd appreciate a more detailed explanation for this. Specifically, what is the scope of the sampler and optimizer? At the minimum, could you explain this using Dirichlet sampling as an helpful example?

Thank you very much for the follow-up, we are glad that most of your concerns are addressed. Let us explain the quoted sentence in more detail.

TL;DR: The up-sampling factor κ\kappa, which is set by the token budget of the target model, allows each domain to take at most κ\kappa times its share in the prior mix. For κ=1\kappa = 1, every domain is capped exactly at its prior share, and because the domain weights have to sum to one, the prior mix is the only mix that satisfies all caps. For a simple example, consider two domains with prior shares of 70% and 30%. With κ=1.1\kappa = 1.1, their caps are 77% and 33%, so every allowed mix lies between 67/33 and 77/23, which is very close to the prior. This holds no matter how the mixes are sampled or optimized, and it is the reason why we wrote that no sampler can produce diverse mixtures in this limit.

More formally, let pjp_j denote the share of domain jj in the prior mix, so that every allowed mix π\pi has πj≤κpj\pi_j \le \kappa p_j for all jj. Then the total variation (TV) distance of any allowed mix to the prior is bounded as

TV(π,p)=12∑j∣πj−pj∣≤1−1κ.\mathrm{TV}(\pi, p) = \tfrac{1}{2} \sum_j |\pi_j - p_j| \le 1 - \tfrac{1}{\kappa}.

To see this, let PP be the prior share of the domains whose weight increases. The caps allow an increase of at most (κ−1)P(\kappa - 1) P, which has to be taken from the remaining domains holding only 1−P1 - P, and the smaller of the two is at most 1−1/κ1 - 1/\kappa. As κ→1\kappa \to 1, the bound goes to zero, so all proxy mixes become identical to the prior and every optimizer has to return the prior mix. For κ>1\kappa > 1 and equal caps, Theorem 2 shows that any design within the caps has an estimation error of at least about d/(κ−1)d/(\kappa - 1) times the optimum without caps, which vertex sampling attains in expectation.

Regarding the scope, the argument uses only the caps, so it applies to any sampler that draws the proxy mixes from the feasible region Pcap\mathcal P_{\text{cap}} and to any optimizer that selects the target mix from it. This includes all methods in our paper: vertex sampling draws its vertices, Dirichlet sampling rejects the mixes that violate the caps (Sec. 4.1, Fig. 2), and all optimizers search within it.

As an example, Dirichlet sampling (Eq. 1) draws each mix as π∼Dirichlet(γaˉ)\pi \sim \mathrm{Dirichlet}(\gamma \bar a) with concentration γ\gamma and tempered prior aˉj∝pj1/τ\bar a_j \propto p_j^{1/\tau}, and rejects it if πj>κpj\pi_j > \kappa p_j for any domain. Decreasing κ\kappa has three effects:

  1. Sparse mixes are rejected. A small γ\gamma puts most weight on a few domains, which exceed their caps.
  2. The concentration has to increase. Each weight varies by about aˉj/γ\sqrt{\bar a_j / \gamma} around its mean aˉj\bar a_j, which has to fit within the slack (κ−1)pj(\kappa - 1) p_j, so γ\gamma grows as 1/(κ−1)21/(\kappa - 1)^2.
  3. The temperature has to decrease. With τ>1\tau > 1, the tempered prior aˉ\bar a itself violates the caps once κ<max⁡jaˉj/pj\kappa < \max_j \bar a_j / p_j, e.g. for κ<2.0\kappa < 2.0 in our 24-domain setup with τ=2\tau = 2.

Thus, Dirichlet sampling stays feasible only by sampling ever closer to the prior, which is the under-dispersion pitfall of Theorem 1.

Example. We simulated the proxy mixes of both samplers under the target model caps for our 24-domain prior (no models trained). The table reports:

  • TV: the above distance averaged over the sampled mixes, with its bound 1−1/κ1 - 1/\kappa in brackets.
  • Error: the mean squared error of the estimated β^\hat\beta (Lemma 1) relative to vertex sampling at κ=5.98\kappa = 5.98, e.g. 48×48\times means that 48 times more proxy runs are needed for the same error.
  • Dirichlet: the share of rejected mixes out of 2×1072 \times 10^7, with τ=2\tau = 2 and γ∈[0.1,10]\gamma \in [0.1, 10] as in our 24-domain experiments.
  • Dirichlet re-tuned: the best γ\gamma and τ∈{1,2}\tau \in \lbrace 1, 2 \rbrace that reject at most 99.9% of the mixes.
κ\kappaVertex: TV (bound)Vertex: errorDirichlet: rejectedDirichlet re-tuned: γ\gamma (τ\tau)Dirichlet re-tuned: TVDirichlet re-tuned: error
5.980.811 (0.833)1.0×1.0\times92.7%10 (2)0.4692.6×2.6\times
30.649 (0.667)2.6×2.6\times99.93%10 (2)0.3907.2×7.2\times
20.487 (0.500)5.1×5.1\timesover 99.9999%30 (2)0.23215×15\times
1.50.325 (0.333)10×10\times100%100 (1)0.13138×38\times
1.10.088 (0.091)48×48\times100%3000 (1)0.026750×750\times

Vertex mixes stay almost at the TV bound for every κ\kappa, and their error grows as 1/(κ−1)1/(\kappa - 1) in line with Theorem 2. Dirichlet sampling as in the paper rejects over 99.9% of the mixes for κ≤3\kappa \le 3, and even after re-tuning, its mixes reach less than 60% of the bound, with 2.6×2.6\times to 16×16\times the error of vertex sampling at the same κ\kappa.

Ultimately, a smaller κ\kappa makes the problem harder for every method, and vertex sampling is affected the least, with no hyperparameters to re-tune. We will add this discussion to the Appendix.

Reviewer_4AdQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes LinMix, it samples mixes at the vertices of the capped simplex and selects the target mix by solving a linear program, which amounts to filling domains to their caps in decreasing order of β̂. The experiments cover OLMo2-1B with 24 and 288 WebOrganizer domains, and Qwen3.5-4B with 40 domains that add code and math. Gains are large at 288 domains (+6–7 accuracy points) and on MBPP and GSM8K at 4B scale.

Critical strengths
  1. Connecting data mixing to optimal experimental design and to datamodels is natural and underexplored.
  2. Theory that matches the experiments qualitatively. The predicted Dirichlet penalty (about 4× at 288 domains and 30× at 40 domains relative to the best capped design) is consistent with Fig. 3 and Fig. 8.
  3. Interpretability and scale transfer. The β̂ coefficients are directly readable (Figs. 4, 10, 11). The linear fit transfers from 30M to 100M proxies (ρ ≥ 0.95) while the Dirichlet fit does not (Fig. 9).
Critical weaknesses
  1. Most of the 288-domain gain seems to come from the optimizer, not the sampler. At 288 domains on HellaSwag, Table 5 shows Dirichlet sampling with the linear model and LP optimizer reaching .597 accuracy, versus .615 for vertex sampling. RegMix and OLMix, both using Dirichlet sampling, reach only .547 and .550. So the sampler accounts for about 2 points, while the surrogate and optimizer account for about 5. This conflicts with the central claim that the design is the main lever.

  2. The pure linear method underperforms at 24 domains, and the fix is chosen after the fact.

Questions for authors
  1. Can you report target-model results for the full sampler × surrogate × optimizer factorial (at least vertex with LightGBM, vertex with exponential, and Dirichlet with linear) across Table 1? How much of the gain is due to vertex sampling alone?

  2. How was r chosen for each setting? Was Table 6, or any other target-model result, used to make that choice? Please report every r for every setting.

3.How do the r > 0 solutions change under shifts of β̂ other than mean-centering? Is there a principled reference point?

2 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

We sincerely thank the reviewer for their time and effort in reviewing our paper.

W1. Most of the 288-domain gain seems to come from the optimizer, not the sampler. At 288 domains on HellaSwag, Table 5 shows [...] So the sampler accounts for about 2 points, while the surrogate and optimizer account for about 5. This conflicts with the central claim that the design is the main lever.

Let us address this concern by restating our two central claims to make their distinction more clear:

  1. Our hyperparameter-free vertex sampling design enables us to fit a reliable linear model (Figure 3), which also transfers better across proxy model sizes (Fig. 9). In comparison, a linear surrogate on Dirichlet-sampled mixes exhibits worse regression fit performance and transfers worse across language model scales. In particular, note that most of the performance of the Dirichlet-sampled design comes from our proposed tuning of its hyperparameters (Fig. 2), since prior work provides no guidance on how to select them.
  2. Vertex sampling together with the linear surrogate and principled optimization (LinMix) achieves state-of-the-art performance across 288 domains (Section 4.3). Consider that with naively chosen hyperparameters, Dirichlet sampling would produce a design matrix in which every mix is close to the prior, and no surrogate–optimizer pair would obtain performance gains from it. In that case, we could have reported a 7-point improvement of LinMix over the baselines on HellaSwag. Finally, LinMix scales well to pretraining a 4B Qwen3.5 model targeting math and coding capabilities (Sec. 4.4), where Dirichlet-based data mixing baselines often provide no gains.

The claim that the vertex design allows for a linear model fit holds across most of the 11 experimental setups we tested (24/40/288 domains times 3–4 benchmarks each), while this concern about transfer to target model performance is specific to a single setup: 288 domains on HellaSwag. In this case, Appendix Table 5 ablates the sampler, showing a 2-point improvement of vertex over Dirichlet, which cannot be mistaken for a weakness of our work.

Q1. Can you report target-model results for the full sampler × surrogate × optimizer factorial (at least vertex with LightGBM, vertex with exponential, and Dirichlet with linear) across Table 1? How much of the gain is due to vertex sampling alone?

Yes, we have trained 8 additional 1B-parameter Olmo2 models and compared their results with those in Table 1 below. It appears as a new ablation Table 7 in Appendix D, with its discussion in lines 320–323 (in blue). It shows that vertex sampling significantly improves Olmix with its exponential surrogate across 288 domains, as well as accounts for more of LinMix's gains for ARC than for HellaSwag. Interestingly, Olmix with our vertex sampling outperforms RegMix.

Surrogate + optimizerSampling24 domains: ARC Acc ↑24 domains: ARC BPB ↓288 domains: ARC Acc ↑288 domains: ARC BPB ↓288 domains: HellaSwag Acc ↑288 domains: HellaSwag BPB ↓
Prior.437.864.436.870.541.788
RegMixDirichlet.496.780.449.850.547.784
Vertex.498.762.477.825.574.776
OlmixDirichlet.486.785.435.872.550.783
Vertex.505.769.484.794.585.768
LinMixDirichlet.500.774.485.814.597.764
Vertex.510.785.511.757.615.758
AuthorsOpenReview ↗

[2/2] Response by Authors

W2. The pure linear method underperforms at 24 domains, and the fix is chosen after the fact.

We believe this sentence to be a misrepresentation of our work. Generally, our LinMix performs on par with baselines at 24 domains as reported in Table 1, while being interpretable (positive coefficients are proportional to domain weights), simpler (e.g. requires no sampling hyperparameters such as γ\gamma or τ\tau), and grounded in data attribution theory (Sec. 5). Also, we do not claim in the paper that Dirichlet sampling is necessarily worse in this setup as observed in the proxy metrics reported in the left panels of Figs. 2 & 3, which transfers to the downstream results shown in Tab. 1. App. Figure 12 best visualizes the fact that when the number of domains dd is small and the prior domain distribution is not skewed (the Zipf exponent ss is small), the Dirichlet-sampled design is appropriate for estimating linear model coefficients.

Let us emphasize that our method substantially outperforms in a larger 40-domain setup, where we pretrain 4B-parameter Qwen3.5 target models on 80B tokens. The improvements reported in Table 2 largely reflect the differences between regression model fits shown in Figure 8, i.e. the smaller differences in the BPB downstream metric for CPile/HellaSwag correspond to their smaller differences in the Spearman proxy metric, and the larger differences in BPB for GSM8K/MBPP (Tab. 2) correspond to their larger differences in Spearman (Fig. 8).

Q2. How was r chosen for each setting? Was Table 6, or any other target-model result, used to make that choice? Please report every r for every setting.

Thank you for this suggestion; to make it clearer, we have added a sentence saying that LinMix uses r=0r=0 unless stated otherwise in line 305 (in blue).

For context, we propose to model the potential diminishing returns of sequences from each domain by transforming the utility of domain weights π\pi with a root π\sqrt{\pi} or a logarithm log⁡π\log\pi. This is mathematically represented as curvature rr in Equation (5), which models π\sqrt{\pi} with r=0.5r=0.5 and ln⁡π\ln\pi with r=1r=1. In brief, any value r>0r>0 relaxes domain weights to include all those with positive coefficients of the linear model (Proposition 1), where how much the weights spread increases with rr (Corollary 1).

r=0r=0 for 9 out of 11 settings. As written in lines 301–304, we visualize the optimized domain weights in App. Figure 10, where relaxing curvature rr retains the full positively attributed support (Prop. 1). Based on this visualization, we transform the utility on 24 domains:

  1. with ln⁡π\ln\pi for HellaSwag, which adds Social Life, Fashion & Beauty, Games, and Industrial to the four domains at their caps (Health, Home & Hobbies, Food & Dining, Adult);
  2. and with π\sqrt{\pi} for CPile, which adds a sixth Hardware domain to the data mix.

Appendix Table 6 is an ablation confirming what is visible in Figure 10 and not a guide to choose rr.


Q3. How do the r > 0 solutions change under shifts of β̂ other than mean-centering? Is there a principled reference point?

This is a great question; we have added this discussion to the paper in lines 202–212 (in blue).

Because the weights of every mix sum to one, adding a constant β0\beta_0 to every β^k\hat\beta_k shifts all predictions by the same amount, i.e. ⟨β^+β01,π⟩=⟨β^,π⟩+β0\langle \hat{\boldsymbol\beta} + \beta_0\mathbf{1}, \boldsymbol\pi \rangle = \langle \hat{\boldsymbol\beta}, \boldsymbol\pi \rangle + \beta_0 for all π∈Δd−1\boldsymbol\pi \in \Delta^{d-1}.

The solution at r=0r = 0 is unchanged, since the optimization in Eq. (4) has the same solution for every constant β0\beta_0.

The solution for r>0r > 0 depends on β0\beta_0 because the shift adds a β0∑kπk1−r−11−r\beta_0 \sum_k \frac{\pi_k^{1-r} - 1}{1-r} term to Eq. (5). In Corollary 1 the weights become min⁡{uk,((β^k+β0)/ω)1/r}\min\lbrace u_k, ((\hat\beta_k + \beta_0)/\omega)^{1/r} \rbrace on the shifted support {k:β^k+β0>0}\lbrace k : \hat\beta_k + \beta_0 > 0 \rbrace. Intuitively, increasing β0\beta_0 enlarges the support and flattens the mix. Decreasing β0\beta_0 shrinks the support toward the top-ranked domains of the r=0r = 0 solution.

Regarding the principled reference point, since the ranking of mixes depends only on the differences between domain attributions β^k\hat\beta_k, we set the baseline level β0\beta_0 to zero, so that positive values of β^k\hat\beta_k are interpreted as ``better than the average domain.'' A natural alternative, which we leave for future work, could be β0=−⟨β^,πprior⟩\beta_0 = -\langle \hat{\boldsymbol\beta}, \boldsymbol\pi_{\text{prior}} \rangle, where a positive β^k\hat\beta_k then means "better than the prior mix."


Thank you again for finding our paper generally understandable, informative, and sound. We would greatly appreciate an increase in rating if you feel we adequately addressed your remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.