acceptodds
Under review as a conference paper at ICLR 2027

Pick the Mixture, Not the Samples: Combinatorial Data Selection for LLMs

Abstract

Data selection for LLM fine-tuning faces two facts about real training: data pools draw from many distinct domains, and evaluation spans diverse tasks. As a result, the same sample helps different tasks differently, and samples interact during training so that a subset’s value is not the sum of its parts. Current data-selection methods leave these realities largely unaddressed. We formalize them as Combinatorial Data Selection (CDS), a framework that separates attribution (how samples are scored) from selection (which subset to pick) and models the non-additive interactions among samples. On a real instruction-tuning data pool, we find substantial cross-domain complementarity (joint utility exceeds the sum of singletons), alongside redundancy within domains. Inspired by this, we propose Mixture-then-Select, a two-stage algorithm that first decides how many samples to take from each domain, then ranks samples within each domain by attribution score. Theoretically, under the stated structural and predictor-accuracy assumptions, the gap to the optimal subset is linear in the selection size and independent of data pool size. Empirically, Mixture-then-Select outperforms strong baselines by 20 to 54% in held-out utility, is robust across attribution methods, and works across different models and data pool sizes. The framework also extends to the data purchasing problem under a budget constraint when sample prices vary. Our findings suggest that allocating selection across domains offers a promising path toward more efficient and effective data selection for modern LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, instruction tuning, data selection, data mixture optimization, sample interactions, budget-constrained optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fuadOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper reconsiders and separates the data selection process for LLM finetuning into two parts: an attribution framework which calculates per-sample scores and a selection framework that identifies the best subset of the data to be picked. The motivation is that, finetuning data often consists of data from a large variety of distinct domains, and how a certain sample interacts with different domains is highly important. The proposed algorithm consists of the two stages directly following the provided motivatino. Stage 1 selects how many samples to be selected from each domain and then Stage 2 selects the top-k samples from each domain using a per-sample score.

Critical strengths
  • Idea and motivation is well constructed and highly important as a research area. Authors provide theoretical justifications supporting their approach with a wide variety of experiments comparing against various baselines. The biggest result that is achieved by the paper is that selecting mixture of samples from multiple domains provides significant improvements compared choosing the 'best' per-sample scoring function that is used for top-k selection. The argument on 'mixture matters more than attribution' in Section 4 further shows the importance of this finding and the motivation behind this work.
Critical weaknesses
  • The authors split the search for the best subset into two problems: (1) choosing the number of samples ncn_c per domain, which accounts for interactions between domains, and (2) scoring each sample and selecting the top n^c\hat n_c samples in each domain. The decomposition of the pairwise utility Φ\Phi into within-domain, cross-domain and residual parts makes sense to me. However, Algorithm 1 is built so that Stage 1 models Φblock\Phi_{\text{block}} and Stage 2 models Φw\Phi_{\text{w}}, and because the two stages are separate, I do not think it captures the paper’s strong motivation. A sample may not be among the top samples in its own domain and still interact better with other domains than a higher-scoring sample from the same domain. When cross-domain interactions affect only the number of samples per domain, and the selection within each domain is independent of Stage 1, the message becomes hard to follow. If the authors argue that cross-domain interactions are highly important, the two stages should be handled jointly in some way to capture these interactions.
  • The claimed size of the gains is not supported by the experimental results (Table 2 and others). For example, the authors state that MtS beats every baseline by 20% to 54%, which Table 2 contradicts. For Qwen3-8B with N=200K, MtS reaches 0.718 and Stratified reaches 0.694, which corresponds to a gain of about 3.5%. The writing also makes many sentences hard to understand on first reading. Some paragraphs read as if they came directly from an AI tool, and several feel more like a summary than an explanation (e.g., the Conclusion). Even though I am familiar with the topic, which is not a hard area to follow, many sentences take extra effort to parse through. In addition, some terms are unclear, such as 'Wave-13 augmentation' (line 1615, page 30) and 'rebuttal-period label-noise experiment' (line 1716, page 32). There are other mistakes, such as the text in Appendix B.8 refers to 18 cells in Table 18, while the caption says 15 combinations and the table lists 19 rows. Overall, the writing, structure and clarity are insufficient throughout the manuscript.
Questions for authors

Building up on my comment to Weakness 1, my biggest question is, whether the two stages of MtS actually capture inter-domain interactions. I like the motivation of this paper and believe it is strong, samples interact with each other during training, thus, a subset's value is not the sum of per-sample scores and we should account for that during selection. I am not convinced, howerver, that the proposed MtS algorithm satisfies this motivation. As it stands out to me, the two stages are mostly decoupled:

  • Stage 1 chooses per-domain n^c\hat n_c.
  • Stage 2 independently takes the top-n^c\hat n_c samples for each domain, which is computed by a fixed per-samples score. Hence, the only information that is passed to Stage 2 is the number of samples to be selected. Stage 2 never uses the information of which samples were selected from other domains.

I think this is also evident in the provided experiments. The manuscript's main takeaway is that 'mixture matters more than attribution' and replacing the oracle scores with a cheap per-sample scores such as EL2N recovers most of the utility. Hence, the answer to data selection is that choosing a good number of samples per domain and then filling each domain using any reasonable per-sample score. This finding, even though interesting, does not necessarily show that modeling sample interactions matter. Hence, I think in its current form, the contributions do not match the stated motivation, even though I consider the motivation compelling.

2 comments
AuthorsOpenReview ↗

Response to Reviewer fuad

We thank the reviewer for recognizing the motivation for interaction-aware selection and the value of separating allocation from attribution. We address W1 together with the related question, followed by W2.

W1 / Q1. Do the two stages capture cross-domain interactions?

We thank the reviewer for stating this concern so clearly. We respectfully clarify that domain counts capture average cross-domain interactions without choosing every sample jointly. MtS uses joint training utilities to learn this allocation before ranking within domains.

  • Why counts carry interaction information. The cross-domain block-mean contribution is ∑i<jφˉijninj\sum_{i \lt j} \bar{\varphi}_{ij} n_i n_j . Domain counts determine this component exactly, so Stage 2 preserves it when filling those counts.
  • How Stage 1 obtains the allocation. In the main implementation (Appendix B.7), its predictor is fitted to measured joint fine-tuning utilities of multi-domain subsets, rather than a sum of separately measured domain utilities. Our response to pwcX (Q2) gives the implemented search procedure.
  • How the reviewer's example enters the analysis. A lower-ranked sample may have a better interaction with particular samples elsewhere. That effect belongs to the sample-specific residual, not the block-mean term. Allocation captures the average effects. Under its assumptions, Theorem 1 bounds the rest through its within-domain, cross-domain residual and higher-order terms.

A 50K pool has approximately 1.25 billion sample pairs, so predicting and optimizing individual pair interactions adds estimation and computational cost that the domain-level allocation avoids.

The experiments support accounting for joint utility when allocating data.

  • Interaction measurements. Singleton and pairwise terms explain 90% of utility variation across approximately 100 measured triplets (Section 4.4). A separate prediction analysis reduces MAE from 0.014 with singleton terms alone to 0.004 after adding pairwise terms.
  • Joint versus additive allocation. In the small bundle-purchasing study, MtS outperforms an additive allocation at all three tested budget levels, with mean utility gains of 0.033, 0.097 and 0.111 (Table 5).
  • Same-score selection controls. With identical oracle scores, global top-mm reaches 0.603 utility, Stratified 0.666 and MtS 0.694 (Table 2). MtS and Stratified also share the within-domain ranking, so they differ only in allocation.
  • Refilling versus keeping the constructed subset. The set-level variant (D1) keeps the subset built by the Stage 1 search instead of refilling its counts. It reaches 0.693 at 1K, against 0.694 for MtS (Table 16), supporting the simpler allocation-plus-ranking design.

With EL2N in place of oracle scores, MtS keeps most of its utility. This does not imply that interactions are irrelevant, since the allocation is still learned from joint fine-tuning outcomes.

Section 3.2 now states which interactions the domain counts capture and which the ranking ignores. Appendix A.12 analyzes an adaptive single-domain alternative that updates pairwise gains after each selection. Extending this refinement to multiple domains at pool scale is future work.

W2. Size of the gains, and writing quality.

We thank the reviewer for the careful reading. We acknowledge these issues and list the corrections below.

The 20-54% claim. The range does not apply to every baseline. As in our response to pwcX (W1), the direct same-score comparisons give 4.2% over Stratified at 50K (0.694 versus 0.666) and 15.1% over global top-mm (0.694 versus 0.603). At 200K, MtS's 0.718 versus Stratified's 0.694 is a 3.5% gain. We have removed the universal range from the abstract and introduction and replaced it with comparator-specific gains in Section 4.2.

Writing and presentation. We rewrote unclear or long sentences and replaced internal labels, mainly in the appendix and Conclusion. The Conclusion now describes the allocation-and-ranking procedure directly. We removed "Wave-13 augmentation" and replaced "rebuttal-period label-noise experiment" with "label-noise experiment". We also corrected the Appendix B.8 heading, text and caption so that they agree with Table 18.

I thank the authors for their response. I had two main concerns before the rebuttal period, first on the performance of the proposed MtS and whether it fulfills the claims behind it, and second on the writing in general. Overall, I still have the same concerns on the proposed MtS, and the writing has not improved at all. In fact, authors' response feels fully AI as well, which was the main concern I had on the readability of the manuscript. The idea and motivation are still well constructed to me, but I do not think the authors do a good job in backing it up, neither in the original manuscript nor during the rebuttal period. Hence, I will be keeping my score of 2 (weak reject). I suggest the authors to provide human written responses in the future.

Reviewer_pwcXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper frames dataset selection for LLM fine-tuning as a combinatorial problem with their algorithm Mixture-then-Select (MtS). The algorithm is split into two stages, stage 1 decides how many samples to take from each pre-specified domain while stage 2 determines which samples to select from individual domains. The paper provides, under stated assumptions, a bound for MtS’ additive utility gap to the optimal subset S* across the data pool. Subset utility is approximated by singleton values plus pairwise interactions, which are further decomposed into 3 components - within domain, cross domain block-mean and residual. Stage 1 addresses cross-domain block-mean implicitly through a learned regression model, while stage 2 is addressed by singleton attribution.

The paper then evaluates MtS in a setting with 8 domains, data pools ranging from 50K to 200K, across 3+ LLMs and a varying budget of 1%-20%. The paper compares MtS against 8 baselines covering per-sample quality, diversity, gradient matching and partition-based sampling. The results in the main paper show MtS outperforming all baselines in the headline settings. Further experiments report downstream benchmark evaluation, ablations on LoRA rank and MtS performance, computational cost-comparison of this method, a data pricing experiment, varying sampling budget, etc. The authors contend that sample-level scores miss interactions and present their framework as modeling pairwise interactions.

Critical strengths
  • Separating data selection from attribution is an interesting perspective. This makes MtS attribution algorithm agnostic, which is a useful property.
  • The paper provides an algorithm with a provable bound on the utility gap to the optimal subset S* under stated assumptions, using pairwise modeling in the analysis and singleton attribution for selection.
  • The paper claims that mixture matters more than attribution which signals the field to focus on domain-aware allocation along with per-sample scoring.
  • Strong set of ablation experiments on the effect of LoRA rank, sampling budget, data pool size (50K vs. 200K), etc. Thank you for including the experiment on data pricing, as it shows that valuing data bundles additively can mispredict joint training utility, an insight relevant to data purchasing under a budget.
Critical weaknesses
  • The paper makes some claims that need to be revisited and explained. For instance in Section 4.2. the authors say that in Panel A MtS beats every baseline by 20% to 54%. Yet, MtS shows only around ~4% (0.694 vs 0.666) increase from stratified sampling and ~15% with oracle-scoring with greedy global top-m (0.603). Both of them use the same oracle-based scoring making them direct contributions.
  • Secondly, assumption 1 (Appendix A.3) - monotonicity and u_t(s) >=0 (Section 3.1) are contradicting the negative utilities seen in Table 2 for the llama 3.2 3 B column. In Table 4, fine-tuning on the full dataset (50K) performs worse than a 1k subset (MtS) of the same 50K which is against the monotonicity assumption that supports Theorem 1. This deserves more discussion.
  • Section 4.2 compares MtS (0.694) against RegMix (0.6566) and DoReMi (0.6272), but these numbers appear nowhere else, and no experimental settings are given. Table 3 lists both methods as downstream rows, also without setup. This comparison is important because both methods share MtS’s within-domain selection rule, so it directly tests Stage 1, the paper’s main contribution. Details such as RegMix’s probe budget relative to MtS’s calibration, DoReMi’s proxy setup, seeds, and variance are needed to evaluate it. Additionally, Table 3 reports MtS’s best setting per benchmark rather than a single configuration, so it is not directly comparable to the other rows.
  • The paper considers computationally intensive baselines like LESS but misses data selection methods for domain adaptation of LLMs, such as DSIR [1], which selects samples via importance resampling and could be applied per domain at a much lower computational cost.

References

[1] Xie, S.M., Santurkar, S., Ma, T. and Liang, P.S., 2023. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36, pp.34201-34227.

Questions for authors
  • Could authors share experiment settings regarding the held-out set against which utility comparisons are made in Table 2. What is the size of this held-out set and is it uniform across domains ? Most importantly, is the held-out set disjoint from the dataset used when computing oracle values and calibration probes ?
  • Can the authors clarify how the argmax for Stage 1 is computed for experiments in the main paper ?
  • Can authors add some context on the results claimed in Section 4.2. on the “comparing with recent mixture methods” paragraph ?
  • What is the intuition for not using pairwise interactions beyond domain averages ? Do the authors plan to include them in future research ?
4 comments
AuthorsOpenReview ↗

W1–W3: Clarifications to setup, theory, and metric

W1. The reported 20-54% improvement range.

We thank the reviewer for pointing out this discrepancy and acknowledge that the 20-54% range does not apply to every baseline in Panel A. The other baselines differ from MtS in both scores and selection rule. On the 50K pool, the gains over kk-DPP and CRAIG are 19.4% and 53.5% (Table 2). These comparisons do not isolate the contribution of domain allocation.

We agree that the same-score comparisons the reviewer identifies are the more direct test. With identical oracle scores, MtS achieves 0.694 utility versus 0.666 for Stratified and 0.603 for global top-mm selection, corresponding to 4.2% and 15.1% relative gains. MtS and Stratified also share the within-domain ranking, so they differ only in allocation.

We have removed the universal percentage range from the abstract and introduction and replaced it with the comparator-specific gains in Section 4.2.

W2. Monotonicity and nonnegative utility versus the empirical results.

We thank the reviewer for identifying this inconsistency. We acknowledge that utility is signed improvement over the starting model. Negative values mean fine-tuning hurt the measured task, and the full-data comparison shows that adding data can hurt. We have defined utility as signed in Section 3.1 and reworded Appendix A.3's "adding training data never hurts" as a conditional assumption rather than an empirical fact.

Monotonicity is not needed for Theorem 1's fixed-size bound. The theorem compares MtS with the best subset of the same size mm. Its proof (Appendix A.11) uses Assumptions 3-6, with the remainder bounded by Theorem 8. No step requires utility to increase when samples are added.

We have removed Assumption 1 from Theorem 1's statement and now state vmax⁡>0v_{\max}>0 explicitly. Monotonicity remains an explicit hypothesis for the budget-saturation arguments in Appendix A.13 and for the pairwise surrogate in the adaptive T=1T=1 result in Appendix A.12.

W3 / Q3. Recent mixture baselines: setup and comparability.

We thank the reviewer for requesting the setup of these comparisons. We respectfully clarify that the Section 4.2 utility comparison holds within-domain ranking fixed to test the allocation decision. MtS, RegMix [2] and DoReMi [1] use Qwen3-8B, the 50K pool, a 1K subset and the same oracle within-domain ranking. Their final training recipe is shared.

The allocation comparison uses the following calibration and search procedures:

MethodCalibration / proxy setupAllocation search
MtS116 labeled subsets: 50 designed probes of 500 samples plus 66 other evaluated subsetsRidge predictor on 11 subset features, greedy construction, then attribution ranking within the resulting domain counts (see Q2)
RegMix [2]The 50 designed probes: eight domain proportions as inputs, utility as target, 40/10 training-validation splitLightGBM regression, 100K candidate mixtures, averaging the best 128 before converting weights to counts
DoReMi [1]Qwen3-0.6B reference and proxy models, fully fine-tuned for 3K sample exposures each, with positive excess-loss reweightingTime-averaged domain weights converted to 1K counts, followed by the shared ranking

RegMix fits on the 50 designed probes, while MtS fits on those probes plus 66 other evaluated subsets. The comparison controls within-domain ranking, but does not equalize calibration data.

The utility means and sample standard deviations, over the same five training seeds as Table 3, are:

MethodUtility meanSeedsSample standard deviation
MtS0.69450.0130
RegMix [2]0.656650.0159
DoReMi [1]0.627250.0116

The revised Appendix B gives these settings, seed counts and standard deviations.

Table 3 compares complete selection procedures, not Stage 1 alone. RegMix and DoReMi obtain their own allocations and sample uniformly within domains, whereas MtS uses attribution ranking. Differences therefore include both allocation and within-domain selection. The downstream comparison that changes only the allocation is MtS versus Stratified (our reply to EQZy, W4). All rows use Qwen3-8B, the 50K pool, a 1K selection budget and five matched training seeds. We acknowledge that the submitted MtS row took the best setting per benchmark, while the baseline rows used one fixed configuration. We have recomputed the MtS row under the same fixed configuration: 0.501 on IFEval, 0.722 on MMLU, 0.803 on GSM8K, 0.575 on ARC-C and 0.795 on HumanEval. We will state this setup in Appendix B.

AuthorsOpenReview ↗

W4 & Questions

W4. Missing low-cost DSIR baseline.

We thank the reviewer for suggesting DSIR [3], a relevant low-cost comparator, and acknowledge that it was missing from our comparison. We evaluated hashed n-gram importance resampling with a domain-balanced target under the same settings as the other Table 3 rows: Qwen3-8B, the 50K pool, a 1K selection budget and five matched training seeds. We have added DSIR to Table 3, and will add its citation and setup to Appendix B.

MethodIFEvalMMLUGSM8KARC-CHumanEval
MtS.501.722.803.575.795
DSIR [3].494.714.785.526.747

MtS scores above DSIR on all five benchmarks.

Q1. Held-out set size, domain balance, and independence.

We thank the reviewer for asking about the held-out set. We respectfully clarify that the utility-test set is disjoint from both the candidate training pool and the data used for oracle attribution and calibration. The selection-validation set contains 20 examples per domain, 160 total. The utility-test set contains 100 examples per domain, 800 total, with equal weighting across the eight domains.

Attribution scores, calibration labels, mixture fitting and hyperparameter selection use only the selection-validation set. Test examples are excluded from all of these steps. MtS, Stratified and global ranking share the same oracle scores, and all methods use the same test set and evaluation procedure. Seed counts vary by comparison. W3/Q3 reports the counts for the recent-mixture utility comparison, and the manuscript tables give the counts for the other comparisons.

Q2. How the Stage 1 argmax is computed.

We thank the reviewer for this question. We respectfully clarify that Algorithm 1 defines the allocation as an argmax over domain counts. In the main 50K experiments we use a greedy heuristic on a learned subset-utility predictor F^(S)\hat F(S) (Appendix B.7), not an exact solver.

  • Predictor. Ridge regression with α=1\alpha=1, selected by five-fold cross-validated R2R^2, on 11 features: eight domain proportions, their entropy, and the mean and standard deviation of attribution scores in the subset.
  • Fitting data. 116 labeled subsets: 50 designed probes of 500 samples and 66 other evaluated subsets, each labeled by its measured joint fine-tuning utility.
  • Search. Starting from the empty set, we repeatedly add the unselected sample maximizing F^(S∪{j})\hat F(S\cup\{j\}) until the selection budget is reached, then extract the resulting domain counts n^\hat{\mathbf n}. No fine-tuning is run during search.
  • Completion. Stage 2 fills each count with the highest-ranked samples in that domain, using the fixed attribution scores.

The search is a heuristic and is not guaranteed to return the global maximizer. Our response to Reviewer EQZy (W6) extends the guarantee to this case through an allocation-regret term. The B.5 pair-interaction predictor is separate and is not used in this allocation procedure.

Q4. Why not optimize pairwise interactions beyond domain averages?

We thank the reviewer for this question. We respectfully clarify that domain averages let MtS account for interactions without estimating and optimizing a separate interaction for every sample pair. A 50K pool contains approximately 1.25 billion pairs. Pair interactions can be predicted from limited measurements, but using these predictions throughout selection adds estimation error and cost. The cross-domain average contribution, by contrast, depends only on the selected domain counts, motivating our mixture-first design.

Sample-specific interactions can still matter. Stage 2 ranks samples by their individual scores rather than updating pairwise marginal gains after each selection. Theorem 1 accounts for this approximation through its within-domain and cross-domain residual terms, under the stated assumptions. We have added this clarification in Section 3.2.

Appendix A.12 gives a single-domain alternative that updates pairwise gains after each selection. Extending it to multi-domain selection at this scale is future work.

References

[1] Xie et al. [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining].

[2] Liu et al. [RegMix: Data Mixture as Regression for Language Model Pre-training].

[3] Xie et al. [Data Selection for Language Models via Importance Resampling].

I thank the authors for making considerable concessions to their claims. This paper had some claims that needed correction and the authors have made their attempts to rectify them. I am keeping my score at this point but do feel that this paper has moved a step closer to acceptance. I leave it to the AC for final merits.

We thank the reviewer for recognizing the improvement in our revised draft and extra experiments, and the expression that this paper has moved a step closer to acceptance.

Reviewer_EQZyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Mixture-then-Select (MtS), a two-stage data selection framework for budget-limited supervised fine-tuning. It first learns a utility predictor from calibration runs to determine how many samples to select from each domain, then fills those allocations using within-domain sample rankings. The paper provides a conditional approximation guarantee based on a pairwise utility decomposition and reports improvements primarily in held-out loss reduction, alongside downstream benchmark results.

Critical strengths
  1. The problem is practically relevant. Studying domain allocation alongside within-domain sample selection addresses a useful question in budget-limited instruction tuning.
  2. The empirical coverage is fairly broad. The paper explores multiple models, data budgets, attribution methods, and domain partitions, including settings where MtS does not outperform Stratified.
Critical weaknesses
  1. The predictor is hard to pin down. Section 3.2 describes a regressor trained on probe utilities, while Appendix A.1 uses the same notation for a pairwise approximation: g^(n)\hat g(\mathbf n) vs. g^(S)=∑j∈Svj+∑{i,j}⊆Sϕij\hat g(S)=\sum_{j\in S}v_j+\sum_{\{i,j\}\subseteq S}\phi_{ij}. How do these relate to the B.5 predictor and B.7 D1 variant? Please specify the main implementation’s model/features, probe construction, and mixture-search procedure.

  2. Held-out loss may not capture the gains that matter. The method optimizes and is mainly evaluated on held-out loss reduction, but it is unclear how reliably this translates into downstream capability gains. Please show whether loss-based utility predicts benchmark performance across mixtures and selection methods, and clarify whether separate validation and test splits are used for selection and final evaluation.

  3. The downstream scores need more context. MtS scores 50.5% on IFEval (instruction-level strict), versus 83.0% / 85.0% for Qwen3-8B in non-thinking / thinking modes (prompt-level strict). Qwen3-8B-Base also reports 76.89% MMLU (5-shot) and 89.84% GSM8K (4-shot CoT), versus MtS’s 72.5% and 80.6%. Different checkpoints and protocols prevent a direct degradation claim, but make the exact checkpoint, evaluation settings, and an identically evaluated pre-SFT baseline important. Qwen3 report, https://arxiv.org/html/2505.09388v1

  4. Table 3 is missing key controls. It omits Stratified and Random, and combines task-specific best settings for MtS. Please report matched multi-seed comparisons with comparable tuning budgets and the same within-domain ranking for MtS and Stratified.

  5. The compute–utility tradeoff needs more discussion. Under Table 4’s reported accounting, MtS uses substantially more compute than Stratified (16.58 vs. 0.58 GPU-hours) for a relatively modest utility gain (0.694 vs. 0.666). How do the authors assess this tradeoff, particularly for a one-off fine-tuning task where calibration cannot be amortized?

  6. The theory assumes more than the calibration establishes. Assumption 6 requires uniform accuracy against the best subset utility at every mixture, sufficient for exact optimal-mixture recovery. As the appendix acknowledges, fitting sampled-subset utilities does not establish this. A guarantee allowing imperfect mixture selection and approximate optimization would better connect the theory to the actual method.

Questions for authors

see weaknesses

7 comments
AuthorsOpenReview ↗

W1. The predictor is hard to pin down.

We thank the reviewer for requesting a clearer implementation description. We respectfully clarify that the main 50K implementation uses a learned subset-utility predictor F^(S)\hat F(S) to construct the domain allocation, then fills that allocation by within-domain attribution ranking. Algorithm 1 defines the allocation as an argmax over domain counts. The search below is a greedy heuristic, not an exact solution of that argmax.

  • Model and features. We use Ridge regression with α=1\alpha=1, selected by five-fold cross-validated R2R^2, to predict subset utility from 11 features: eight domain proportions, their entropy, and the mean and standard deviation of attribution scores in the subset.
  • Probe construction. We fit the allocation predictor using 116 labeled subsets: 50 designed probes of 500 samples and 66 other evaluated subsets. The designed probes include 40 Dirichlet mixtures (20, 10 and 10 with concentrations 1, 0.3 and 3, respectively), eight domain-heavy mixtures and two balanced mixtures, with sampling without replacement within domains. Labels are measured joint fine-tuning utilities.
  • Mixture search. Starting from the empty set, we repeatedly add the unselected sample maximizing F^(S∪{j})\hat F(S\cup\{j\}) until the selection budget is reached. We extract the resulting domain counts for Stage 2. No fine-tuning is run during search.

Stage 2. We select the highest-ranked n^c\hat n_c samples within each domain using the fixed attribution scores.

Relation to A.1, B.5 and B.7. The theory analyzes the pairwise approximation u~(S)\tilde u(S) defined in Section 3.1. Appendix A.1 also labeled it g^(S)\hat g(S), and we have corrected the notation table. B.5's pair-interaction predictor is a separate diagnostic, not the Stage 1 allocation predictor. D1 in B.7 uses the same predictor and greedy construction but retains the constructed subset. MtS instead retains its domain counts and refills them by attribution ranking. The revised Appendix B.7 describes this procedure.

AuthorsOpenReview ↗

W2. Held-out loss may not capture the gains that matter.

We thank the reviewer for asking how loss-based utility relates to downstream performance. We respectfully clarify that held-out loss guides calibration and provides the utility measure, while downstream benchmarks directly assess task performance.

We use held-out loss for two reasons:

  • Established practice. DoReMi [1] uses per-domain excess loss to learn domain weights, while RegMix [2] predicts validation loss from mixture proportions.
  • Finer-grained feedback. Token-level loss measures changes in reference-answer probabilities, providing feedback even when benchmark accuracy is unchanged.

Is utility aligned with benchmark performance? In our experiments, utility and downstream performance agree in the following comparisons against Stratified:

  • Same models, both measures. In Table 22 (Appendix B.9), utility and benchmarks are measured on the same trained models, comparing the MtS variant in that table with Stratified. At both ambiguity levels, the method with higher utility also has the higher four-benchmark mean. At 30% ambiguity, utility rises by 3.1% and the benchmark mean by 2.4%, and both gains hold in all three seeds.
  • MtS versus Stratified. The two share the within-domain ranking and differ only in domain counts. MtS has the higher utility (0.694 versus 0.666, Table 2) and the higher four-benchmark mean in the matched control comparison (W4).

The revised Section 4.1 states that we report downstream benchmarks separately rather than infer task performance from loss reduction (Table 3 and W4).

Validation and test splits.

We partition the data into a candidate training pool, a selection-validation set of 20 examples per domain (160 total), and a disjoint utility-test set of 100 examples per domain (800 total), with duplicate examples removed across splits. Attribution scores, calibration labels, mixture fitting and hyperparameter selection use only the selection-validation set. All probe and final fine-tuning runs train exclusively on samples from the candidate pool. After freezing each method's configuration and selected subset, we measure loss reduction on the utility-test set with equal domain weights and evaluate downstream benchmarks under a common protocol. We will add this split description to Appendix B.

AuthorsOpenReview ↗

W3. The downstream scores need more context.

We thank the reviewer for requesting the checkpoint, evaluation settings and pre-SFT reference. We respectfully clarify the protocol below.

  • Checkpoint. Our Qwen experiments start from the post-trained Qwen/Qwen3-8B, not Qwen3-8B-Base. "Base model" in the submission denotes this starting checkpoint.
  • Evaluation settings. IFEval, MMLU, GSM8K and ARC-Challenge use raw task prompts, without a chat template or thinking mode. IFEval, MMLU and ARC-Challenge are zero-shot, and GSM8K is five-shot. IFEval reports instruction-level strict accuracy. These settings differ from the published Qwen protocols, so the published scores are not directly comparable. The same settings are used for the starting checkpoint and all fine-tuned models. The revised Appendix B gives the full settings for all five benchmarks and the snapshot hash.
  • Pre-SFT reference. We evaluate the unmodified Qwen/Qwen3-8B under the same settings as the fine-tuned models (first row below). The five methods below score above this reference on all five benchmarks.

Table 3 with the pre-SFT reference added. The MtS row is recomputed under the same fixed configuration as the baselines (see W4). The revised Table 3 includes the pre-SFT row.

MethodIFEvalMMLUGSM8KARC-CHumanEval
Pre-SFT Qwen3-8B.392.678.756.521.725
MtS.501.722.803.575.795
RegMix [2].482.714.796.556.770
DoReMi [1].480.715.802.532.762
LESS [3].448.721.793.549.751
DSIR [4].494.714.785.526.747

Comments on the paper

1 comment

Dear Reviewers,

I hope you're doing well.

Please respond to the authors' rebuttal if you have not already done so.

Thank you for your valuable time and engagement.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.