- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper reconsiders and separates the data selection process for LLM finetuning into two parts: an attribution framework which calculates per-sample scores and a selection framework that identifies the best subset of the data to be picked. The motivation is that, finetuning data often consists of data from a large variety of distinct domains, and how a certain sample interacts with different domains is highly important. The proposed algorithm consists of the two stages directly following the provided motivatino. Stage 1 selects how many samples to be selected from each domain and then Stage 2 selects the top-k samples from each domain using a per-sample score.
Critical strengths
- Idea and motivation is well constructed and highly important as a research area. Authors provide theoretical justifications supporting their approach with a wide variety of experiments comparing against various baselines. The biggest result that is achieved by the paper is that selecting mixture of samples from multiple domains provides significant improvements compared choosing the 'best' per-sample scoring function that is used for top-k selection. The argument on 'mixture matters more than attribution' in Section 4 further shows the importance of this finding and the motivation behind this work.
Critical weaknesses
- The authors split the search for the best subset into two problems: (1) choosing the number of samples per domain, which accounts for interactions between domains, and (2) scoring each sample and selecting the top samples in each domain. The decomposition of the pairwise utility into within-domain, cross-domain and residual parts makes sense to me. However, Algorithm 1 is built so that Stage 1 models and Stage 2 models , and because the two stages are separate, I do not think it captures the paper’s strong motivation. A sample may not be among the top samples in its own domain and still interact better with other domains than a higher-scoring sample from the same domain. When cross-domain interactions affect only the number of samples per domain, and the selection within each domain is independent of Stage 1, the message becomes hard to follow. If the authors argue that cross-domain interactions are highly important, the two stages should be handled jointly in some way to capture these interactions.
- The claimed size of the gains is not supported by the experimental results (Table 2 and others). For example, the authors state that MtS beats every baseline by 20% to 54%, which Table 2 contradicts. For Qwen3-8B with N=200K, MtS reaches 0.718 and Stratified reaches 0.694, which corresponds to a gain of about 3.5%. The writing also makes many sentences hard to understand on first reading. Some paragraphs read as if they came directly from an AI tool, and several feel more like a summary than an explanation (e.g., the Conclusion). Even though I am familiar with the topic, which is not a hard area to follow, many sentences take extra effort to parse through. In addition, some terms are unclear, such as 'Wave-13 augmentation' (line 1615, page 30) and 'rebuttal-period label-noise experiment' (line 1716, page 32). There are other mistakes, such as the text in Appendix B.8 refers to 18 cells in Table 18, while the caption says 15 combinations and the table lists 19 rows. Overall, the writing, structure and clarity are insufficient throughout the manuscript.
Questions for authors
Building up on my comment to Weakness 1, my biggest question is, whether the two stages of MtS actually capture inter-domain interactions. I like the motivation of this paper and believe it is strong, samples interact with each other during training, thus, a subset's value is not the sum of per-sample scores and we should account for that during selection. I am not convinced, howerver, that the proposed MtS algorithm satisfies this motivation. As it stands out to me, the two stages are mostly decoupled:
- Stage 1 chooses per-domain .
- Stage 2 independently takes the top- samples for each domain, which is computed by a fixed per-samples score. Hence, the only information that is passed to Stage 2 is the number of samples to be selected. Stage 2 never uses the information of which samples were selected from other domains.
I think this is also evident in the provided experiments. The manuscript's main takeaway is that 'mixture matters more than attribution' and replacing the oracle scores with a cheap per-sample scores such as EL2N recovers most of the utility. Hence, the answer to data selection is that choosing a good number of samples per domain and then filling each domain using any reasonable per-sample score. This finding, even though interesting, does not necessarily show that modeling sample interactions matter. Hence, I think in its current form, the contributions do not match the stated motivation, even though I consider the motivation compelling.
2 comments
I thank the authors for their response. I had two main concerns before the rebuttal period, first on the performance of the proposed MtS and whether it fulfills the claims behind it, and second on the writing in general. Overall, I still have the same concerns on the proposed MtS, and the writing has not improved at all. In fact, authors' response feels fully AI as well, which was the main concern I had on the readability of the manuscript. The idea and motivation are still well constructed to me, but I do not think the authors do a good job in backing it up, neither in the original manuscript nor during the rebuttal period. Hence, I will be keeping my score of 2 (weak reject). I suggest the authors to provide human written responses in the future.
Response to Reviewer fuad
We thank the reviewer for recognizing the motivation for interaction-aware selection and the value of separating allocation from attribution. We address W1 together with the related question, followed by W2.
W1 / Q1. Do the two stages capture cross-domain interactions?
We thank the reviewer for stating this concern so clearly. We respectfully clarify that domain counts capture average cross-domain interactions without choosing every sample jointly. MtS uses joint training utilities to learn this allocation before ranking within domains.
A 50K pool has approximately 1.25 billion sample pairs, so predicting and optimizing individual pair interactions adds estimation and computational cost that the domain-level allocation avoids.
The experiments support accounting for joint utility when allocating data.
With EL2N in place of oracle scores, MtS keeps most of its utility. This does not imply that interactions are irrelevant, since the allocation is still learned from joint fine-tuning outcomes.
Section 3.2 now states which interactions the domain counts capture and which the ranking ignores. Appendix A.12 analyzes an adaptive single-domain alternative that updates pairwise gains after each selection. Extending this refinement to multiple domains at pool scale is future work.
W2. Size of the gains, and writing quality.
We thank the reviewer for the careful reading. We acknowledge these issues and list the corrections below.
The 20-54% claim. The range does not apply to every baseline. As in our response to pwcX (W1), the direct same-score comparisons give 4.2% over Stratified at 50K (0.694 versus 0.666) and 15.1% over global top-m (0.694 versus 0.603). At 200K, MtS's 0.718 versus Stratified's 0.694 is a 3.5% gain. We have removed the universal range from the abstract and introduction and replaced it with comparator-specific gains in Section 4.2.
Writing and presentation. We rewrote unclear or long sentences and replaced internal labels, mainly in the appendix and Conclusion. The Conclusion now describes the allocation-and-ranking procedure directly. We removed "Wave-13 augmentation" and replaced "rebuttal-period label-noise experiment" with "label-noise experiment". We also corrected the Appendix B.8 heading, text and caption so that they agree with Table 18.