- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors focus on sampling uncertainty in the validation gradient, which they contend is an ignored problem in gradient-based data selection for LLMs. They claim that gains over random selection get attenuated by an approximately multiplicative factor (the relative gain retention ρ), and Section 3 provides a closed-form characterization of it. The factor measures how much of the maximal selection gain is retained when aligning with the finite-sample mean validation gradient instead of the population one. It depends on validation set size and noise, but not on training pool size or selection budget, and the attenuation persists even if population influence scores perfectly capture one-step training utility. The analysis assumes isotropic Gaussian training updates and validation gradients, with theoretical extensions to cosine scoring and anisotropic noise in the Appendix. Guided by this analysis, the authors propose SUADS. This method clusters the validation gradients with spherical k-means, admits clusters whose coherence exceeds a threshold, and assigns each training example to the cluster it aligns with best. It then selects the top examples per cluster, with budget proportional to cluster size. Gradients are collected on a warmed-up model as in LESS. The authors note that too many clusters give diminishing returns and can hurt. On code instruction tuning, SUADS has the highest average of the compared methods at a 5% budget (Table 1) and the highest mean across budgets from 5% to 60% on Qwen3.5-4B (Figure 3), with results among the most stable. Random is a strong baseline, as echoed by the authors: on Gemma-2-9B it beats every other targeted method, yet SUADS is the only method above random in every configuration.
Critical strengths
- This paper offers a novel data-selection strategy with formal theorems and explicit error bounds.
- Results in Table 1 show SUADS beating all baselines, including Random, which is the naive baseline that beats all other methods on Gemma 2 9B (MagicCoder).
- SUADS needs one-fourth of LESS’ gradient computation ( just one checkpoint).
Critical weaknesses
-
Results in Table 1 use Magicoder as the training pool, with HumanEval(+) and MBPP(+) for evaluation, on Llama and Gemma, but use a different pool (DS-rStar) and LiveCodeBench on Qwen. It was not apparent why the same data was not used across models, and no model is evaluated in both settings. The Qwen results also come from a single training run (as the authors note in Table 1), with standard deviations over sampled generations rather than training seeds.
-
It is not clear immediately how the method generalizes to dataset selection for fine-tuning outside the coding domain. The title and analysis are domain-general, but all experiments are on code; results on the benchmarks used by LESS (e.g., MMLU, TydiQA, BBH) would round it out in a future effort.
-
The experiments don’t directly corroborate the motivation, i.e., whether reduced sampling uncertainty is actually what drives SUADS’s gains over the other selection methods. Figure 2 shows clusters are more coherent than the global set, but in Table 2 SUADS with a single cluster (51.5) falls below Random (53.6), so the gain appears only with clustering. This could equally reflect better coverage of distinct validation components rather than reduced sampling noise (the authors acknowledge in Appendix B.6/F that the gap lies outside their first-order model).
Questions for authors
Do the authors have results on how much validation data is needed? The paper notes that in-domain validation data is often scarce (100 to 500 examples), and SUADS further splits it into clusters, so each cluster centroid is built from even fewer examples. From Theorem 1, the gain retained depends on the size of the validation set and not on the training set size, so smaller clusters should mean noisier centroids and less gain. But since the paper does not vary the validation set size in its experiments, it is not clear how much SUADS’s advantage over Random and global top-K holds up when validation data is limited, or whether the method simply falls back to global top-K in that case.
How was the cap of four clusters chosen, and was it selected with reference to test-set results? Given that c=3 falls below Random in Table 2, what recommendation do authors provide for a practitioner choosing it for a new task?
5 comments
Weakness 3: The experiments do not directly corroborate the motivation, i.e. whether reduced sampling uncertainty is what drives SUADS's gains. Figure 2 shows clusters are more coherent than the global set, but in Table 2 SUADS with a single cluster (51.5) falls below Random (53.6), so the gain appears only with clustering. This could equally reflect better coverage of distinct validation components rather than reduced sampling noise, and the authors acknowledge in Appendix B.6/F that the gap lies outside the first-order model.
The reviewer's inference from Table 2 is right: the gain appears only with more than one reference. We tested what else the gain needs, on Gemma with the same final-checkpoint features.
- A single reference gives 51.5 to 53.2 whatever its source (global top-K 51.5, a single MBPP reference 52.8, two-source LESS 53.2). All are at or within noise of Random (53.6), none above it.
- Admitting all k-means clusters with the gate off gives 52.7 ± 2.0. The known MBPP/EvoEval partition with quotas and no gate gives 52.0. Both are below Random.
- SUADS, with references for the two coherent sub-clusters the gate admits, gives 55.5 ± 0.5.
So the gain comes from serving the components of the validation set with their own references, and it requires rejecting clusters that are not real components. The sampling-uncertainty analysis motivates that design: references from coherent clusters rather than from single validation examples (a single example retains 0.54 to 0.69, Table 3), and no singletons. By Eq. (94), these coherent Gemma clusters admitted here have high retention of 0.977 to 0.999, predicting the superior performance. The revision says so in Section 3, where Theorem 8 (per-component retention) is presented next to Theorem 1.
Question 1: How much validation data is needed? In-domain validation data is often scarce (100 to 500 examples) and SUADS splits it further, so each cluster centroid is built from fewer examples, and by Theorem 1 smaller clusters should mean noisier centroids and less gain. Does SUADS's advantage over Random and global top-K hold up when validation data is limited, or does the method fall back to global top-K?
We subsampled the validation set to 25 / 50 / 100% (stratified by source, with the same features, budget and recipe, and three training seeds on one fixed subsample per size). The table reports the reliability of the global reference and what the gate admits at each size:
| Validation size | Half-split reliability | Clusters admitted | Global top-K | LESS | Random | SUADS |
|---|---|---|---|---|---|---|
| 135 (25%) | 0.985 | 2 (MBPP 17 + 77) | 52.6 ± 1.7 | 49.4 ± 0.7 | 53.6 ± 1.0 | 55.4 ± 1.5 |
| 269 (50%) | 0.992 | 2 (MBPP 186 + EvoEval 23) | 52.3 ± 0.4 | 48.6 ± 1.4 | 53.6 ± 1.0 | 53.6 ± 1.0 |
| 538 (100%) | 0.996 | 2 (both MBPP) | 51.5 ± 1.0 | 50.2 ± 1.0 | 53.6 ± 1.0 | 55.5 ± 0.5 |
We remark two observations:
- SUADS stays effective with little validation data. With 135 examples the gate still admits two clusters (17 and 77 examples, retention 0.977 and 0.995 by Eq. 94). SUADS is above Random, LESS, and global top-K by a similar margin as with 538 examples (+1.8 / +6.0 / +2.8 vs +1.9 / +5.3 / +4.0).
- At 50% the gate admits MBPP as one cluster of 186 plus an EvoEval cluster of 23, so there is no split inside MBPP. With one MBPP reference and one EvoEval reference, SUADS is still among the best methods, 1.3 above global top-K.
Within the range we can test, a single global reference does not improve with more validation data (from 52.6 with 135 examples to 51.5 with 538). So we do not expect the advantage to close at moderate sizes. Whether it would close at thousands of examples, or whether SUADS degrades at tens, is open.
Question 2: How was the cap of four clusters chosen, and was it selected with reference to test-set results? Given that c = 3 falls below Random in Table 2, what recommendation do the authors give a practitioner choosing it for a new task?
c_max = 4 was fixed a priori, before the Table 2 sweep and without reference to test results. The validation set has two known subtasks, and we set the cap to 4 so that k-means can find structure inside a subtask while the gate rejects clusters that are not coherent. The sweep in Table 2 runs with the gate on. c_max = 1 / 4 / 8 / 12 / 16 admitted c = 1 / 2 / 3 / 6 / 7 clusters and gave 51.5 / 55.5 / 52.7 / 55.1 / 53.9. The gate rejects clusters without real structure: the gate-off variant, which admits the two EvoEval clusters, falls to 52.7. But the gate cannot reject the pieces of a coherent cluster, so the result is sensitive to how finely k-means splits MBPP, as the Limitations section says.
Our recommendation: keep the gate on and use c_max = 4 as the default. It was our a priori choice, we did not tune it, and we used it unchanged on both code pools and on the tool-calling setting. Do not choose c from test results, and read the admitted cluster sizes and coherences that the gate prints at selection time.
We hope these additions address the reviewer's questions, and we are more than happy to take further questions.
We thank the reviewer for the thorough summary and for questions that led to two of the most informative new experiments. Gemma numbers below are mean ± std over three training seeds at the 5% budget. A revised PDF is uploaded, and every change is in blue.
On pools. We matched pool difficulty to model capability. HumanEval/MBPP are at the right difficulty for Llama-3.1-8B and Gemma-2-9B but close to saturation for a recent model such as Qwen3.5-4B-Base. DS_RSTAR consists of long reasoning traces targeting LiveCodeBench, which the two older bases cannot exploit, so we paired it with Qwen3.5-4B-Base. We have not run the cross settings. The revision states this rationale in the experimental setup and does not claim more.
On seeds. We added two training seeds for SUADS and Random at 5% on DS_RSTAR (using paper’s Table 1 protocol, three generations each). The training-seed-level means are SUADS 32.4 / 31.0 / 32.1 = 31.8 ± 0.7 and Random 19.6 / 16.0 / 18.3 = 18.0 ± 1.8. We observe that the training-seed std (0.7 and 1.8) is at or below the generation std of the single runs in Table 1 (1.8 and 2.9), so the generation std we report does not understate the variation. The 13.8-point gap over Random is far outside either training or generation std. The revision reports the seed-level results in the appendix and states this check in the Table 1 caption.
We focused on code because it offers an execution-based metric. To test the method outside code, we added a multi-turn tool-calling setting with the same pipeline.
SUADS is again the only selection method above Random, by 3.9 points. With 5% of the data it reaches 96% of full-pool fine-tuning. Every other targeted method falls below Random (LESS −6.4, BIDS −4.4, ClusterUCB −10.3, GIST −23.8). GIST's score is not an evaluation artifact. All 850 examples it selected are decline rows: 13.7% of the 6,203 decline rows in the pool, and none of the 10,797 rows with a tool call. So the fine-tuned model declines on every test case. GIST's validation subspace is dominated by the decline direction on this validation set, and we report the result as run. Random draws contain decline rows in proportion to the pool (35 to 37%), and SUADS allocates its budget across the admitted clusters in proportion to their size, so no single direction can take the whole budget. The analysis makes no code-specific assumption. This setting is added to the paper. We did not include experiments on MMLU, TydiQA, and BBH because these benchmarks are mostly saturated in our selected model bases.
[1] Xu, Z., Meza Soria, A., Tan, S., Roy, A., Agrawal, A. S., Poovendran, R., and Panda, R. TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments. EMNLP 2026.