acceptodds
Under review as a conference paper at ICLR 2027

SUADS: Sampling-Uncertainty-Aware Data Selection for Large Language Models

Abstract

Gradient-based data selection has shown promise for data-efficient fine-tuning of large language models (LLMs). However, we find that its gains diminish or disappear on challenging coding tasks, sometimes falling below random selection. Our analysis identifies finite-reference sampling uncertainty as a key source of this unreliability: estimating the target gradient direction from a limited reference set introduces noise that scales with the incoherence of the references. Under a stylized model, we prove that this uncertainty attenuates the expected gain over random selection by a multiplicative factor determined by reference reliability, independent of the selection fraction. This degradation occurs even when population influence scores exactly capture one-step training utility, revealing a bottleneck beyond instance-level score accuracy. Guided by this analysis, we propose SUADS, a clustering-based method that combines within-cluster gradient aggregation with stratified selection to exploit more reliable local reference directions. Across Qwen3.5-4B, Llama-3.1-8B, and Gemma-2-9B on MBPP, HumanEval, and LiveCodeBench, SUADS is the only selection method whose average exceeds random selection in every setting and has the highest average pass@1 of any method under matched selection budgets. On Gemma-2-9B it matches full-data fine-tuning on average with 5% of the data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data selection, instruction tuning, large language models, gradient-based influence

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_PsBhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors focus on sampling uncertainty in the validation gradient, which they contend is an ignored problem in gradient-based data selection for LLMs. They claim that gains over random selection get attenuated by an approximately multiplicative factor (the relative gain retention ρ), and Section 3 provides a closed-form characterization of it. The factor measures how much of the maximal selection gain is retained when aligning with the finite-sample mean validation gradient instead of the population one. It depends on validation set size and noise, but not on training pool size or selection budget, and the attenuation persists even if population influence scores perfectly capture one-step training utility. The analysis assumes isotropic Gaussian training updates and validation gradients, with theoretical extensions to cosine scoring and anisotropic noise in the Appendix. Guided by this analysis, the authors propose SUADS. This method clusters the validation gradients with spherical k-means, admits clusters whose coherence exceeds a threshold, and assigns each training example to the cluster it aligns with best. It then selects the top examples per cluster, with budget proportional to cluster size. Gradients are collected on a warmed-up model as in LESS. The authors note that too many clusters give diminishing returns and can hurt. On code instruction tuning, SUADS has the highest average of the compared methods at a 5% budget (Table 1) and the highest mean across budgets from 5% to 60% on Qwen3.5-4B (Figure 3), with results among the most stable. Random is a strong baseline, as echoed by the authors: on Gemma-2-9B it beats every other targeted method, yet SUADS is the only method above random in every configuration.

Critical strengths
  • This paper offers a novel data-selection strategy with formal theorems and explicit error bounds.
  • Results in Table 1 show SUADS beating all baselines, including Random, which is the naive baseline that beats all other methods on Gemma 2 9B (MagicCoder).
  • SUADS needs one-fourth of LESS’ gradient computation ( just one checkpoint).
Critical weaknesses
  • Results in Table 1 use Magicoder as the training pool, with HumanEval(+) and MBPP(+) for evaluation, on Llama and Gemma, but use a different pool (DS-rStar) and LiveCodeBench on Qwen. It was not apparent why the same data was not used across models, and no model is evaluated in both settings. The Qwen results also come from a single training run (as the authors note in Table 1), with standard deviations over sampled generations rather than training seeds.

  • It is not clear immediately how the method generalizes to dataset selection for fine-tuning outside the coding domain. The title and analysis are domain-general, but all experiments are on code; results on the benchmarks used by LESS (e.g., MMLU, TydiQA, BBH) would round it out in a future effort.

  • The experiments don’t directly corroborate the motivation, i.e., whether reduced sampling uncertainty is actually what drives SUADS’s gains over the other selection methods. Figure 2 shows clusters are more coherent than the global set, but in Table 2 SUADS with a single cluster (51.5) falls below Random (53.6), so the gain appears only with clustering. This could equally reflect better coverage of distinct validation components rather than reduced sampling noise (the authors acknowledge in Appendix B.6/F that the gap lies outside their first-order model).

Questions for authors

Do the authors have results on how much validation data is needed? The paper notes that in-domain validation data is often scarce (100 to 500 examples), and SUADS further splits it into clusters, so each cluster centroid is built from even fewer examples. From Theorem 1, the gain retained depends on the size of the validation set and not on the training set size, so smaller clusters should mean noisier centroids and less gain. But since the paper does not vary the validation set size in its experiments, it is not clear how much SUADS’s advantage over Random and global top-K holds up when validation data is limited, or whether the method simply falls back to global top-K in that case.

How was the cap of four clusters chosen, and was it selected with reference to test-set results? Given that c=3 falls below Random in Table 2, what recommendation do authors provide for a practitioner choosing it for a new task?

5 comments

We thank the reviewer for the thorough summary and for questions that led to two of the most informative new experiments. Gemma numbers below are mean ± std over three training seeds at the 5% budget. A revised PDF is uploaded, and every change is in blue.

Weakness 1: Table 1 in the paper uses Magicoder with HumanEval(+)/MBPP(+) on Llama and Gemma but a different pool (DS_RSTAR) and LiveCodeBench on Qwen. It is not apparent why the same data was not used across models, and no model is evaluated in both settings. The Qwen results come from a single training run, with std over sampled generations rather than training seeds.

On pools. We matched pool difficulty to model capability. HumanEval/MBPP are at the right difficulty for Llama-3.1-8B and Gemma-2-9B but close to saturation for a recent model such as Qwen3.5-4B-Base. DS_RSTAR consists of long reasoning traces targeting LiveCodeBench, which the two older bases cannot exploit, so we paired it with Qwen3.5-4B-Base. We have not run the cross settings. The revision states this rationale in the experimental setup and does not claim more.

On seeds. We added two training seeds for SUADS and Random at 5% on DS_RSTAR (using paper’s Table 1 protocol, three generations each). The training-seed-level means are SUADS 32.4 / 31.0 / 32.1 = 31.8 ± 0.7 and Random 19.6 / 16.0 / 18.3 = 18.0 ± 1.8. We observe that the training-seed std (0.7 and 1.8) is at or below the generation std of the single runs in Table 1 (1.8 and 2.9), so the generation std we report does not understate the variation. The 13.8-point gap over Random is far outside either training or generation std. The revision reports the seed-level results in the appendix and states this check in the Table 1 caption.

Weakness 2: It is not clear how the method generalizes to fine-tuning outside the coding domain. The title and analysis are domain-general but all experiments are on code, and results on the benchmarks used by LESS (MMLU, TydiQA, BBH) would round it out.

We focused on code because it offers an execution-based metric. To test the method outside code, we added a multi-turn tool-calling setting with the same pipeline.

  1. Training pool: 17,000 tool-use trajectories sampled from Toucan-1.5M's SFT split [1], with its subset mix preserved and rendered in BFCL's multi-turn format. 36.5% are single-decision rows where the correct answer is to decline without calling a tool. These are the only source of "answer without calling", which BFCL's missing-function and missing-parameter turns require. 38.4% have one call step and 25.1% have two or more. Each method selects 5% (850 examples).
  2. Validation set: 80 of the 800 BFCL v4 multi-turn entries (all four categories). They are held out from evaluation and used only for the reference gradients. This is a split inside the benchmark, since BFCL has no training split.
  3. Evaluation: the remaining 720 entries, scored by BFCL's own checker (an entry passes only if the state-based and response-based checks pass in every turn), reported as the pass rate. Model: Qwen3.5-4B-Base, mean ± std over three seeds.
Method (5% unless stated)BFCL multi-turn
Base model2.5
Random23.8 ± 1.2
LESS17.4 ± 1.3
BIDS19.4 ± 0.7
ClusterUCB13.5 ± 0.9
GIST0.0 (all three seeds, see below)
SUADS27.7 ± 0.6
Full pool (100%)28.9 ± 0.6

SUADS is again the only selection method above Random, by 3.9 points. With 5% of the data it reaches 96% of full-pool fine-tuning. Every other targeted method falls below Random (LESS −6.4, BIDS −4.4, ClusterUCB −10.3, GIST −23.8). GIST's score is not an evaluation artifact. All 850 examples it selected are decline rows: 13.7% of the 6,203 decline rows in the pool, and none of the 10,797 rows with a tool call. So the fine-tuned model declines on every test case. GIST's validation subspace is dominated by the decline direction on this validation set, and we report the result as run. Random draws contain decline rows in proportion to the pool (35 to 37%), and SUADS allocates its budget across the admitted clusters in proportion to their size, so no single direction can take the whole budget. The analysis makes no code-specific assumption. This setting is added to the paper. We did not include experiments on MMLU, TydiQA, and BBH because these benchmarks are mostly saturated in our selected model bases.

[1] Xu, Z., Meza Soria, A., Tan, S., Roy, A., Agrawal, A. S., Poovendran, R., and Panda, R. TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments. EMNLP 2026.

Weakness 3: The experiments do not directly corroborate the motivation, i.e. whether reduced sampling uncertainty is what drives SUADS's gains. Figure 2 shows clusters are more coherent than the global set, but in Table 2 SUADS with a single cluster (51.5) falls below Random (53.6), so the gain appears only with clustering. This could equally reflect better coverage of distinct validation components rather than reduced sampling noise, and the authors acknowledge in Appendix B.6/F that the gap lies outside the first-order model.

The reviewer's inference from Table 2 is right: the gain appears only with more than one reference. We tested what else the gain needs, on Gemma with the same final-checkpoint features.

  1. A single reference gives 51.5 to 53.2 whatever its source (global top-K 51.5, a single MBPP reference 52.8, two-source LESS 53.2). All are at or within noise of Random (53.6), none above it.
  2. Admitting all k-means clusters with the gate off gives 52.7 ± 2.0. The known MBPP/EvoEval partition with quotas and no gate gives 52.0. Both are below Random.
  3. SUADS, with references for the two coherent sub-clusters the gate admits, gives 55.5 ± 0.5.

So the gain comes from serving the components of the validation set with their own references, and it requires rejecting clusters that are not real components. The sampling-uncertainty analysis motivates that design: references from coherent clusters rather than from single validation examples (a single example retains 0.54 to 0.69, Table 3), and no singletons. By Eq. (94), these coherent Gemma clusters admitted here have high retention of 0.977 to 0.999, predicting the superior performance. The revision says so in Section 3, where Theorem 8 (per-component retention) is presented next to Theorem 1.

Question 1: How much validation data is needed? In-domain validation data is often scarce (100 to 500 examples) and SUADS splits it further, so each cluster centroid is built from fewer examples, and by Theorem 1 smaller clusters should mean noisier centroids and less gain. Does SUADS's advantage over Random and global top-K hold up when validation data is limited, or does the method fall back to global top-K?

We subsampled the validation set to 25 / 50 / 100% (stratified by source, with the same features, budget and recipe, and three training seeds on one fixed subsample per size). The table reports the reliability of the global reference and what the gate admits at each size:

Validation sizeHalf-split reliabilityClusters admittedGlobal top-KLESSRandomSUADS
135 (25%)0.9852 (MBPP 17 + 77)52.6 ± 1.749.4 ± 0.753.6 ± 1.055.4 ± 1.5
269 (50%)0.9922 (MBPP 186 + EvoEval 23)52.3 ± 0.448.6 ± 1.453.6 ± 1.053.6 ± 1.0
538 (100%)0.9962 (both MBPP)51.5 ± 1.050.2 ± 1.053.6 ± 1.055.5 ± 0.5

We remark two observations:

  1. SUADS stays effective with little validation data. With 135 examples the gate still admits two clusters (17 and 77 examples, retention 0.977 and 0.995 by Eq. 94). SUADS is above Random, LESS, and global top-K by a similar margin as with 538 examples (+1.8 / +6.0 / +2.8 vs +1.9 / +5.3 / +4.0).
  2. At 50% the gate admits MBPP as one cluster of 186 plus an EvoEval cluster of 23, so there is no split inside MBPP. With one MBPP reference and one EvoEval reference, SUADS is still among the best methods, 1.3 above global top-K.

Within the range we can test, a single global reference does not improve with more validation data (from 52.6 with 135 examples to 51.5 with 538). So we do not expect the advantage to close at moderate sizes. Whether it would close at thousands of examples, or whether SUADS degrades at tens, is open.

Question 2: How was the cap of four clusters chosen, and was it selected with reference to test-set results? Given that c = 3 falls below Random in Table 2, what recommendation do the authors give a practitioner choosing it for a new task?

c_max = 4 was fixed a priori, before the Table 2 sweep and without reference to test results. The validation set has two known subtasks, and we set the cap to 4 so that k-means can find structure inside a subtask while the gate rejects clusters that are not coherent. The sweep in Table 2 runs with the gate on. c_max = 1 / 4 / 8 / 12 / 16 admitted c = 1 / 2 / 3 / 6 / 7 clusters and gave 51.5 / 55.5 / 52.7 / 55.1 / 53.9. The gate rejects clusters without real structure: the gate-off variant, which admits the two EvoEval clusters, falls to 52.7. But the gate cannot reject the pieces of a coherent cluster, so the result is sensitive to how finely k-means splits MBPP, as the Limitations section says.

Our recommendation: keep the gate on and use c_max = 4 as the default. It was our a priori choice, we did not tune it, and we used it unchanged on both code pools and on the tool-calling setting. Do not choose c from test results, and read the admitted cluster sizes and coherences that the gate prints at selection time.

We hope these additions address the reviewer's questions, and we are more than happy to take further questions.

Reviewer_uHBgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why gradient-based data selection methods fail to outperform random sampling in LLM fine-tuning.

Critical strengths

The motivation of this paper is clear. The theoretical analysis give an interesting perspective by showing how uncertainty in validation gradients affects data selection. The proposed method is simple, combining validation-gradient clustering with stratified selection. The experimental results show some improvements.

Critical weaknesses

I think the connection between the theoretical analysis and empirical improvements is not sufficiently clear. Appendix B shows that the global validation-gradient estimation already achieves near-perfect gain retention. So the improvements from clustering can caused by better scopes of different tasks rather than reduced sampling uncertainty.

Questions for authors

Given that the global validation-gradient estimation already has very high gain retention, what is the improvements from clustering? Can the author provide some ablation studies?

1 comment

We thank the reviewer for the clear summary and for a question that goes to the center of the paper. We ran the ablations to answer it. A revised PDF is uploaded, and every change is in blue.

Weakness 1 and Question 1: The connection between the theoretical analysis and the empirical improvements is not sufficiently clear. Appendix B shows that the global validation-gradient estimate already achieves near-perfect gain retention, so the improvement from clustering may come from better scoping of different tasks rather than from reduced sampling uncertainty. Given the high retention of the global estimate, what is the improvement from clustering, and can the authors provide ablation studies?

We agree with the reviewer's reading of Appendix B. An estimated retention of as high as 0.996 means the global reference is an accurate estimate of the global mean, so the clustering gain may not come from averaging out global noise in it. Instead, the ablation experiment below confirms that the SUADS improvement comes from the clustering procedure and from estimating a more precise reference direction for each cluster. We first state the connection between the analysis and the method, then give the ablations.

  1. A cluster reference relies on far fewer examples than the global one, so sampling uncertainty becomes a real risk there. By Eq. (94), a single validation example retains only 0.54 to 0.69 of the achievable gain (Table 3), and a small coherent cluster retains about 0.93 (DS_RSTAR cluster A1, 13 examples, coherence 0.34, Table 4). The risk disappears only when a cluster is large enough and coherent enough: a 17-example cluster at coherence 0.55 retains 0.977, and a 319-example cluster retains 0.999.

  2. SUADS's design keeps cluster references reliable. It builds references from clusters rather than single examples (as GIST and BIDS do), discards singletons, and admits a cluster only when its coherence exceeds what clustering structureless data produces. The gate is a coherence test calibrated against structureless pools. It does not compute Eq. (94).

The gain then comes from giving each coherent component of the validation set its better reference.

We revised the manuscript to state these two roles that really drive SUADS gain in Section 3 and present Theorem 8 (per-component gain retention, Appendix B.6) next to Theorem 1, and we will add a discussion about the limitations of the current numerical retention estimation in Appendix B in explaining SUADS effectiveness.

Ablation experiment (clustering and gate). Setting: Gemma-2-9B, Magicoder pool, 5% budget (1,750 examples), final-checkpoint features, full validation set (538 examples), three training seeds, as in Table 2. Methods:

  • SUADS as in Table 2.
  • The same k-means clustering with the gate switched off, so all four clusters are admitted.
  • The known MBPP/EvoEval partition used in place of the clusters, with SUADS's proportional quota rule and no gate.
  • Global top-K (c = 1), which is SUADS's pipeline with one global reference.
  • LESS with the final checkpoint only, which gives it the features SUADS uses.
  • MBPP-only LESS, which is LESS with the validation set restricted to the 374 MBPP examples (final checkpoint).
  • Random.

Results are average pass@1 over HumanEval, HumanEval+, MBPP and MBPP+, mean ± std over the three seeds.

MethodAvg pass@1
SUADS (k-means + gate, two MBPP sub-clusters of 319 and 55 admitted)55.5 ± 0.5
k-means with c_max = 4, no gate (all four clusters admitted: 319, 55, 119, 45)52.7 ± 2.0
Known MBPP/EvoEval partition, same quotas, no gate52.0 ± 1.3
Global top-K (c = 1)51.5 ± 1.0
LESS, final checkpoint53.2 ± 1.2
MBPP-only LESS, final checkpoint (validation set restricted to the 374 MBPP examples)52.8 ± 1.1
Random53.6 ± 1.0

Two points.

  1. The improvement from clustering is, as the reviewer suggests, that each component of the validation set is served by its own reference. A single reference gives 51.5 to 53.2 whatever its source, being at the same level of Random selection. References for the two coherent MBPP sub-clusters give 55.5.

  2. More references are not automatically better. Admitting all four k-means clusters gives 52.7. This is the same clustering plus the two EvoEval clusters the gate rejects (coherence 0.43 and 0.46, against a threshold of 0.47). The known source partition gives 52.0. Both are below Random. So the gate's rejection of incoherent clusters is necessary.

Varying the validation-set size agrees with this. SUADS's improvement over global top-K selection is +2.8 / +1.3 / +4.0 at 25 / 50 / 100% of the validation set, being consistent and large.

We hope this answers the reviewer's question.

Reviewer_PPVROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SUADS, a clustering-based data selection method that combines within-cluster gradient aggregation with stratified selection. The authors develop a theoretical framework to characterize sampling uncertainty in validation-gradient estimation, which depends on the number of validation samples, noise dimension, and noise magnitude. Motivated by these theoretical findings, the proposed method achieves promising performance across several benchmarks.

Critical strengths

The paper provides a theoretical framework for characterizing sampling uncertainty in validation-gradient estimation. This framework forms the basis of the theoretical analysis in Section 3 and offers insights into the design of SUADS.

Critical weaknesses
    1. The motivation and organization of the paper need substantial improvement. The authors do not clearly explain why sampling uncertainty is an important problem worth investigating. The abstract and introduction contain many abstract concepts that are difficult to follow without reading the entire paper. The presentation would benefit from a clearer explanation of the research problem and its motivation.
    1. The relationship between population validation loss and downstream performance is not sufficiently investigated. The theoretical analysis focuses on sampling uncertainty in validation-gradient estimation, whereas the experimental evaluation uses pass@1 accuracy. The relationship between these objectives remains unclear. Moreover, the paper does not report the validation loss before and after data selection. Consequently, it is difficult to determine whether the improvements achieved by SUADS can be attributed to reduced sampling uncertainty.
Questions for authors
    1. What is the computational cost of SUADS compared with other data selection methods, including the overhead of gradient clustering?
    1. How does the population validation loss relate to downstream task performance, particularly pass@1 accuracy?
3 comments

We thank the reviewer for detailed comments. We address your concerns as below and revised the manuscripts accordingly (changes highlighted in blue). All Gemma numbers are mean ± std over three training seeds at the 5% budget. A revised PDF is uploaded, and every change is in blue.

Weakness 1: The motivation and organization need substantial improvement. The paper does not clearly explain why sampling uncertainty is an important problem worth investigating. The abstract and introduction contain many abstract concepts that are difficult to follow without reading the entire paper. The research problem and its motivation should be explained more clearly.

To improve clarity, here we illustrate the motivation in concrete, plain words.

  1. With a limited fine-tuning budget, we want the 5% of a training pool that helps a target task most. Gradient-based selection scores each training example against a reference gradient direction. The reference is computed from a small validation set, typically a few hundred examples. This is the standard approach to targeted selection (LESS, GIST, BIDS, ClusterUCB).
  2. The weak point is the reference direction, in two ways. First, a reference built from too few examples is a noisy estimate of the direction. Eq. (3) and Theorem 1 give how much of the achievable gain this noise costs as a function of the reference’s size and coherence, and Table 3 measures it: a single validation example, which per-example methods score against, retains only 0.54 to 0.69, while a coherent cluster of a few dozen examples or more retains about 0.97 to 0.99. This is why SUADS builds references from clusters rather than single examples. For the references it admits in our experiments, the loss is small. Second, when the validation set contains several kinds of problems, one averaged direction can represent a component poorly (Theorem 8). On DS_RSTAR, one component has cosine −0.52 with the global reference (Table 4). Selecting against such a reference under-serves that component.
  3. SUADS clusters the validation gradients to obtain one reference per coherent component, and admits a cluster only when its coherence is higher than what clustering structureless data produces. The first consideration shapes how its references are built. The second is where its gain comes from.
  4. Why this matters: in-domain validation data is scarce. Whether a method can turn a few hundred examples into reliable reference directions that serve each kind of problem in the set decides the data selection performance. On Gemma-2-9B, every published gradient-based method in Table 1, at its published configuration, falls even below random downsampling, and SUADS is the only one above it.

In the revision, the abstract is edited to improve clarity. It now introduces in this order: the problem, why a single averaged reference fails, what SUADS does, and what the uncertainty analysis contributes. The introduction adds a short statement of the problem before the technical paragraphs and introduces ρ with its measured values (Table 3).

Weakness 2 and Question 2: The relationship between population validation loss and downstream performance is not sufficiently investigated. The theory concerns sampling uncertainty in validation-gradient estimation while the evaluation uses pass@1, and the relationship between these objectives is unclear. The paper does not report validation loss before and after selection, so it is hard to tell whether SUADS's improvements can be attributed to reduced sampling uncertainty. How does the population validation loss relate to downstream task performance, particularly pass@1 accuracy?

We measured validation loss before and after selection, and we read the relationship between validation loss and pass@1 from the same table. The measurement is the per-token loss on the 538-example validation set under the training template (the selection reference set, which is not used for evaluation), for the base model, Random, LESS, SUADS and the new ablations (Gemma-2-9B, 5%, three training seeds each):

ModelValidation lossAvg pass@1
Base model (before fine-tuning)0.421n/a
Random0.461 ± 0.00553.6
LESS0.383 ± 0.00250.2
SUADS0.395 ± 0.00155.5
Other gradient-based variants (global top-K, known partition, no gate, MBPP-only LESS at four checkpoints and at the final checkpoint, single-checkpoint LESS)0.384 to 0.40051.3 to 53.2

Every gradient-based selection lowers validation loss relative to the base model by similar amounts (0.38 to 0.40 vs 0.42). Random selection raises it (0.46). So the first-order objective these methods optimize is achieved. We observe that a smaller validation loss goes with better downstream performance in some comparisons, for example SUADS has a smaller loss than Random and a higher pass@1, but not in general, since LESS has the smallest loss and the lowest pass@1. Downstream performance such as pass@1 is the more reliable capability metric, because validation loss measures the fit to one reference solution per question and so carries uncertainty of its own. On attribution, the table answers this: SUADS does not reduce validation loss more than the other gradient-based methods (LESS reduces it more), so its gains cannot be attributed to a larger reduction of validation loss on a particular dataset, but more like a more generalizable performance improvement.

On the relationship between population validation loss and downstream performance, we have three points.

  1. Among the gradient-based methods, validation loss does not predict pass@1. The two with the lowest loss, LESS (0.383) and MBPP-only LESS (0.384), have the lowest pass@1 (50.2 and 51.3), while SUADS at 0.395 has the highest (55.5). Across all gradient-based models we trained, the rank correlation between loss and pass@1 is slightly positive (+0.21).
  2. We believe the reason is: validation loss measures the fit to one reference solution per problem, and fitting that reference style more closely is not the same as solving more problems. On the other hand, pass@1 under execution measures the latter, which is why we evaluate with it. Moreover, as an aggregated likelihood of generating any feasible solution, we think pass@1 is itself a strong proxy of “generalizable validation loss of problem solutions”.
  3. What separates the methods is which parts of the validation set the reference directions cover (see the validation-size and partition experiments in our response to qv4i). SUADS's stratified selection gives up a little global alignment to cover each admitted component. Validation loss, averaged over the whole set, does not reflect this difference, but pass@1 does. We revised the manuscript accordingly to reflect the discussion.

Question 1: What is the computational cost of SUADS compared with other data selection methods, including the overhead of gradient clustering?

Measured on one GPU for Gemma-2-9B, 35k-example pool, 538 validation examples, 5% budget (ClusterUCB's column is the method's nominal cost, see the notes below):

Stage (GPU-hours)RandomSUADSLESSGISTClusterUCB
Warmup LoRA (shared)00.740.740.740.74
Training-pool gradients04.216.713.48.3
Validation gradients00.040.180.070.18
Selection012 s20 s149 s6 s
Fine-tuning on the selected 5%0.830.830.830.830.83
Total0.835.818.515.110.1

We highlight that SUADS is the most efficient method among gradient-based data selection methods. Collecting training-pool gradients is the dominant cost for every gradient-based method: 84% of SUADS's overhead over Random, 95% of LESS's. The clustering and null-pool calibration take 12 seconds on a 538 × 8192 matrix. Measured as overhead over Random, SUADS costs only 28% of LESS as published (one checkpoint instead of four), 35% of GIST (random projections instead of full-dimension gradients), and 54% of ClusterUCB as specified by its authors. LESS restricted to the final checkpoint, which we report to Reviewer qv4i (53.2 vs 50.2 with four checkpoints), costs the same as SUADS (5.8 GPU-hours). This table is added to the appendix.

We hope the response, revision, and additional experiments address the reviewer's concerns, and we would be grateful for a re-evaluation.

Reviewer_qv4iOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how uncertainty in validation gradients affects data selection for supervised fine-tuning. Under a Gaussian, first-order model, it characterizes how reference noise reduces selection gains over random sampling. The authors propose SUADS, which clusters validation gradients, retains coherent clusters, and selects training examples using proportional cluster quotas. Across three models on coding tasks, SUADS reports the highest average pass@1 among selection methods in the main comparisons. On Gemma-2-9B, it achieves 55.5%55.5\% average pass@1 using 5%5\% of the training examples, compared with 55.2%55.2\% for full-data training.

Critical strengths

1. The authors check the theory against real gradients. Appendix B.6 compares predictions from Equation (94) with leave-one-out and half-split cosines. All but one comparison agrees within 0.010.01. This calibration helps assess where the model applies.

2. The component analysis has a concrete example. Theorem 8 relates component-level first-order gain to reference alignment under its model. DS_RSTAR cluster A1 has cosine −0.52-0.52 with the empirical global reference (Table 4), illustrating how an average direction can poorly represent a component.

Critical weaknesses

1. I do not see how the analysis explains the main gain. Section 3 motivates clustering through reference uncertainty. Yet Appendix B.6 estimates full global-reference retention at 0.9990.999 on Magicoder and 0.9920.992 on DS_RSTAR under its model. The global reference is already highly reliable by this measure. The appendix proposes coverage as an explanation and acknowledges that the analysis does not explain the clustering gain in Table 2.

Theorem 2 also allows several clusters to select the same example and counts each selection. SUADS instead assigns each candidate to one cluster before selection. The theorem therefore does not directly characterize the implemented method's gain.

2. The comparisons change more than clustering. LESS uses four warmup checkpoints; SUADS uses only the last. In the main Gemma configuration, both admitted reference clusters contain only MBPP examples, whereas LESS references both MBPP and EvoEval. EvoEval still influences SUADS's clustering and threshold calibration (Appendix E.3).

On DS_RSTAR, median selected-example lengths are 20,640 tokens for random selection, 2,756 for SUADS, and 1,176 for LESS (Figure 6). Being less biased toward short examples than other targeted methods does not rule out length as an explanation. I want to see what SUADS adds after controlling the checkpoint, reference source, and example length.

Questions for authors

Clarification

How was cmax⁡=4c_{\max}=4 chosen, and was it fixed before the Table 2 sweep?

Experiments that could change my assessment

1. What drives the gain? When you subsample the reference set, for example to 25%25\%, 50%50\%, and 100%100\% of MM, how does SUADS's advantage over global selection change? Please hold source proportions, gradient features, and selection budget fixed, and report reference reliability using Table 3's half-split cosine alongside downstream performance. A gap that shrinks as reference reliability improves would support the noise explanation; a persistent gap would be consistent with coverage.

2. What do clustering and the coherence gate add? Using the same final-checkpoint features on Gemma, how does SUADS compare with the known MBPP/EvoEval partition, without the gate and with the same quota rule, and with an MBPP-only LESS baseline?

3. Does the gain survive length matching? On DS_RSTAR, does SUADS outperform random selection matched to its example-length distribution at the same example budget? Please report processed training tokens and the truncation policy. A shortest-5%5\% baseline would also test whether a simple length criterion explains the improvement.

7 comments

We thank the reviewer for the careful reading and for three experiments with a clear statement of what each outcome would mean. We ran all three as specified, plus the checkpoint control. All new Gemma methods use the Table 2 configuration: Gemma-2-9B, 5% budget (1,750 examples), mean ± std over three training seeds on final-checkpoint features, except the four-checkpoint LESS rows marked. The length controls are on DS_RSTAR, as the reviewer specified. A revised PDF is uploaded, and every change is in blue.

Weakness 1-a and Question 1: The analysis does not explain the main gain. Section 3 motivates clustering through reference uncertainty, yet Appendix B.6 estimates the full global reference's retention at 0.999 on Magicoder and 0.992 on DS_RSTAR, so the global reference is already highly reliable by the paper's own measure. The appendix proposes coverage as the explanation and acknowledges that the analysis does not explain the clustering gain in Table 2. When the validation set is subsampled to 25%, 50% and 100% with source proportions, gradient features and budget fixed, how does SUADS's advantage over global selection change, with reliability measured by Table 3's half-split cosine? A gap that shrinks as reliability improves would support the noise explanation, and a persistent gap would be consistent with coverage.

We answer Weakness 1-a here and Question 1 with Experiment 1 below. We agree with the reviewer's reading of B.6. A retention of 0.999 means the global reference is an accurate estimate of the global mean. So the gain over global top-K does not come from reducing noise in the global reference. Where the uncertainty in Theorem 1 does matter is at the cluster level, for two reasons.

  1. A cluster reference relies on far fewer examples than the global one, so sampling uncertainty becomes a real risk there. By Eq. (94), a single validation example retains only 0.54 to 0.69 of the achievable gain (Table 3), and a small coherent cluster retains about 0.93 (DS_RSTAR cluster A1, 13 examples, coherence 0.34, Table 4). The risk disappears only when a cluster is large enough and coherent enough: a 17-example cluster at coherence 0.55 retains 0.977, and a 319-example cluster retains 0.999.

  2. SUADS's design keeps cluster references reliable. It builds references from clusters rather than single examples (as GIST and BIDS do), discards singletons, and admits a cluster only when its coherence exceeds what clustering structureless data produces. The gate is a coherence test calibrated against structureless pools. It does not compute Eq. (94).

The gain itself comes from giving each coherent component of the validation set its better reference, which Experiment 2 (Question 2 below) tests. We revise in Section 3 and the abstract to reflect this: the analysis motivates building references from coherent clusters rather than single examples, and clustering with the coherence gate decides which components get one.

Experiment 1 (validation-set size). Setting: Gemma-2-9B, Magicoder pool, 5% budget (1,750 examples), final-checkpoint features, three training seeds, as in Table 2. The validation set is subsampled to 25 / 50 / 100% (MBPP:EvoEval = 94:41, 187:82, 374:164), stratified by source, with features and budget fixed. Global top-K is SUADS's pipeline with one global reference (c = 1), serving as a baseline. Reliability is Table 3's half-split cosine of the global reference. Results are average pass@1 over HumanEval, HumanEval+, MBPP, and MBPP+, mean ± std over three seeds.

Validation sizeHalf-split reliabilityGlobal top-KLESS (four checkpoints)RandomSUADSGap (SUADS vs global top-K)
135 (25%)0.98552.6 ± 1.749.4 ± 0.753.6 ± 1.055.4 ± 1.5+2.8
269 (50%)0.99252.3 ± 0.448.6 ± 1.453.6 ± 1.053.6 ± 1.0+1.3
538 (100%)0.99651.5 ± 1.050.2 ± 1.053.6 ± 1.055.5 ± 0.5+4.0

The gap does not shrink as the validation set grows. Two readings.

  1. At every size, SUADS is above global top-K and not below Random. The margins are strong at 100% (+4.0 over global top-K, +1.9 over Random, std 0.5 to 1.0) and also preserved at 25% (+2.8 and +1.8, std 1.5 to 1.7). At 50% SUADS is among the best methods. This consistent tendency highlights that SUADS is uniformly effective across multiple M scales, mainly due to providing a better cluster-level reference as explained above.

  2. The half-split reliability remains between 0.985 and 0.996. It may appear high, but it measures global reliability rather than cluster-level reliability, and further higher reliability may be desired due to alignment of general default directions. The cluster-level reliability suggests that all these settings may fall in the under-saturation regime of validation sizes as explained above, so the gap does not shrink as the validation set grows. We foresee that scaling the high-quality validation set to thousands of examples can show the shrinking tendency, though it is practically hard due to the scarcity of validation data, so we leave it as future work.

We revised the manuscript to state what really drives SUADS gain in Section 3 and present Theorem 8 (per-component gain retention, Appendix B.6) next to Theorem 1, and we will add a discussion about the limitations of the current numerical retention estimation in Appendix B in explaining SUADS effectiveness.

Weakness 1-b: Theorem 2 allows several clusters to select the same example and counts each selection, whereas SUADS assigns each candidate to one cluster before selection, so the theorem does not directly characterize the implemented method's gain.

SUADS's single-assignment rule changes little in what Theorem 2 measures. Theorem 2 analyses a relaxed rule in which each admitted cluster takes its own top quota from the whole pool, so one training example can be picked by two clusters. SUADS assigns each training example to one cluster first and takes the quota inside that cluster, so each example is picked once. We compared the two rules on the paper's Table 2 selection, where the two admitted clusters are MBPP sub-clusters: A has 319 validation examples and a quota of 1,493 training examples, B has 55 and 257. For each cluster we measure the mean cosine between the training gradients it picks and its own direction (its normalized centroid, the same under both rules), so the only thing that differs between the two rules is which examples are picked.

Mean cosine of picks to the cluster's direction, relaxed ruleSame, SUADS ruleSUADS / relaxed
Cluster A (319 validation examples, quota 1,493)0.0880.0780.88
Cluster B (55 validation examples, quota 257)0.1020.1010.99
Whole selection (1,750 picks)0.0900.0810.90

The two rules pick partly different training examples, but the quantity Theorem 2 is about changes very little: B's picks keep 99% of their mean cosine, A's picks 88%, and the whole selection keeps 90%. So Theorem 2 describes SUADS's selection up to a 10% gap, and the rule SUADS actually uses stays close to the one the theorem analyses. Section 3.2 and Appendix B.4 state this.

Question 2: What do clustering and the coherence gate add? Using the same final-checkpoint features on Gemma, how does SUADS compare with the known MBPP/EvoEval partition, without the gate and with the same quota rule, and with an MBPP-only LESS baseline?

We ran Experiment 2 to answer this question. The checkpoint and reference-source controls the reviewer asked for under Weakness 2 are in the table under that weakness.

Experiment 2 (clustering and gate). Setting: same model, pool, budget, features, seeds, and metric as Experiment 1, with the full validation set (538 examples). Methods:

  • SUADS as in Table 2.
  • The same k-means clustering with the gate switched off, so all four clusters are admitted.
  • The known MBPP/EvoEval partition used in place of the clusters, with SUADS's proportional quota rule and no gate.
  • Global top-K (c = 1).
  • LESS with the final checkpoint only, which gives it the features SUADS uses.
  • MBPP-only LESS, which is LESS with the validation set restricted to the 374 MBPP examples, on the same final-checkpoint features.
  • Random.
MethodAvg pass@1
SUADS (k-means, gate on, two MBPP sub-clusters of 319 and 55 admitted)55.5 ± 0.5
k-means with c_max = 4, no gate (all four clusters admitted: 319, 55, 119, 45)52.7 ± 2.0
Known MBPP/EvoEval partition, same quota rule, no gate52.0 ± 1.3
Global top-K (c = 1)51.5 ± 1.0
LESS, final checkpoint (one reference per source)53.2 ± 1.2
MBPP-only LESS, final checkpoint52.8 ± 1.1
Random53.6 ± 1.0

Removing either part removes the gain.

  1. Without the gate, clustering alone does not work. Admitting all four k-means clusters gives 52.7 ± 2.0 with twice Random's std. This method uses the same clustering as SUADS plus the two EvoEval clusters the gate rejects (coherence 0.43 and 0.46, against a calibrated threshold of 0.47). The known MBPP/EvoEval partition without the gate also uses an EvoEval reference and gives 52.0. Both are below Random.
  2. Without per-component references, a single reference gives 51.5 to 53.2 whatever its source, all on the same final-checkpoint features: global top-K on all 538 examples 51.5, a single MBPP reference 52.8, and LESS with one reference per source 53.2. All are at or within noise of Random (53.6), none above it.
  3. With references for the two coherent MBPP sub-clusters the gate admits, the result is 55.5.

So the gain depends on which components get a reference. The useful components are the two sub-clusters of MBPP, which the source labels do not expose. A single MBPP reference on the same features gives 52.8, so the gain is not from dropping EvoEval, it is from splitting MBPP. All methods are in the revised appendix.

Changes to the paper: Section 3 states the two roles (1: the analysis motivates references from coherent clusters rather than single examples, 2: the gated clustering decides which components get one) and presents Theorem 8 next to Theorem 1. The abstract’s bottleneck sentence and the introduction’s sentence attributing the baselines’ below-Random results to attenuation are also rephrased to match.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.