acceptodds
Under review as a conference paper at ICLR 2027

On the Robustness of Targeted Instruction Data Selection and Beyond

Abstract

Targeted instruction data selection and target-agnostic instruction selection are treated as two separate lines of work, yet they meet at a natural limit: when the target set is entirely absent or carries no task signal, a targeted selector reduces to a target-agnostic one. In practice, this limit is approached more often than it appears. Target sets collected from real deployments are typically small or may contain redundant or mislabeled examples, causing the estimated target profile to be noisy and unstable. A matching signal built on such a profile can steer selection away from high-quality source data and toward noise, producing worse results than simply ignoring the target set. We call a targeted selector robust if it consistently beats target-agnostic selection across varying target sizes and noise levels. Measured by this criterion, most existing targeted selectors are not robust: four of the five we evaluate score below a target-agnostic baseline, meaning the target set hurts rather than helps. This failure is largely invisible in existing benchmarks, because targeted methods are evaluated against random selection rather than against target-agnostic selection. We identify two structural causes and address them with our method, CLASS (Closed-set LAbel Space Selection), which combines a quality term to supply the target-agnostic floor with closed-set capability matching to keep profile estimation stable under both target scarcity and target noise across different benchmarks. CLASS reaches an average of 54.1 across five tasks, above the target-agnostic baseline at 52.0 and above every targeted baseline, with highest selection stability.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
targeted instruction data selection, task-specific instruction data selection, robustness of instruction data selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N9VfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies targeted instruction data selection with small or noisy target sets and emphasizes comparison with a strong quality-only baseline. CLASS uses one LLM call per sample to score quality and capability relevance. It then greedily selects high-quality samples to match the target’s capability profile. Using a TULU-V2 source pool and a 10K selection budget, CLASS averages 54.1 across five benchmarks, compared with 52.0 for quality-only selection, and retains its advantage under the tested target perturbations.

Critical strengths
  • The paper highlights the importance of a strong quality-only baseline. This comparison shows whether target information provides value beyond selecting good training data, a distinction that comparisons with random selection alone cannot establish.

  • CLASS is simple and practical. It obtains quality and capability scores via a single LLM call per sample and reuses source annotations across tasks. It consistently outperforms quality-only selection across the four tested conditions.

Critical weaknesses
  • The experiments do not fully explain why CLASS is more robust. Table 2 controls the quality signal, and Table 4 varies label granularity, but neither separates the effects of capability representations and dynamic deficit matching. LESS and RDS+ also aggregate target information (Appendix C.3), so aggregation alone does not explain the claimed advantage.

  • The evaluation protocol is unclear. Section 3.1 states that evaluation data are not used to construct targets or tune hyperparameters, while Section 5.1 says targets are drawn from the evaluation set. This makes it difficult to assess whether the reported gains hold on independent held-out data.

Questions for authors
  • Which exact splits are used for target construction, hyperparameter selection, and final evaluation? How were CLASS’s default hyperparameters selected?

  • Can you separate representation choice from selection rule? For example, by applying the same greedy rule to clustered embeddings, or replacing CLASS’s dynamic deficit matching with static matching over the same capability profiles.

3 comments
AuthorsOpenReview ↗

Our Reply (Part-1)

Thank you for your valuable time and insightful comments. We have carefully read your comments and respond to each of them below.

W1.1: The experiments do not fully explain why CLASS is more robust. Table 2 controls the quality signal, and Table 4 varies label granularity, but neither separates the effects of capability representations and dynamic deficit matching.

A: To separate these two factors, we crossed the representation (closed-set labels vs. clusters of embeddings) with the selection rule (dynamic deficit vs. static matching), holding the quality term and all hyperparameters fixed. Specifically: (1) We cluster embeddings used by TSDS into K=12K=12 groups with spherical k-means (the CLASS label space also has size 12), and use the soft assignments of each sample to the centroids as its profile, with the temperature set so that their mean entropy matches that of the label profiles. This replaces only the representation. (2) We replace the deficit rSr_S in Eq. (10) with the fixed target profile, so that the score no longer depends on the current selection. This replaces only the selection rule. The results are reported below:

RepresentationSelection RuleBBHGSMTruthfulQATyDiQAMMLUAverage
quality-only42.447.542.164.763.452.0
embedding clustersstatic39.451.044.363.962.952.3
embedding clustersdynamic42.652.046.961.863.453.3
closed-set labelsstatic43.246.047.064.963.452.9
closed-set labelsdynamic (CLASS)43.848.049.465.663.854.1

These results suggest that both factors contribute: the dynamic rule adds 1.0 and 1.2 points over the static rule with clusters and with labels, and closed-set labels add 0.6 and 0.8 points over clusters under the static and the dynamic rule. Their combination (CLASS) achieves the best average performance and is the only variant that beats quality-only selection on every benchmark. Thank you again for your insightful comments. We have added these results and analyses to the main paper (Section 5.4, Table 4). The corresponding code, named run_select_sep.py, is also uploaded in the Supplementary Material.

W1.2: LESS and RDS+ also aggregate target information (Appendix C.3), so aggregation alone does not explain the claimed advantage.

A: We agree that aggregation alone does not explain the advantage. Rather, the advantage lies in the fact that our aggregated profile is low-dimensional and takes the form of a soft distribution, and that it captures which capabilities a task requires rather than which particular examples are drawn. This yields better robustness across target changes (scarcity and noise) as well as across target resampling (Appendix D.9). We have revised our wording accordingly in Sections 4.4 and 5.3. In addition, since the quality score and the capability score are obtained from a single prompt, our aggregated profile is easy to obtain and does not require computing gradients or embeddings for the entire source pool, as LESS and TSDS do.

AuthorsOpenReview ↗

Our Reply (Part-2)

W2: The evaluation protocol is unclear. Section 3.1 states that evaluation data are not used to construct targets or tune hyperparameters, while Section 5.1 says targets are drawn from the evaluation set. This makes it difficult to assess whether the reported gains hold on independent held-out data.

A: We apologize for the unclear wording. We explain our setting below:

  • Dataset Construction and final evaluation: For each benchmark, a script with a fixed seed samples m=50m=50 examples from the official test split as the target set and leaves all remaining examples to a separate evaluation split. Every model is evaluated only on this evaluation split. This can be directly verified in the code from our initial submission included in the Supplementary Material: dataset/1_dataset_splitter.py splits the official test split into two folders, split_target and split_eval, where split_target contains the target samples and split_eval contains the samples used for final evaluation. There is no sample overlap between these two folders. The evaluation script finetune_and_eval/eval_all_tasks.sh only accesses the split_eval folder. All methods use the same evaluation protocol and see the same evaluation set for each benchmark. Thus, there is no leakage in our data split. We recorded the original benchmark test size in Appendix C.2, not the actual size of evaluation set after target removing, which causes ambiguity. We have rewritten the corresponding description to make the dataset construction in Section 3.1, Section 5.1, and Appendix C.2 consistent, and Table 7 in Appendix C.2 now lists, for every benchmark, the size of the target set and the final evaluation set.

  • Hyper-parameter tuning: Many instruction selection methods have hyperparameters. However, most of them, including all methods used in our comparison, follow a “Base Configuration + Ablations on Hyperparameters” protocol. That is, they use the base configuration in the main results and perform ablations to test robustness/variance over a certain hyperparameter range. Usually, the base configuration has the best average performance. Since we use their official code and hyper-parameters, and our evaluation benchmarks (BBH, GSM, TruthfulQA, TyDiQA, MMLU) largely coincide with those used in​ the baseline methods we compare against, we follow the same “Base Configuration + Ablations on Hyperparameters” protocol to ensure a fair comparison. For example, in Table 14 (Appendix D.5; Table 11 in the original submission), every configuration except the extreme threshold τ=9\tau=9, which leaves only 53.1% of the source pool covered, surpasses the quality-only baseline on average. At the default τ=5\tau=5, every α\alpha in the grid yields a five-task average between 53.1 and 54.1, above the quality-only floor of 52.0. The α\alpha for every quality-augmented baseline in Appendix C.6 was selected in the same way, which can only raise their scores rather than amounting to arbitrary parameter selection. We have rewritten Appendix C.6 to make this clearer. Meanwhile, since the per-example predictions of all runs are stored, we conducted an additional experiment to verify the stability of CLASS's hyperparameter selection. We randomly split the evaluated examples of every benchmark into two disjoint halves: one used as a development (validation) set and the other as the final evaluation set. Hyperparameters are chosen on the development half and tested on the final evaluation half. The results below (Table 9 in Appendix C.6) show that the development half selects the same configuration as our original paper, (λ,α,τ)=(1,50,5)(\lambda,\alpha,\tau)=(1,50,5) with K=12K=12, which scores 54.2 on the final evaluation half.

(λ,α,τ)(\lambda,\alpha,\tau), KK(1,10,5), K = 12(1,30,5), K = 12(1,50,5), K = 12(1,70,5), K = 12(1,50,5), K = 3(1,50,5), K = 47
Development half (validation)53.253.354.053.253.152.5

Q1: Which exact splits are used for target construction, hyperparameter selection, and final evaluation? How were CLASS’s default hyperparameters selected?

A: Please refer to our reply to your W2.

Q2: Can you separate representation choice from selection rule?

A: Yes. Following your advice, we have done the experiments. Please refer to our reply to your W1.1.

Thank you again for your valuable time and insightful comments, which have helped us improve our paper. We hope our reply addresses your concerns. Should you have any questions, please do not hesitate to let us know.

Thanks for clarifying the data split and adding the ablation I asked for. The split-half check on the saved predictions helps, but I'm still concerned about tuning on the original evaluation set. I'll keep my score at 2.

Reviewer_2z8fOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that targeted instruction selection should be compared against a target-agnostic quality-only selector (an LLM judge score, AlpaGasus style) rather than random selection, and calls a targeted method robust if it beats that floor under small or noisy target sets. Under this view, most existing targeted selectors (LESS, RDS+, TSDS) fall below the floor. The proposed method, CLASS, gets a quality score and 12 capability-label scores from a single GPT-4o-mini call per sample, then greedily selects data to match the target's label distribution while adding a quality bonus. Experiments use a TULU-V2 pool, a 10K budget, and LoRA on Llama-3.1-8B, evaluated on five benchmarks under target scarcity (m=5), paraphrase noise, and 30% OOD noise. CLASS averages 54.1 vs 52.0 for quality-only.

Critical strengths

The reframing is useful. Asking whether a targeted selector beats a method that ignores the target, instead of beating random, is the right question, and Table 1 makes the case that several popular methods do not clear that bar. This alone could change how people evaluate this line of work.

Critical weaknesses
  1. The paper's own results suggest that the targeted part of CLASS is not what helps. In Table 2, the closed-set matching term alone (lambda=0) scores 51.4 / 52.1 / 51.3 / 51.9, i.e. at or below the 52.0 floor in three of four conditions, which is the same failure the paper attributes to other methods. CLASS is also nearly insensitive to the target: 54.1, 53.9, 54.2, 53.7 across clean, m=5, paraphrased, and 30% OOD targets, and K=3 labels still gets 53.0. That looks less like robustness and more like the selection barely depending on the target. Since the matching term also penalizes over-represented labels, it can act as a generic diversity/balance regularizer. Without a control that replaces pi_T with a uniform or source-pool label distribution (no target at all), I can't tell whether CLASS is a better targeted selector or simply a better target-agnostic one. If it is the latter, the paper supports its own thesis but not its method contribution.

  2. The evaluation protocol leaks test information. The target sets are "drawn from the evaluation set", and the test-set sizes in Table 5 (e.g. GSM8K 1,319, MMLU 14,042) look like the full test sets, so the 50 target examples may be part of the evaluation data. Hyperparameters are also picked on test results: C.6 says alpha for the +qual baselines is chosen "by maximizing the five-task average at m=50", and CLASS's default (1,50,5) is the best row in Table 11, with K=12 the best of {3,12,47}. There is no validation split. On top of this, most of the gain over quality-only comes from TruthfulQA (+7.3; the other four benchmarks average about +0.8, comparable to the reported stds), and the appendix tables for the noise conditions have no stds. Under perturbation some numbers go up (GSM 48.0 to 51.5 with paraphrased targets), so run-to-run variation is on the order of the claimed differences.

Questions for authors
  1. Are the target examples disjoint from the test examples used for evaluation?
  2. Variance across target draws for each condition, or a significance test, would help a lot.
5 comments
AuthorsOpenReview ↗

Our Reply (Part-1)

Thank you for your valuable time and insightful comments. We have carefully read your comments and respond to each of them below.

W1.1: The paper's own results suggest that the targeted part of CLASS is not what helps. In Table 2, the closed-set matching term alone (lambda=0) scores 51.4 / 52.1 / 51.3 / 51.9, i.e. at or below the 52.0 floor in three of four conditions, which is the same failure the paper attributes to other methods.

A: The matching-only rule (λ=0\lambda=0) falling below the floor is consistent with our paper rather than in tension with it: without the quality term, any matching signal may steer selection toward low-quality data that resembles the target, which is precisely the failure mode we diagnose in other methods. Our central claim concerns robustness and the combination of the two terms. With the quality term held fixed, Tables 2 and 3 show that the gain of CLASS over quality-only selection comes from matching the target-specific profile, and that this gain remains stable under scarcity and noise conditions.

In addition, CLASS has the largest efficiency advantage over other quality-augmented methods, since the quality score and the capability score are obtained from a single prompt. By contrast, combining a separate quality scorer with an existing per-sample matching method requires two distinct inference passes over the source pool, one for quality and one for matching, which greatly increases computation.

W1.2: CLASS is also nearly insensitive to the target: 54.1, 53.9, 54.2, 53.7 across clean, m=5, paraphrased, and 30% OOD targets, and K=3 labels still gets 53.0. That looks less like robustness and more like the selection barely depending on the target. Since the matching term also penalizes over-represented labels, it can act as a generic diversity/balance regularizer. Without a control that replaces pi_T with a uniform or source-pool label distribution (no target at all), I can't tell whether CLASS is a better targeted selector or simply a better target-agnostic one. If it is the latter, the paper supports its own thesis but not its method contribution.

A: Thank you for raising this point. We clarify it below:

  • Insensitive to the target perturbations: Our (low-dimensional and aggregated) target profile captures which capabilities a task requires rather than which particular examples are drawn. The perturbations in Table 1 change the target examples but preserve the task, and a robust selector should be insensitive to them. However, insensitivity to perturbations is different from insensitivity to the target. We report the selection overlap (%) for several pairs of selections below:
Pair of SelectionsOverlap @ 10K
Same task, m=50 vs. m=577.9
Same task, m=50 vs. stylistic noise94.5
Same task, m=50 vs. 30% OOD noise81.3
Different tasks, average over the 10 pairs36.3
Different tasks, GSM8K vs. TruthfulQA1.4
Different tasks, GSM8K vs. TyDiQA0.5

These results show that CLASS selection is insensitive to target perturbations but does depend on the task.

  • K = 3 results: Because each capability label is accompanied by a natural-language definition, the LLM assigns relevance scores by semantic matching rather than keyword lookup. The resulting target profile is a soft distribution that spreads probability mass across related capabilities. Thus, even when K = 3, the target profile can still represent the target.

  • Targeted selector or a better target-agnostic one: In addition to the selection-overlap table above (which also provides strong evidence that CLASS is a targeted selector), we followed your advice, ran experiments that replace πT\pi_T with a uniform or a source-pool label distribution, and report the results below:

Target profileBBHGSMTruthfulQATyDiQAMMLUAverage
quality only42.447.542.164.763.452.0
uniform41.837.046.162.763.550.2
source-pool label distribution41.644.541.863.263.350.9
own target (CLASS)43.848.049.465.663.854.1

These results show that replacing πT\pi_T with these two variants does not improve on quality-only selection. Thus, the gain of CLASS comes from the target-specific profile rather than from label balancing.

Thank you again for your insightful comments. We have incorporated these analyses into the main paper (Section 5.4, Table 3), and the corresponding code, named run_select_target_profile.py, is also uploaded in the Supplementary Material.

AuthorsOpenReview ↗

Our Reply (Part-2)

W2.1: The evaluation protocol leaks test information. The target sets are "drawn from the evaluation set", and the test-set sizes in Table 5 (e.g. GSM8K 1,319, MMLU 14,042) look like the full test sets, so the 50 target examples may be part of the evaluation data. Hyperparameters are also picked on test results: C.6 says alpha for the +qual baselines is chosen "by maximizing the five-task average at m=50", and CLASS's default (1,50,5) is the best row in Table 11, with K=12 the best of {3,12,47}. There is no validation split.

A: We apologize for the unclear wording. We explain our setting below:

  • Dataset Construction and final evaluation: For each benchmark, a script with a fixed seed samples m=50m=50 examples from the official test split as the target set and leaves all remaining examples to a separate evaluation split. Every model is evaluated only on this evaluation split. This can be directly verified in the code from our initial submission included in the Supplementary Material: dataset/1_dataset_splitter.py splits the official test split into two folders, split_target and split_eval, where split_target contains the target samples and split_eval contains the samples used for final evaluation. There is no sample overlap between these two folders. The evaluation script finetune_and_eval/eval_all_tasks.sh only accesses the split_eval folder. All methods use the same evaluation protocol and see the same evaluation set for each benchmark. Thus, there is no leakage in our data split. We recorded the original benchmark test size in Appendix C.2, not the actual size of evaluation set after target removing, which causes ambiguity. We have rewritten the corresponding description to make the dataset construction in Section 3.1, Section 5.1, and Appendix C.2 consistent, and Table 7 in Appendix C.2 now lists, for every benchmark, the size of the target set and the final evaluation set.

  • Hyper-parameter tuning: Many instruction selection methods have hyperparameters. However, most of them, including all methods used in our comparison, follow a “Base Configuration + Ablations on Hyperparameters” protocol. That is, they use the base configuration in the main results and perform ablations to test robustness/variance over a certain hyperparameter range. Usually, the base configuration has the best average performance. Since we use their official code and hyper-parameters, and our evaluation benchmarks (BBH, GSM, TruthfulQA, TyDiQA, MMLU) largely coincide with those used in​ the baseline methods we compare against, we follow the same “Base Configuration + Ablations on Hyperparameters” protocol to ensure a fair comparison. For example, in Table 14 (Appendix D.5; Table 11 in the original submission), every configuration except the extreme threshold τ=9\tau=9, which leaves only 53.1% of the source pool covered, surpasses the quality-only baseline on average. At the default τ=5\tau=5, every α\alpha in the grid yields a five-task average between 53.1 and 54.1, above the quality-only floor of 52.0. The α\alpha for every quality-augmented baseline in Appendix C.6 was selected in the same way, which can only raise their scores rather than amounting to arbitrary parameter selection. We have rewritten Appendix C.6 to make this clearer. Meanwhile, since the per-example predictions of all runs are stored, we conducted an additional experiment to verify the stability of CLASS's hyperparameter selection. We randomly split the evaluated examples of every benchmark into two disjoint halves: one used as a development (validation) set and the other as the final evaluation set. Hyperparameters are chosen on the development half and tested on the final evaluation half. The results below (Table 9 in Appendix C.6) show that the development half selects the same configuration as our original paper, (λ,α,τ)=(1,50,5)(\lambda,\alpha,\tau)=(1,50,5) with K=12K=12, which scores 54.2 on the final evaluation half.

(λ,α,τ)(\lambda,\alpha,\tau), KK(1,10,5), K = 12(1,30,5), K = 12(1,50,5), K = 12(1,70,5), K = 12(1,50,5), K = 3(1,50,5), K = 47
Development half (validation)53.253.354.053.253.152.5
AuthorsOpenReview ↗

Our Reply (Part-3)

W2.2: most of the gain over quality-only comes from TruthfulQA (+7.3; the other four benchmarks average about +0.8, comparable to the reported stds).

A: Our larger gain on TruthfulQA arises because it is the benchmark whose target requires a capability (Dialogue) that quality-only selection almost never picks. We agree that on some benchmarks, CLASS does not achieve the best performance among all baselines. However, CLASS is the only method that beats quality-only selection on every benchmark and remains stable across scarcity and noise, which is our central claim in the paper. Notably, CLASS also achieves the best average performance among all compared methods.

W2.3: the appendix tables for the noise conditions have no stds. Under perturbation some numbers go up (GSM 48.0 to 51.5 with paraphrased targets), so run-to-run variation is on the order of the claimed differences.

A: GSM exhibits larger variation partly because the untuned base model performs very weakly on this benchmark (only 20 in Table 1). Thus, even when a small fraction of the selected IDs changes, the model may behave differently. The same holds for LESS (43.3 → 47.0) and RDS+ (49.8 → 47.5) in our initial manuscript. Without target changes, and when the selection method itself is near-deterministic, the GSM results can be stable; however, they can be affected under target variation. In contrast, the base model already performs relatively well on MMLU, so the variation under target change is small.

We have rerun each noise condition three times and now report the mean and standard deviation in the appendix tables (Tables 10 to 12, Appendices D.1 to D.3).

We agree that variation across target draws is important; Reviewer LzJQ raised the same concern about targeting gains over independent target draws. We have conducted these experiments: we drew two additional target sets per benchmark (seeds 41 and 43), which together with the original draw (seed 42) give three independent draws, and regenerated all four conditions from each draw. We report the targeting gain of CLASS over quality-only selection (five-task average) across the independent target draws in the table below (Appendix D.9):

ConditionMean gainRange over draws
m = 50+1.8[+1.5, +2.1]
m = 5+2.1[+1.9, +2.2]
Stylistic noise+2.1[+1.6, +2.4]
OOD noise (30%)+1.6[+1.2, +1.9]

These results show that the five-task gain is positive in every draw and every condition, and that no benchmark falls significantly below the floor in any draw. We also add an experiment that tests robustness to target resampling without any fine-tuning. The target set enters the CLASS selection rule only through the aggregated profile πT\pi_T, so comparing the profiles obtained from independent draws directly measures how much resampling can change what CLASS optimizes. The table below reports the L2 distance between the resulting profiles, averaged over the five benchmarks.

Seed 41Seed 42Seed 43
Seed 410.000.030.04
Seed 420.030.000.04
Seed 430.040.040.00

The distance between any two profiles is small. In contrast, under a single draw, the profiles of two different benchmarks are 0.789 apart on average and never closer than 0.239. Thus, a benchmark's profile is about 21 times closer to its own profile under a different draw than to the profile of another benchmark on average. This also supports the claim in Section 4.4: the profile captures which capabilities a task requires rather than which particular examples are drawn (Appendix D.9, Tables 17 and 18).

Q1: Are the target examples disjoint from the test examples used for evaluation?

A: Yes, the target examples are disjoint from the test examples used for evaluation. Please refer to our reply to your W2.1.

Q2: Variance across target draws for each condition, or a significance test, would help a lot.

A: We have conducted the corresponding experiments. Please refer to our reply to your W2.3.

Thank you again for your valuable time and insightful comments, which have helped us improve our paper. We hope our reply addresses your concerns. Should you have any questions, please do not hesitate to let us know.

Reviewer_LzJQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies targeted instruction selection when the target set is small or noisy, using quality-only selection as the reference for whether targeting helps. It proposes CLASS, which obtains a quality score and a fixed capability profile from one LLM call per source example. Greedy selection combines quality with the deficit between selected and target capability distributions. Experiments use a TULU-V2 source pool, five benchmarks, target scarcity, paraphrasing, and off-task contamination. CLASS improves the reported five-task average over quality-only selection, and the appendix includes annotator, vocabulary, hyperparameter, model-transfer, and cost analyses.

Critical strengths
  1. The evaluation asks a useful question: whether target conditioning improves upon a strong selector that ignores the target. Table 1 demonstrates that outperforming random selection can coexist with underperforming quality-only selection. Table 2 adds a shared quality signal to several matching mechanisms, making the comparison more informative than comparing CLASS only against matching-only baselines.
  2. The method is simple and its source annotations can be reused across tasks. Table 1 reports averages of 54.1, 53.9, 54.2, and 53.7 across the four conditions, versus 52.0 for quality-only selection; Table 3 also shows higher selection overlap under perturbation. The appendix includes an open annotator and a second base model, which strengthen the practical case.
Critical weaknesses
  1. The separation of selection, hyperparameter tuning, and final evaluation is unclear. Section 3.1 says that the evaluation set E is never used to build targets or tune hyperparameters, but Section 5.1 explicitly draws the 50 targets from the evaluation set. Appendix C.2 lists standard test-set sizes without explaining whether these targets are removed. Appendix C.6 selects quality-augmented baseline weights by the five-task average at m=50 without identifying a separate development set. These statements do not establish leakage, but the ambiguity prevents a reliable interpretation of the central performance comparisons.
  2. The evidence does not fully substantiate the claimed mechanism or breadth of robustness. Averaging bounded profiles limits one example's influence, but 30% systematic contamination shifts the mean toward the contaminating distribution; it need not cancel. Proposition 1 concerns the selection objective, rather than downstream robustness. Table 1 reports three-run standard deviations without stating whether target sets are redrawn, and several per-task gains over quality-only are small. Appendix D.7 omits quality-only on the second base model. Thus the strongest claim, consistently positive targeting gain, remains insufficiently tested against target-sampling variation.
Questions for authors
  1. Can you specify the exact target, development, and evaluation splits, confirm whether every target ID is excluded from final evaluation, and explain where all hyperparameters were chosen? Please reconcile Section 3.1 with Section 5.1 and Appendix C.6. If overlap occurred, disjoint evaluation would materially affect my assessment.
  2. What randomness do the three runs cover? Can you provide paired targeting gains over independently drawn target sets for the existing scarcity/noise conditions, and the missing quality-only result for Mistral? Please also distinguish the bounded single-example influence argument from a guarantee under systematic contamination.
5 comments
AuthorsOpenReview ↗

Our Reply (Part-1)

Thank you for your valuable time and insightful comments. We have carefully read your comments and respond to each of them below.

W1: The separation of selection, hyperparameter tuning, and final evaluation is unclear. Section 3.1 says that the evaluation set E is never used to build targets or tune hyperparameters, but Section 5.1 explicitly draws the 50 targets from the evaluation set. Appendix C.2 lists standard test-set sizes without explaining whether these targets are removed. Appendix C.6 selects quality-augmented baseline weights by the five-task average at m=50 without identifying a separate development set. These statements do not establish leakage, but the ambiguity prevents a reliable interpretation of the central performance comparisons.

A: We apologize for the unclear wording. We explain our setting below:

  • Dataset Construction and final evaluation: For each benchmark, a script with a fixed seed samples m=50m=50 examples from the official test split as the target set and leaves all remaining examples to a separate evaluation split. Every model is evaluated only on this evaluation split. This can be directly verified in the code from our initial submission included in the Supplementary Material: dataset/1_dataset_splitter.py splits the official test split into two folders, split_target and split_eval, where split_target contains the target samples and split_eval contains the samples used for final evaluation. There is no sample overlap between these two folders. The evaluation script finetune_and_eval/eval_all_tasks.sh only accesses the split_eval folder. All methods use the same evaluation protocol and see the same evaluation set for each benchmark. Thus, there is no leakage in our data split. We recorded the original benchmark test size in Appendix C.2, not the actual size of evaluation set after target removing, which causes ambiguity. We have rewritten the corresponding description to make the dataset construction in Section 3.1, Section 5.1, and Appendix C.2 consistent, and Table 7 in Appendix C.2 now lists, for every benchmark, the size of the target set and the final evaluation set.

  • Hyper-parameter tuning: Many instruction selection methods have hyperparameters. However, most of them, including all methods used in our comparison, follow a “Base Configuration + Ablations on Hyperparameters” protocol. That is, they use the base configuration in the main results and perform ablations to test robustness/variance over a certain hyperparameter range. Usually, the base configuration has the best average performance. Since we use their official code and hyper-parameters, and our evaluation benchmarks (BBH, GSM, TruthfulQA, TyDiQA, MMLU) largely coincide with those used in​ the baseline methods we compare against, we follow the same “Base Configuration + Ablations on Hyperparameters” protocol to ensure a fair comparison. For example, in Table 14 (Appendix D.5; Table 11 in the original submission), every configuration except the extreme threshold τ=9\tau=9, which leaves only 53.1% of the source pool covered, surpasses the quality-only baseline on average. At the default τ=5\tau=5, every α\alpha in the grid yields a five-task average between 53.1 and 54.1, above the quality-only floor of 52.0. The α\alpha for every quality-augmented baseline in Appendix C.6 was selected in the same way, which can only raise their scores rather than amounting to arbitrary parameter selection. We have rewritten Appendix C.6 to make this clearer. Meanwhile, since the per-example predictions of all runs are stored, we conducted an additional experiment to verify the stability of CLASS's hyperparameter selection. We randomly split the evaluated examples of every benchmark into two disjoint halves: one used as a development (validation) set and the other as the final evaluation set. Hyperparameters are chosen on the development half and tested on the final evaluation half. The results below (Table 9 in Appendix C.6) show that the development half selects the same configuration as our original paper, (λ,α,τ)=(1,50,5)(\lambda,\alpha,\tau)=(1,50,5) with K=12K=12, which scores 54.2 on the final evaluation half.

(λ,α,τ)(\lambda,\alpha,\tau), KK(1,10,5), K = 12(1,30,5), K = 12(1,50,5), K = 12(1,70,5), K = 12(1,50,5), K = 3(1,50,5), K = 47
Development half (validation)53.253.354.053.253.152.5
AuthorsOpenReview ↗

Our Reply (Part-2)

W2.1: The evidence does not fully substantiate the claimed mechanism or breadth of robustness. Averaging bounded profiles limits one example's influence, but 30% systematic contamination shifts the mean toward the contaminating distribution; it need not cancel.

A: We agree that these are different statements. Under systematic contamination, every aggregate-based selector, including CLASS, LESS, and RDS+, receives a biased target summary, so the relevant question is not whether the selection changes but what replaces the original selection. A contaminated profile is a mixture of the intended task profile and the profiles of the contaminating tasks, and CLASS selects for this mixture, so a growing share of its budget moves to off-task capabilities. Two observations distinguish CLASS from the other selectors:

First, contamination changes the capability mix of CLASS, not the quality of what it selects. The matching term of CLASS decides how much budget each capability receives, while the samples that fill this budget within a capability are chosen largely by quality. Contamination therefore changes the capability mix of the selection, but the samples that enter are still among the highest-quality ones: under 30% contamination, the samples entering the CLASS selection have mean quality 8.67, close to that of the quality-only selection (8.71) and well above the pool average (7.03). Per-sample similarity signals rank individual samples instead, so a biased target reorders them regardless of quality: the samples entering the quality-augmented RDS+ and TSDS (gradient) selections have mean quality 7.96 and 7.55.

Second, CLASS stays stable while departing from quality-only selection. The table below reports the selection behavior under 30% OOD contamination, averaged over the five benchmarks.

MethodShared with quality-only (%)Retained under contamination (%)Quality of entering samplesAvg. performance (m=50)Avg. performance 30% OOD
RDS+ + qual17.080.47.9651.251.1
LESS + qual30.668.18.3852.352.2
TSDS (grad) + qual15.419.57.5552.251.7
TSDS (sem) + qual14.519.87.5352.851.3
CLASS36.281.38.6754.153.6

The CLASS selection (m=50 setting) shares 36.2% of its samples with quality-only selection, yet 81.3% of it survives under 30% contamination. The quality-augmented baselines either fill their departure with lower-quality samples or change it almost entirely. CLASS is the only method that combines a substantial targeted departure with stability and high-quality replacements, and it retains most of its margin over the floor. We have added these analyses to the revision (Section 4.4 and Appendix D.10, Table 19).

W2.2: Proposition 1 concerns the selection objective, rather than downstream robustness.

A: We agree that Proposition 1 shows that the greedy rule is first-order descent on the matching objective, and it makes no claim about downstream robustness. The revision states this scope explicitly in Section 4.3 and supports the robustness claims using the experiments and analyses.

AuthorsOpenReview ↗

Our Reply (Part-3)

W2.3: Table 1 reports three-run standard deviations without stating whether target sets are redrawn. The strongest claim, consistently positive targeting gain, remains insufficiently tested against target-sampling variation.

A: In Table 1, the three-run standard deviations are computed on a fixed target set (following the same setting as LESS, TSDS, and TADS, our compared baselines). Randomness arises from two sources: (i) selection randomness within each selection method, for example, LESS first randomly selects 5% of source samples for warm-up training, and TSDS samples source samples according to a probability distribution; and (ii) randomness in fine-tuning, for example, data order and LoRA initialization. We therefore follow the standard protocol and run each method multiple times. We now state this in the caption of Table 1.

We agree that providing targeting gains over independently drawn target sets is necessary to demonstrate the robustness of CLASS. We have conducted these experiments: we drew two additional target sets per benchmark (seeds 41 and 43), which together with the original draw (seed 42) give three independent draws, and regenerated all four conditions from each draw. We report the targeting gain of CLASS over quality-only selection (five-task average) across the independent target draws in the table below (Appendix D.9):

ConditionMean gainRange over draws
m = 50+1.8[+1.5, +2.1]
m = 5+2.1[+1.9, +2.2]
Stylistic noise+2.1[+1.6, +2.4]
OOD noise (30%)+1.6[+1.2, +1.9]

These results show that the five-task gain is positive in every draw and every condition, and that no benchmark falls significantly below the floor in any draw. We also add an experiment that tests robustness to target resampling without any fine-tuning. The target set enters the CLASS selection rule only through the aggregated profile πT\pi_T, so comparing the profiles obtained from independent draws directly measures how much resampling can change what CLASS optimizes. The table below reports the L2 distance between the resulting profiles, averaged over the five benchmarks.

Seed 41Seed 42Seed 43
Seed 410.000.030.04
Seed 420.030.000.04
Seed 430.040.040.00

The distance between any two profiles is small. In contrast, under a single draw, the profiles of two different benchmarks are 0.789 apart on average and never closer than 0.239. Thus, a benchmark's profile is about 21 times closer to its own profile under a different draw than to the profile of another benchmark on average. This also supports the claim in Section 4.4: the profile captures which capabilities a task requires rather than which particular examples are drawn (Appendix D.9, Tables 17 and 18).

W2.4: several per-task gains over quality-only are small.

A: We agree that several per-task gains over quality-only selection are small. However, CLASS is the only method that beats quality-only selection on every benchmark and remains stable across scarcity and noise conditions, which is our central claim in the paper. Notably, CLASS also achieves the best average performance among all compared methods.

W2.5: Appendix D.7 omits quality-only on the second base model.

A: We have added the quality-only results to Appendix D.7.

Q1: Can you specify the exact target, development, and evaluation splits, confirm whether every target ID is excluded from final evaluation, and explain where all hyperparameters were chosen? Please reconcile Section 3.1 with Section 5.1 and Appendix C.6. If overlap occurred, disjoint evaluation would materially affect my assessment.

A: There is no overlap between the target set and the final evaluation set. Please refer to our reply to your W1.1.

Q2.1: What randomness do the three runs cover? Can you provide paired targeting gains over independently drawn target sets for the existing scarcity/noise conditions, and the missing quality-only result for Mistral?

A: Please refer to our reply to your W2.3 and W2.5.

Q2.2: Please also distinguish the bounded single-example influence argument from a guarantee under systematic contamination.

A: In the revision, we have distinguished the bounded single-example influence from a guarantee under systematic contamination. Section 4.4 now restricts the O(1/m)O(1/m) argument to isolated errors and presents robustness under systematic contamination as an empirical finding, supported by the analyses in our reply to your W2.1.

Thank you again for your valuable time and insightful comments, which have helped us improve our paper. We hope our reply addresses your concerns. Should you have any questions, please do not hesitate to let us know.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.