- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies targeted instruction data selection with small or noisy target sets and emphasizes comparison with a strong quality-only baseline. CLASS uses one LLM call per sample to score quality and capability relevance. It then greedily selects high-quality samples to match the target’s capability profile. Using a TULU-V2 source pool and a 10K selection budget, CLASS averages 54.1 across five benchmarks, compared with 52.0 for quality-only selection, and retains its advantage under the tested target perturbations.
Critical strengths
-
The paper highlights the importance of a strong quality-only baseline. This comparison shows whether target information provides value beyond selecting good training data, a distinction that comparisons with random selection alone cannot establish.
-
CLASS is simple and practical. It obtains quality and capability scores via a single LLM call per sample and reuses source annotations across tasks. It consistently outperforms quality-only selection across the four tested conditions.
Critical weaknesses
-
The experiments do not fully explain why CLASS is more robust. Table 2 controls the quality signal, and Table 4 varies label granularity, but neither separates the effects of capability representations and dynamic deficit matching. LESS and RDS+ also aggregate target information (Appendix C.3), so aggregation alone does not explain the claimed advantage.
-
The evaluation protocol is unclear. Section 3.1 states that evaluation data are not used to construct targets or tune hyperparameters, while Section 5.1 says targets are drawn from the evaluation set. This makes it difficult to assess whether the reported gains hold on independent held-out data.
Questions for authors
-
Which exact splits are used for target construction, hyperparameter selection, and final evaluation? How were CLASS’s default hyperparameters selected?
-
Can you separate representation choice from selection rule? For example, by applying the same greedy rule to clustered embeddings, or replacing CLASS’s dynamic deficit matching with static matching over the same capability profiles.
3 comments
Our Reply (Part-2)
W2: The evaluation protocol is unclear. Section 3.1 states that evaluation data are not used to construct targets or tune hyperparameters, while Section 5.1 says targets are drawn from the evaluation set. This makes it difficult to assess whether the reported gains hold on independent held-out data.
A: We apologize for the unclear wording. We explain our setting below:
-
Dataset Construction and final evaluation: For each benchmark, a script with a fixed seed samples examples from the official test split as the target set and leaves all remaining examples to a separate evaluation split. Every model is evaluated only on this evaluation split. This can be directly verified in the code from our initial submission included in the Supplementary Material:
dataset/1_dataset_splitter.pysplits the official test split into two folders,split_targetandsplit_eval, wheresplit_targetcontains the target samples andsplit_evalcontains the samples used for final evaluation. There is no sample overlap between these two folders. The evaluation scriptfinetune_and_eval/eval_all_tasks.shonly accesses thesplit_evalfolder. All methods use the same evaluation protocol and see the same evaluation set for each benchmark. Thus, there is no leakage in our data split. We recorded the original benchmark test size in Appendix C.2, not the actual size of evaluation set after target removing, which causes ambiguity. We have rewritten the corresponding description to make the dataset construction in Section 3.1, Section 5.1, and Appendix C.2 consistent, and Table 7 in Appendix C.2 now lists, for every benchmark, the size of the target set and the final evaluation set. -
Hyper-parameter tuning: Many instruction selection methods have hyperparameters. However, most of them, including all methods used in our comparison, follow a “Base Configuration + Ablations on Hyperparameters” protocol. That is, they use the base configuration in the main results and perform ablations to test robustness/variance over a certain hyperparameter range. Usually, the base configuration has the best average performance. Since we use their official code and hyper-parameters, and our evaluation benchmarks (BBH, GSM, TruthfulQA, TyDiQA, MMLU) largely coincide with those used in the baseline methods we compare against, we follow the same “Base Configuration + Ablations on Hyperparameters” protocol to ensure a fair comparison. For example, in Table 14 (Appendix D.5; Table 11 in the original submission), every configuration except the extreme threshold , which leaves only 53.1% of the source pool covered, surpasses the quality-only baseline on average. At the default , every in the grid yields a five-task average between 53.1 and 54.1, above the quality-only floor of 52.0. The for every quality-augmented baseline in Appendix C.6 was selected in the same way, which can only raise their scores rather than amounting to arbitrary parameter selection. We have rewritten Appendix C.6 to make this clearer. Meanwhile, since the per-example predictions of all runs are stored, we conducted an additional experiment to verify the stability of CLASS's hyperparameter selection. We randomly split the evaluated examples of every benchmark into two disjoint halves: one used as a development (validation) set and the other as the final evaluation set. Hyperparameters are chosen on the development half and tested on the final evaluation half. The results below (Table 9 in Appendix C.6) show that the development half selects the same configuration as our original paper, with , which scores 54.2 on the final evaluation half.
| , | (1,10,5), K = 12 | (1,30,5), K = 12 | (1,50,5), K = 12 | (1,70,5), K = 12 | (1,50,5), K = 3 | (1,50,5), K = 47 |
|---|---|---|---|---|---|---|
| Development half (validation) | 53.2 | 53.3 | 54.0 | 53.2 | 53.1 | 52.5 |
Q1: Which exact splits are used for target construction, hyperparameter selection, and final evaluation? How were CLASS’s default hyperparameters selected?
A: Please refer to our reply to your W2.
Q2: Can you separate representation choice from selection rule?
A: Yes. Following your advice, we have done the experiments. Please refer to our reply to your W1.1.
Thank you again for your valuable time and insightful comments, which have helped us improve our paper. We hope our reply addresses your concerns. Should you have any questions, please do not hesitate to let us know.
Thanks for clarifying the data split and adding the ablation I asked for. The split-half check on the saved predictions helps, but I'm still concerned about tuning on the original evaluation set. I'll keep my score at 2.
Our Reply (Part-1)
Thank you for your valuable time and insightful comments. We have carefully read your comments and respond to each of them below.
A: To separate these two factors, we crossed the representation (closed-set labels vs. clusters of embeddings) with the selection rule (dynamic deficit vs. static matching), holding the quality term and all hyperparameters fixed. Specifically: (1) We cluster embeddings used by TSDS into K=12 groups with spherical k-means (the CLASS label space also has size 12), and use the soft assignments of each sample to the centroids as its profile, with the temperature set so that their mean entropy matches that of the label profiles. This replaces only the representation. (2) We replace the deficit rS in Eq. (10) with the fixed target profile, so that the score no longer depends on the current selection. This replaces only the selection rule. The results are reported below:
These results suggest that both factors contribute: the dynamic rule adds 1.0 and 1.2 points over the static rule with clusters and with labels, and closed-set labels add 0.6 and 0.8 points over clusters under the static and the dynamic rule. Their combination (CLASS) achieves the best average performance and is the only variant that beats quality-only selection on every benchmark. Thank you again for your insightful comments. We have added these results and analyses to the main paper (Section 5.4, Table 4). The corresponding code, named
run_select_sep.py, is also uploaded in the Supplementary Material.A: We agree that aggregation alone does not explain the advantage. Rather, the advantage lies in the fact that our aggregated profile is low-dimensional and takes the form of a soft distribution, and that it captures which capabilities a task requires rather than which particular examples are drawn. This yields better robustness across target changes (scarcity and noise) as well as across target resampling (Appendix D.9). We have revised our wording accordingly in Sections 4.4 and 5.3. In addition, since the quality score and the capability score are obtained from a single prompt, our aggregated profile is easy to obtain and does not require computing gradients or embeddings for the entire source pool, as LESS and TSDS do.