- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes HARP, a training-based data selection method that provides a middle ground between existing training-free and training-based data selection approaches. It clusters training data into node-leaf groupings, after which some leaves are selected for evaluation and creating Bayes posteriors for the unmeasured data. Then, two algorithms HARP-C and HARP-E are proposed to select the final training set based on these measurements. The paper evaluates across a range of datasets and models, showing that HARP is more performant and efficient in aggregate compared to strong existing baselines. Ablations are also provided for the various hyperparameters involved in the approach.
Critical strengths
- The biggest strength of the approach lies in it's performance and efficiency. The approach performs better on aggregate compared to presented baselines, while resulting in a much smaller final training set.
- The approach itself is quite intuitive and sound, and hyperparameter ablations are provided, which is a good reference for practical use of this method.
Critical weaknesses
- My biggest concern with this paper lies in it's presentation. Section 3 as a whole is quite jargon-heavy and hard to follow. I would strongly recommend the authors to provide simpler, natural language descriptions of the approach in the main section of the paper, and improve the presentation of the mathematical details. For example, proper equations can be used to separate text and mathematical terminology. The main results table is also quite difficult to understand.
- While it is true that HARP-C finds a training subset with 9x fewer training examples than the 10k example baselines presented, the "leaf-level queries" are themselves more expensive than individual-example queries that just train on one example. To establish the practicality of the approach, a comparison of computation incurred during the entire HARP data selection process, as compared to the baselines, should be included. It might be the case that the end-to-end runtimes or GPU hours for HARP exceed even full fine-tuning, due to repeated evaluation on the full validation set.
- A lot of implementation details are missing in the manuscript. For example, how are the train set embeddings computed? What is the target number of child groups, kernel temperature and threshold used in the experiments?
Questions for authors
- Have the authors tried comparing against other standard choices for their design decisions, eg. k-means clustering for the "node-leaf groupings"?
- Do the authors have a recommendation about which of HARP-C and HARP-E is recommended for practical use? Are these approaches complementary in any way?
- The baselines are fixed at 10k examples, whereas HARP has the freedom to choose to select a smaller training set. I would be interested in seeing how well the baselines perform at smaller scale (eg. 1k examples) as well.
5 comments
Response to Reviewer 46bk (Part 2/3)
Weakness 3.
A lot of implementation details are missing in the manuscript. For example, how are the train set embeddings computed? What is the target number of child groups, kernel temperature and threshold used in the experiments?
A3. Thank you for pointing this out. Sections 3.2–3.3 describe the hierarchy construction and utility interpolation, while Appendix Table 4 reports the main hyperparameters. We will add the following implementation details to the main paper.
We use all-MiniLM-L6-v2 to compute normalized embeddings of instruction–response pairs. The target number of child groups is , where controls group granularity. Section 5.3, Figure 4 evaluates , showing that HARP maintains stable performance across different group sizes. We set to provide smooth similarity weights across representatives and to filter domains with negligible estimated utility changes.
Question 1.
Have the authors tried comparing against other standard choices for their design decisions, eg. k-means clustering for the "node-leaf groupings"?
A4. Thank you for the suggestion. HARP estimates the utility of unmeasured leaves from representative leaves, so keeping these leaves close helps reduce interpolation error. This motivates our use of farthest-first to cover diverse regions of the training pool, whereas k-means minimizes the average squared distance to cluster centers and may leave some regions poorly covered. Farthest-first is also computationally efficient, requiring time for distance calculations over embeddings of dimension and groups, compared with for full iterations of Lloyd's k-means.
Question 2.
Do the authors have a recommendation about which of HARP-C and HARP-E is recommended for practical use? Are these approaches complementary in any way?
A5. Thank you for asking for a practical choice. We conducted an additional analysis of the subsets selected by HARP-C and HARP-E in nine MMLU settings with seed 42, reported in Table R11. HARP-C selects fewer leaves and fewer training examples than HARP-E in all nine settings, and in six of them all of its leaves are also selected by HARP-E. Even so, HARP-C achieves higher accuracy in six settings, so the extra leaves selected by HARP-E mostly do not improve accuracy. This is consistent with HARP-C's design of limiting redundant positive contributions. For example, on Qwen3-4B/Self-Instruct, HARP-C selects 5 leaves and 2,162 examples, compared with HARP-E's 18 leaves and 10,000 examples. HARP-E includes 80% of HARP-C's selected leaves, while HARP-C achieves higher accuracy, 71.4 versus 70.3, with substantially less training data.
Table R11. Selected training data and MMLU accuracy for seed 42. Values are HARP-C / HARP-E. Bold marks the higher accuracy.
| Model / Dataset | Train Rows | Leaves | C in E Leaves (%) | Acc. (%) |
|---|---|---|---|---|
| Llama-3.1-8B / SI | 968 / 1,733 | 1 / 3 | 100 | 63.0 / 63.2 |
| Llama-3.1-8B / Wiz | 825 / 5,237 | 1 / 7 | 100 | 64.6 / 63.1 |
| Llama-3.1-8B / Tulu | 445 / 3,508 | 2 / 7 | 100 | 64.3 / 64.4 |
| Qwen3-8B / SI | 3,063 / 7,005 | 5 / 10 | 100 | 75.4 / 74.3 |
| Qwen3-8B / Wiz | 3,123 / 10,000 | 4 / 13 | 100 | 75.3 / 74.6 |
| Qwen3-8B / Tulu | 5,467 / 10,000 | 8 / 40 | 62.5 | 75.8 / 76.0 |
| Qwen3-4B / SI | 2,162 / 10,000 | 5 / 18 | 80.0 | 71.4 / 70.3 |
| Qwen3-4B / Wiz | 647 / 2,750 | 3 / 9 | 100 | 73.0 / 72.2 |
| Qwen3-4B / Tulu | 4,729 / 7,864 | 6 / 12 | 83.3 | 72.5 / 72.0 |
Based on these results, we recommend HARP-C as the default when dataset redundancy or quality is unknown, since it avoids overcounting redundant gains and is less likely to select noisy examples. HARP-E is more suitable when the dataset has low redundancy and reliable quality, because it can capture complementary gains from multiple useful leaves within each domain. We will add this analysis and these recommendations to the revision.
(Continued in Part 3/3.)
Response to Reviewer 46bk (Part 3/3)
Question 3.
The baselines are fixed at 10k examples, whereas HARP has the freedom to choose to select a smaller training set. I would be interested in seeing how well the baselines perform at smaller scale (eg. 1k examples) as well.
A6. Thank you for suggesting the 1k comparison. We use 10k as the default budget for all baselines for a fair comparison. The baselines use a fixed number of training examples, but the optimal budget for each method cannot be determined before selection. In contrast, HARP dynamically selects the subset size, treating the budget as an upper bound, and achieves better performance with fewer training examples.
We further evaluate LESS, NICE, and SHED at 1k, HARP-C's size, and HARP-E's size, as shown in Table R12. Compared with 10k examples, LESS decreases from 51.6 to 49.9 at 1k, 49.7 at HARP-C's size, and 50.5 at HARP-E's size. NICE improves from 59.3 to 62.1, 61.8, and 60.9, respectively, and SHED improves from 56.4 to 60.7, 61.5, and 59.8, respectively. However, all these results remain below HARP-C's 64.2 and HARP-E's 63.5, so HARP's advantage holds at smaller budgets as well. For the computational cost, please refer to A2.
Table R12. Dataset-size comparisons (bold: best; italic: second best)
| Method | Rows trained on | MMLU | ARC | GSM8K | MATH | Avg |
|---|---|---|---|---|---|---|
| LESS | 1k | 68.9 ± 0.4 | 84.7 ± 0.4 | 29.9 ± 1.4 | 15.9 ± 3.1 | 49.9 ± 1.3 |
| LESS | HARP-C's size | 68.0 ± 0.4 | 85.0 ± 0.4 | 30.5 ± 2.4 | 15.2 ± 3.1 | 49.7 ± 1.6 |
| LESS | HARP-E's size | 67.5 ± 0.6 | 83.8 ± 0.7 | 29.1 ± 1.8 | 21.4 ± 1.9 | 50.5 ± 1.3 |
| LESS | 10k | 66.1 ± 0.7 | 83.0 ± 0.8 | 33.6 ± 6.1 | 23.8 ± 1.7 | 51.6 ± 2.3 |
| NICE | 1k | 69.9 ± 0.5 | 86.2 ± 0.4 | 63.8 ± 2.0 | 28.3 ± 1.6 | 62.1 ± 1.1 |
| NICE | HARP-C's size | 69.9 ± 0.5 | 86.4 ± 0.4 | 62.1 ± 2.2 | 29.0 ± 1.9 | 61.8 ± 1.2 |
| NICE | HARP-E's size | 68.8 ± 0.7 | 85.7 ± 0.5 | 60.3 ± 3.9 | 28.7 ± 2.1 | 60.9 ± 1.8 |
| NICE | 10k | 68.6 ± 0.7 | 85.9 ± 0.4 | 55.1 ± 2.8 | 27.9 ± 1.5 | 59.3 ± 1.3 |
| SHED | 1k | 69.7 ± 0.9 | 85.0 ± 0.9 | 59.8 ± 5.0 | 28.3 ± 1.8 | 60.7 ± 2.1 |
| SHED | HARP-C's size | 69.6 ± 0.7 | 85.1 ± 1.0 | 62.5 ± 3.0 | 29.0 ± 2.1 | 61.5 ± 1.7 |
| SHED | HARP-E's size | 68.6 ± 0.9 | 84.5 ± 1.3 | 57.0 ± 5.6 | 29.1 ± 2.1 | 59.8 ± 2.5 |
| SHED | 10k | 67.8 ± 0.9 | 83.8 ± 1.1 | 47.3 ± 3.2 | 26.8 ± 1.3 | 56.4 ± 1.6 |
| HARP-C | Own | 70.6 ± 0.2 | 85.7 ± 0.5 | 68.2 ± 2.1 | 32.1 ± 1.9 | 64.2 ± 1.2 |
| HARP-E | Own | 69.9 ± 0.3 | 85.8 ± 0.5 | 64.1 ± 3.2 | 34.4 ± 1.5 | 63.5 ± 1.4 |
We thank the reviewer again for the constructive feedback.
Response to Reviewer 46bk (Part 1/3)
We thank the reviewer for the positive assessment of HARP's effectiveness, efficiency, and methodology, and for the constructive suggestions.
Weakness 1.
A1. Thank you for the suggestion. We will revise Section 3 and Table 1 accordingly to improve their readability.
Weakness 2.
A2. Thank you for requesting the full cost comparison. A leaf-level query does cost more than a single-example query, but HARP needs far fewer train–evaluate iterations. For the same M training examples, evaluating them individually requires M full-validation evaluations, while grouping them into Q≪M leaves requires only Q: Cindividual≈Ttrain(M)+M⋅Teval and Cleaf≈Ttrain(M)+Q⋅Teval. As shown in Table R9, validation accounts for 22.0% of HARP-C's total cost and 17.2% of HARP-E's, so evaluating examples individually would multiply the total cost by about 1+0.220(M/Q−1) and 1+0.172(M/Q−1), respectively; with M/Q=10, this is about 3.0× and 2.5×. Reducing the number of train–evaluate iterations therefore matters more than the cost of each individual query.
We conducted additional experiments that measure the end-to-end cost of each method in two parts: data selection and final fine-tuning. Costs are averaged over the 36 settings (3 models × 3 training sets × 4 target benchmarks) and reported in units of 1017 FLOPs, as shown in Table R10. Our cost breakdown further confirms HARP's efficiency. Including final fine-tuning, HARP-C/E cost 3.9/4.9, compared with 9.8 for SHED, 33.6 for LESS, 36.6 for NICE, and 27.4 for full fine-tuning. Thus, even after accounting for repeated full-validation evaluations, HARP remains substantially cheaper than the other train-based baselines and full fine-tuning.
Table R9. HARP cost breakdown (% of HARP's total cost)
Table R10. Mean selection and training costs (×1017 FLOPs)
(Continued in Part 2/3.)