HARP: Efficient Data Selection for Finetuning Large Language Models

Finetuning data selection requires balancing two competing goals: selecting examples that improve the downstream objective, and doing so without repeatedly finetuning models. Train-free selectors are scalable but rely on proxies such as embedding similarity or clustering, which may not match the target objective. Train-based selectors better reflect downstream utility through gradient signals, subset evaluation, or Shapley attribution, but require many costly train–evaluate iterations. We propose Hierarchical Active Region Pruning (HARP), an efficient train-based selector that preserves downstream alignment while reducing selection cost. HARP organizes the training pool into a node–leaf hierarchy, evaluates only representative leaves, and infers unmeasured utilities with empirical Bayes posteriors. It then selects data using two complementary envelopes: HARP-C, which conservatively controls redundancy, and HARP-E, which additively rewards complementary regions. We theoretically show that, under local smoothness and bounded estimation error, HARP controls selection error while reducing train–evaluate cost. Across $3$ base models, $3$ finetuning datasets and $4$ reasoning benchmarks ($36$ settings), HARP variants take the per-cell best among ranked methods on $23$ cells and achieve $36$-setting means of $64.2$ / $63.5$, compared with $59.3$ for the strongest baseline, NICE. HARP-C trains on roughly $9\times$ fewer examples than the budgeted baselines and $75\times$ fewer than full finetuning, combining higher mean accuracy with a smaller mean training-set size than every ranked baseline.