- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TACS, a target-aware data selection method that constructs a short reference trajectory using only a compact target-validation proxy. Candidates are ranked by normalized loss reduction along this trajectory, enabling forward-only scoring and reuse of the same reference path across multiple candidate pools. The method is evaluated on synthetic, vision, and instruction-tuning benchmarks.
Critical strengths
-
The paper identifies reference-path construction as an important but underexplored factor in trajectory-based data selection and proposes a simple target-aligned alternative.
-
TACS provides substantial compute and storage savings while remaining competitive with strong trajectory-based baselines.
Critical weaknesses
-
The large-scale instruction-tuning gains are uneven. Averaged across pools, TACS improves TyDiQA substantially, but does not outperform the untrained base model on MMLU or BBH, making the claimed cross-task benefit less convincing.
-
TACS critically depends on the target proxy, but robustness to different proxy samples is not evaluated. This is especially concerning for TyDiQA, where the proxy contains only nine examples.
Questions for authors
-
How sensitive are the selected subsets and downstream results to different samples of the target proxy?
-
Why does TACS fail to improve over the base model on average for MMLU and BBH?
-
Do the reported efficiency gains remain similar under compute-matched lower-cost configurations of LESS and ToV?
5 comments
W1 / Q2: "Averaged across pools, TACS improves TyDiQA substantially, but does not outperform the untrained base model on MMLU or BBH." "Why does TACS fail to improve over the base model on average for MMLU and BBH?"
Response: This is a fair concern: the strong TyDiQA gains do not by themselves establish improvement across targets. We first compare every selector and full-pool training with the base model, averaged over the four pools of Table 1. None of the methods improves over the base model on average on MMLU or BBH, so the small gains are shared across methods.
| Average over four pools | Base | Random | LESS | ToV | TACS | Full pool |
|---|---|---|---|---|---|---|
| MMLU accuracy | 56.22 | 55.72 | 55.69 | 54.93 | 56.08 | 54.42 |
| BBH accuracy | 48.61 | 47.40 | 47.55 | 47.86 | 47.85 | 47.34 |
To understand why, we analyzed the base model and 48 trained runs per task, and asked whether these tasks offer the same answer-behavior gains as TyDiQA. On TyDiQA, about 84% of base-model F1 loss is associated with answers that are not passage spans or contain extra text. BBH already provides three demonstrations, and only 10.6% of base errors lack the expected answer marker; training leaves that count almost unchanged (59 versus 58.7). MMLU scores option likelihoods after five demonstrations, so changing generated-answer format cannot improve its metric. On BBH, training corrects 96 base errors but breaks 107 previously correct answers on average, leaving little net gain. At our scale—5% subsets and four epochs of LoRA training—these results suggest that general instruction data more readily improves answer behavior than supplies missing knowledge or symbolic skills. This analysis appears in Appendix J.5, Table 14.
To separate content from answer format, we add a control: training on 1,493 held-out MMLU questions, and on 1,493 puzzles that use the same MMLU answer format. Format-only training gives little change, while matched-content training raises Llama-2-7B accuracy by 4.75 points. The new control appears in Appendix J.5, Table 15.
| MMLU accuracy | Llama-3.2-3B | Llama-2-7B |
|---|---|---|
| Base model | 56.24 | 45.53 |
| Format-only training | 55.92 | 45.43 |
| TACS on a generic pool | 55.82 | 45.20 |
| Matched-content training | 56.99 | 50.28 |
This supports the distinction between improving answer behavior and supplying task-relevant content. We acknowledge that TACS trails LESS on COT → BBH (46.11 versus 48.12). Across our experiments, we observe a general pattern: TACS is less stable on targets with little headroom for general instruction data and more competitive where that headroom is larger. We have added this finding to the paper's limitations, in the conclusion and in Appendix B, in blue. The paper presents TACS as an efficient selector with competitive downstream performance.
Q3: "Do the reported efficiency gains remain similar under compute-matched lower-cost configurations of LESS and ToV?"
Response: This is an important question, because our reported savings are measured against the standard configurations of LESS and ToV, and a cheaper configuration could narrow them. We add two measurements for a lower-cost LESS configuration that keeps the method intact and uses one warmup checkpoint. First, we measure its selection quality on Dolly → TyDiQA. Second, we time both methods in one job per pool, on the same GPU and the same 3,000 sampled candidates. The cheaper LESS keeps a quality comparable to standard LESS, and TACS remains cheaper per candidate up to about three targets per pool. The new results appear in Appendix J.6, Table 16.
| Dolly → TyDiQA | F1, validation-selected / final |
|---|---|
| LESS, 4 checkpoints | 61.50 / 62.44 |
| LESS, 1 checkpoint | 61.28 / 64.28 |
| TACS | 63.35 / 63.63 |
| Compute per candidate (GPU-seconds) | Dolly | COT | Flan v2 | OASST1 |
|---|---|---|---|---|
| TACS, per target (2 forward passes) | 0.039 | 0.042 | 0.056 | 0.046 |
| LESS, per checkpoint (1 per-example gradient) | 0.171 | 0.159 | 0.187 | 0.170 |
| LESS, standard (4 checkpoints) | 0.68 | 0.63 | 0.75 | 0.68 |
| Adding one 100k-row pool | Training on the pool | Candidate compute (GPU-hours) | Stored per candidate |
|---|---|---|---|
| TACS, 1 target | none | 1.3 | 1 score |
| TACS, 3 targets | none | 3.8 | 3 scores |
| LESS, 1 checkpoint | warmup on 5% of the pool | 4.8 | 8,192 values |
| LESS, 4 checkpoints | warmup on 5% of the pool | 19.1 | 32,768 values |
The advantage comes from how TACS scales with the pool. TACS never trains on the candidate pool: its reference path is built once per target from the proxy and reused for every pool, so a new pool costs two forward passes per candidate and target. LESS must warm up on each new pool and compute a per-example gradient for every candidate at every checkpoint. At three targets, TACS needs 4.5 to 5.9 times less compute per candidate than standard LESS, and 1.1 to 1.5 times less than one-checkpoint LESS, before the LESS warmup. The gap closes as more targets share one pool, because LESS reuses its stored gradients for every target; our limitations section (Appendix B) states this trade-off. For ToV, per-candidate scoring is 120.5 of its 124.7 GPU-hours in our benchmark, at about 2.4 times the TACS cost per candidate and target. Because almost all of its cost is per-candidate scoring, a cheaper warmup would barely change it; only scoring at fewer checkpoints would, and we did not test that configuration. We conclude that, against a cheaper LESS, TACS keeps a lower marginal scoring cost when new pools are screened for a few targets, together with its storage advantage under every configuration.
We thank the reviewer for their thoughtful questions and hope these additions address their concerns. We would be happy to clarify any remaining points and would appreciate their reconsideration in light of the new evidence.
Thank you for the careful review and the precise questions. They led us to test how TACS depends on the proxy sample, why gains over the base model are small on MMLU and BBH, and how its cost compares with cheaper configurations of the baselines. We have added the new experiments below to the revised manuscript in blue.
Response: We agree that this question matters, because TACS builds its reference path from the proxy alone, and nine examples is a small sample. We add a proxy-sampling study on Dolly → TyDiQA. We draw five new nine-example proxies, one training question per language, with no overlap with the original proxy or the evaluation set. We also build proxies that cover only three languages or one language. For each proxy, we rebuild the reference, rescore the full Dolly pool, select the top 5%, and retrain, with everything else fixed. Test scores use the fine-tuning checkpoint with the highest validation score ("validation-selected") and the last checkpoint ("final"). Across the five new proxies, F1 varies by about one point and stays more than 11 points above Random. The new results appear in Appendix J.4, Table 13.
The selected subsets have a Jaccard index of 0.38 to 0.54 with the original selection, so 55 to 70% of the selected examples are shared, and different proxies select partly different data of similar value. Proxies that cover part of the target lose at most 3 points, and a language does not score consistently higher when the proxy covers it. On Dolly → TyDiQA, downstream quality therefore stays stable across proxy samples, and even a narrow proxy keeps most of the gain.