- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ACE, a framework for reducing the cost and redundancy of data selection for language model training. ACE groups examples using projected gradients from a small proxy model and reuses these clusters throughout training. At each selection round, it scores a few randomly sampled members using an existing attribution method and takes their average to estimate the cluster’s utility. Higher-scoring clusters receive more training samples, which are drawn uniformly within each cluster. The authors prove that this estimator is unbiased for the cluster’s mean attribution score and derive a variance bound linking its reliability to the number of representatives and within-cluster gradient similarity.
Experiments apply ACE to LESS, TracIn, and loss-based selection for instruction tuning, and to OPUS for pre-training. The paper reports higher average downstream scores than the corresponding sample-level methods, with end-to-end speedups of up to 33.8× for instruction tuning and 8.86× for pre-training. Using approximately 10% of the candidate data, selected ACE variants also achieve higher average benchmark performance than full-data fine-tuning on Llama-3.2-3B and Mistral-7B.
Critical strengths
- Treating groups of similar learning signals as the units of selection is well motivated. It offers a way to balance target relevance with coverage when many individually high-scoring examples provide overlapping training information. Estimating cluster utility from a few representatives also reduces expensive per-example scoring, making the selection process more computationally efficient.
- I find the distinction between reusable data structure and changing training utility appealing. It provides a practical route to keeping selection adaptive while reducing repeated attribution work.
- The cross-model and cross-checkpoint analyses directly examine whether fixed clusters retain coherent learning signals. Evaluating them with independently computed gradients makes this evidence more convincing.
- The gains in matched post-training comparisons, including the additional OOD averages, are encouraging. Testing multiple attribution rules and extending the approach to pre-training also strengthens the case for its practical usefulness.
Critical weaknesses
- The current experiments do not clearly separate the contributions of the main design choices. Figure 3b compares clustering features, but representative estimation and budget allocation are not independently evaluated through downstream performance. Comparisons with exact cluster means and uniform allocation over the same clusters would help explain which components drive the gains.
- The method’s sensitivity to the clustering model is not yet fully understood. Figure 3b compares several representations, with Qwen2.5-0.5B gradients giving the strongest downstream results. However, it remains unclear why this model produces more effective clusters and whether comparably small models from other families would work similarly well. Further analysis and a comparison with another small proxy would help establish the robustness of the reported gains.
- The reported speedups would be more convincing with a clearer breakdown of their sources. Preprocessing is included, but attribution scoring, selection, and training are largely grouped together. Timings for each stage and actual gradient evaluation counts would help clarify the contributions of reduced scoring, batching, and cache reuse.
- The practical advantage over other selectors is not yet clear under matched total compute. Several baselines lack runtime measurements in Tables 2–3. Comparing performance at equal GPU-hour budgets, including a relevant efficient method such as ClusterUCB, would help establish whether ACE is preferable to cheaper selection strategies or simply training on more data.
Questions for authors
Which components account for ACE’s downstream gains? On one post-training backbone with LESS (or any other method you prefer), could you keep the clustering and training budget fixed and compare: (a) default ACE, estimating each cluster’s mean utility from up to eight representatives, (b) the same selection rule using exact cluster means computed from all members, and (c) equal sampling probabilities across clusters, without utility-based weighting? Sampling within each cluster should remain uniform in all three variants. Comparing (a) with (b) would assess the representative approximation, while comparing (b) with (c) would test the value of utility-based budget allocation.
Why do results vary with the clustering model, and how robust are ACE’s gains to this choice? Figure 3b compares several clustering representations, with Qwen2.5-0.5B gradients giving the strongest downstream results. Could you analyze why these gradients produce more effective clusters than those from Mistral-7B, and test one additional similarly sized model from another family under the same settings to show if the method is sensitive?
Which stages contribute most to the reported speedups? For one reported post-training comparison, could you provide corresponding timings for ACE and its sample-level counterpart, separating preprocessing, attribution scoring, selection, and training?
Does ACE retain an advantage under the same total compute budget? On one post-training backbone, could you compare ACE with an existing baseline such as Random or TSDS at matched total GPU-hour budgets, including selection costs and allowing any savings to support additional training? This would clarify whether ACE provides a better practical use of compute. If a ClusterUCB implementation is readily available, it would also be a useful comparison.
3 comments
To Reviewer cMUv(2/2)
W4 & Q4. Matched Total Compute
We agree that inexpensive selection methods should be allowed to use their saved computational budget for additional training.
We therefore compare ACE against Random and TSDS under approximately the same total runtime, allowing both baselines to continue training with fresh samples:
| Method | Runtime | Training steps | Avg ↑ |
|---|---|---|---|
| Cluster-LESS | 1h05m41s | 420 | 59.35 |
| Random | 1h04m14s | 1,377 | 59.15 |
| TSDS | 1h04m46s | 1,141 | 58.76 |
| ClusterUCB | 7h47m53s | 420 | 58.45 |
Although Random and TSDS perform 3.28× and 2.72× more training steps, respectively, ACE achieves the highest observed average score under comparable total runtime.
This suggests that improved data selection quality can provide benefits that are not fully recovered by simply increasing the number of training steps in this experimental setting.
We additionally evaluate an ClusterUCB baseline, which achieves an average score of 58.45 after approximately 7 h 48 min with 420 training steps`. We report this result separately because its runtime can not compute-matched to approximately 1 hour.
These results support ACE's practical quality–compute tradeoff, rather than claiming that ACE has the lowest selection overhead.
We sincerely appreciate your careful review and constructive suggestions. We hope these additional experiments directly address your concerns and help strengthen your assessment of our work. We would be happy to provide any further clarification.
Thank you author for the detailed response and additional experiments. The new results address my concerns, and I encourage the authors to incorporate them into the revised manuscript. I am happy to maintain my positive score.
Our new ablations confirm the value of representative-based estimation and utility-weighted allocation. We further provide a detailed runtime breakdown, demonstrate proxy-model robustness, and show competitive performance under matched compute budgets.
To Reviewer cMUv(1/2)
We thank you for the detailed and constructive suggestions. We have conducted additional component ablations, proxy-model comparisons, runtime analysis, and matched-compute evaluations to address your concerns.
W1 & Q1. Component Ablation
Following your suggestion, we compare three variants under the same clustering and training budget:
A vs. B: Representative-based estimation achieves performance close to exact cluster-mean estimation (only 0.28 points lower), while being approximately
9.8×faster end-to-end.B vs. C: Utility-weighted cluster allocation improves average performance by
2.70points over uniform allocation.These results help disentangle the contributions of the two main components: a small number of representatives achieves competitive downstream performance at substantially lower cost, while target-conditioned utility weighting contributes considerably beyond clustering alone.
W2 & Q2. Proxy-model Robustness
We extend gradient clustering to alternative models with different family and projection dimensions, evaluating the resulting clusters using independently computed Mistral-7B gradients across training checkpoints.
Notably, SmolLM2-360M produces more coherent clusters than Qwen2.5-0.5B, and improves downstream average performance from 59.35 to 59.99.
Under the same 2,048-dimensional projection, Qwen2.5-0.5B achieves higher intra-cluster similarity than Qwen2.5-1.5B (0.5267 vs. 0.5038) and Mistral-7B (0.3365). However, increasing the projection dimension to 16,384 improves Mistral-7B's similarity to 0.5815, surpassing Qwen2.5-0.5B. Interestingly, SmolLM2-360M, which is even smaller than Qwen2.5-0.5B, achieves a higher similarity of 0.5865 at only 2,048 dimensions.
These results suggest that aggressive dimensionality reduction may cause substantial information loss for larger models, potentially offsetting their advantages in representation quality. This may explain the weaker performance of Mistral-7B at 2,048 dimensions. Nevertheless, gradient extraction from Mistral-7B and high-dimensional clustering are considerably much more expensive than using lightweight proxies.
Therefore, we choose small-model projected gradients to balance clustering quality and computational cost. These findings also demonstrate that ACE is not tied to the Qwen model family.
W3 & Q3. Where Does the Speedup Come From?
Following your suggestion, we provide the requested runtime breakdown on Mistral-7B:
This corresponds to approximately
24.3×end-to-end speedup in this configuration.The dominant saving comes from attribution computation, which decreases from over 26 hours to approximately 9 minutes. Thus, ACE's additional preprocessing cost is substantially outweighed by reduced target-model attribution cost.