acceptodds
Under review as a conference paper at ICLR 2027

ACE: Lifting Data Attribution from Samples to Clusters via Intrinsic Gradient Similarity

Abstract

Data quality has become increasingly central to language model training, making effective data selection important. Yet many influential data-selection methods score each sample independently at high cost. Such pointwise scoring overlooks similarities in gradient behavior across samples, leading to limited scalability and redundant selection. To address these limitations, we introduce **ACE** (Data **A**ttribution via **C**luster-Level **E**stimation), a scalable framework that exploits intrinsic gradient similarity to lift both utility estimation and budget allocation from individual samples to clusters. ACE groups samples into clusters by gradient similarity, scores a few representatives per cluster to estimate cluster-level utility, and allocates the selection budget across clusters. We prove that the resulting estimator is unbiased and has controlled variance, providing a principled basis for reliable cluster-level attribution. Across post-training experiments, ACE achieves an end-to-end speedup of up to **33.8×** and consistently outperforms its sample-level counterparts. Using only **10%** of the data, it even surpasses **full-data** training on Llama-3.2-3B and Mistral-7B. Pre-training experiments further validate its scalability to larger data pools: ACE improves upon sample-level OPUS, achieves an **8.86×** speedup, and delivers the strongest overall results among the baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
training data attribution, data selection, large language models, clustering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cMUvOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ACE, a framework for reducing the cost and redundancy of data selection for language model training. ACE groups examples using projected gradients from a small proxy model and reuses these clusters throughout training. At each selection round, it scores a few randomly sampled members using an existing attribution method and takes their average to estimate the cluster’s utility. Higher-scoring clusters receive more training samples, which are drawn uniformly within each cluster. The authors prove that this estimator is unbiased for the cluster’s mean attribution score and derive a variance bound linking its reliability to the number of representatives and within-cluster gradient similarity.

Experiments apply ACE to LESS, TracIn, and loss-based selection for instruction tuning, and to OPUS for pre-training. The paper reports higher average downstream scores than the corresponding sample-level methods, with end-to-end speedups of up to 33.8× for instruction tuning and 8.86× for pre-training. Using approximately 10% of the candidate data, selected ACE variants also achieve higher average benchmark performance than full-data fine-tuning on Llama-3.2-3B and Mistral-7B.

Critical strengths
  1. Treating groups of similar learning signals as the units of selection is well motivated. It offers a way to balance target relevance with coverage when many individually high-scoring examples provide overlapping training information. Estimating cluster utility from a few representatives also reduces expensive per-example scoring, making the selection process more computationally efficient.
  2. I find the distinction between reusable data structure and changing training utility appealing. It provides a practical route to keeping selection adaptive while reducing repeated attribution work.
  3. The cross-model and cross-checkpoint analyses directly examine whether fixed clusters retain coherent learning signals. Evaluating them with independently computed gradients makes this evidence more convincing.
  4. The gains in matched post-training comparisons, including the additional OOD averages, are encouraging. Testing multiple attribution rules and extending the approach to pre-training also strengthens the case for its practical usefulness.
Critical weaknesses
  1. The current experiments do not clearly separate the contributions of the main design choices. Figure 3b compares clustering features, but representative estimation and budget allocation are not independently evaluated through downstream performance. Comparisons with exact cluster means and uniform allocation over the same clusters would help explain which components drive the gains.
  2. The method’s sensitivity to the clustering model is not yet fully understood. Figure 3b compares several representations, with Qwen2.5-0.5B gradients giving the strongest downstream results. However, it remains unclear why this model produces more effective clusters and whether comparably small models from other families would work similarly well. Further analysis and a comparison with another small proxy would help establish the robustness of the reported gains.
  3. The reported speedups would be more convincing with a clearer breakdown of their sources. Preprocessing is included, but attribution scoring, selection, and training are largely grouped together. Timings for each stage and actual gradient evaluation counts would help clarify the contributions of reduced scoring, batching, and cache reuse.
  4. The practical advantage over other selectors is not yet clear under matched total compute. Several baselines lack runtime measurements in Tables 2–3. Comparing performance at equal GPU-hour budgets, including a relevant efficient method such as ClusterUCB, would help establish whether ACE is preferable to cheaper selection strategies or simply training on more data.
Questions for authors

Which components account for ACE’s downstream gains? On one post-training backbone with LESS (or any other method you prefer), could you keep the clustering and training budget fixed and compare: (a) default ACE, estimating each cluster’s mean utility from up to eight representatives, (b) the same selection rule using exact cluster means computed from all members, and (c) equal sampling probabilities across clusters, without utility-based weighting? Sampling within each cluster should remain uniform in all three variants. Comparing (a) with (b) would assess the representative approximation, while comparing (b) with (c) would test the value of utility-based budget allocation.

Why do results vary with the clustering model, and how robust are ACE’s gains to this choice? Figure 3b compares several clustering representations, with Qwen2.5-0.5B gradients giving the strongest downstream results. Could you analyze why these gradients produce more effective clusters than those from Mistral-7B, and test one additional similarly sized model from another family under the same settings to show if the method is sensitive?

Which stages contribute most to the reported speedups? For one reported post-training comparison, could you provide corresponding timings for ACE and its sample-level counterpart, separating preprocessing, attribution scoring, selection, and training?

Does ACE retain an advantage under the same total compute budget? On one post-training backbone, could you compare ACE with an existing baseline such as Random or TSDS at matched total GPU-hour budgets, including selection costs and allowing any savings to support additional training? This would clarify whether ACE provides a better practical use of compute. If a ClusterUCB implementation is readily available, it would also be a useful comparison.

3 comments
AuthorsOpenReview ↗

Our new ablations confirm the value of representative-based estimation and utility-weighted allocation. We further provide a detailed runtime breakdown, demonstrate proxy-model robustness, and show competitive performance under matched compute budgets.

To Reviewer cMUv(1/2)

We thank you for the detailed and constructive suggestions. We have conducted additional component ablations, proxy-model comparisons, runtime analysis, and matched-compute evaluations to address your concerns.

W1 & Q1. Component Ablation

Following your suggestion, we compare three variants under the same clustering and training budget:

VariantRuntimeAvg ↑
A. ACE (8 representatives)1h05m41s59.35
B. Exact cluster means10h39m59.63
C. Uniform cluster allocation22m56.93

A vs. B: Representative-based estimation achieves performance close to exact cluster-mean estimation (only 0.28 points lower), while being approximately 9.8× faster end-to-end.

B vs. C: Utility-weighted cluster allocation improves average performance by 2.70 points over uniform allocation.

These results help disentangle the contributions of the two main components: a small number of representatives achieves competitive downstream performance at substantially lower cost, while target-conditioned utility weighting contributes considerably beyond clustering alone.

W2 & Q2. Proxy-model Robustness

We extend gradient clustering to alternative models with different family and projection dimensions, evaluating the resulting clusters using independently computed Mistral-7B gradients across training checkpoints.

Clustering representationDimensionIntra-similarity ↑
Qwen2.5-0.5B gradients2,0480.5267
Qwen2.5-0.5B gradients4,0960.5335
Qwen2.5-1.5B gradients2,0480.5038
Mistral-7B gradients2,0480.3365
Mistral-7B gradients16,3840.5815
SmolLM2-360M gradients2,0480.5865
RepresentationMMLUBBHTyDiQAAvg ↑
EmbeddingGemma 260.9855.1959.9258.70
Qwen2.5-0.5B gradients60.8156.0261.2259.35
SmolLM2-360M-Instruct gradients61.4559.1759.3559.99

Notably, SmolLM2-360M produces more coherent clusters than Qwen2.5-0.5B, and improves downstream average performance from 59.35 to 59.99.

Under the same 2,048-dimensional projection, Qwen2.5-0.5B achieves higher intra-cluster similarity than Qwen2.5-1.5B (0.5267 vs. 0.5038) and Mistral-7B (0.3365). However, increasing the projection dimension to 16,384 improves Mistral-7B's similarity to 0.5815, surpassing Qwen2.5-0.5B. Interestingly, SmolLM2-360M, which is even smaller than Qwen2.5-0.5B, achieves a higher similarity of 0.5865 at only 2,048 dimensions.

These results suggest that aggressive dimensionality reduction may cause substantial information loss for larger models, potentially offsetting their advantages in representation quality. This may explain the weaker performance of Mistral-7B at 2,048 dimensions. Nevertheless, gradient extraction from Mistral-7B and high-dimensional clustering are considerably much more expensive than using lightweight proxies.

Therefore, we choose small-model projected gradients to balance clustering quality and computational cost. These findings also demonstrate that ACE is not tied to the Qwen model family.

W3 & Q3. Where Does the Speedup Come From?

Following your suggestion, we provide the requested runtime breakdown on Mistral-7B:

StageCluster-LESSSample-LESS
Feature extraction13m48s—
Clustering22m37s—
Attribution9m11s26h15m30s
Selection16s2m
Training19m49s19m49s
Total1h05m41s26h37m19s

This corresponds to approximately 24.3× end-to-end speedup in this configuration.

The dominant saving comes from attribution computation, which decreases from over 26 hours to approximately 9 minutes. Thus, ACE's additional preprocessing cost is substantially outweighed by reduced target-model attribution cost.

To Reviewer cMUv(2/2)

W4 & Q4. Matched Total Compute

We agree that inexpensive selection methods should be allowed to use their saved computational budget for additional training.

We therefore compare ACE against Random and TSDS under approximately the same total runtime, allowing both baselines to continue training with fresh samples:

MethodRuntimeTraining stepsAvg ↑
Cluster-LESS1h05m41s42059.35
Random1h04m14s1,37759.15
TSDS1h04m46s1,14158.76
ClusterUCB7h47m53s42058.45

Although Random and TSDS perform 3.28× and 2.72× more training steps, respectively, ACE achieves the highest observed average score under comparable total runtime.

This suggests that improved data selection quality can provide benefits that are not fully recovered by simply increasing the number of training steps in this experimental setting.

We additionally evaluate an ClusterUCB baseline, which achieves an average score of 58.45 after approximately 7 h 48 min with 420 training steps`. We report this result separately because its runtime can not compute-matched to approximately 1 hour.

These results support ACE's practical quality–compute tradeoff, rather than claiming that ACE has the lowest selection overhead.

We sincerely appreciate your careful review and constructive suggestions. We hope these additional experiments directly address your concerns and help strengthen your assessment of our work. We would be happy to provide any further clarification.

Thank you author for the detailed response and additional experiments. The new results address my concerns, and I encourage the authors to incorporate them into the revised manuscript. I am happy to maintain my positive score.

Reviewer_v2oaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a data selection method ACE, which incorporates clustering into conventional utility-score-based methods, where utility scores can be loss values, gradient similarities, and trajectories, etc. Through introducing clustering, it significantly reduces the computational cost for estimation and selection. ACE shows obvious effectiveness in both post-training and pre-training settings.

Critical strengths
  1. The writing framework is clear and easy to follow.
  2. Comprehensive experiments to support its claims.
Critical weaknesses
  1. Limited novelty and unclear boundaries of when clustering helps. The proposed method appears incremental to existing utility-score-based methods and does not fundamentally change how data utility is defined. Instead, it inserts a clustering stage before applying existing attribution score, and further replacing large models with proxy models to save computational costs. Clustering serves as an implicit deduplication or diversity mechanism. For example, suppose 10% of the candidate data are highly aligned with the target, while the remaining 90% are irrelevant, and clustering perfectly separates the relevant 10% into one of ten clusters. A sample-level selector would already concentrate its budget on the relevant examples. In this case, clustering fails to achieve its goal of deduplication, and it seems that this method only works when the target itself requires multiple distinct data modes. However, ACE still represents the entire target set by a single average gradient direction, which cannot demonstrate which target modes remain uncovered. A more systematic characterization of the data distributions under which cluster-level allocation is beneficial would strengthen the paper.
  2. Insufficient analysis of the selection budget. The experiments mainly use a fixed 10% selection budget, without systematically analyzing performance across budgets. A more principled approach could choose the budget based on the target-distribution coverage criterion, followed by empirical validation.
Questions for authors

The experiments show that small-model gradient clusters remain coherent under larger models, but this demonstrates transfer of candidate–candidate geometry, not necessarily target-conditioned utility. Could the authors report the correlation between small- and large-model utility scores? If small-model utilities transfer well, why not directly use the proxy model for utility estimation? If they do not, why does candidate–candidate gradient similarity transfer while candidate–target alignment does not?

6 comments
AuthorsOpenReview ↗

We highlight ACE's key innovation: separating reusable gradient-based data structures from dynamic target-conditioned utility estimation, with theoretical guarantees connecting them through unbiasedness and variance control. Additional experiments validate its robustness across selection budgets, explore a more principled approach to budget selection, and demonstrate the importance of target-model attribution.

To Reviewer v2oa (1/2)

We thank you for the insightful comments. We would like to clarify ACE's theoretical motivation, applicability, and the distinction between proxy-derived structure and target-conditioned utility.

W1. Novelty and Applicability

ACE is not simply clustering followed by existing attribution. Its central contribution is to divide reusable candidate structure from dynamic target-conditioned utility, and connect them through a theoretically justified cluster-level estimator.

Our theorem establishes unbiased cluster-mean utility estimation, with variance controlled by intra-cluster gradient coherence under the stated assumptions: Var(μ^k)≲∥hT∥2Hk2m(1−ρk).\mathrm{Var}(\hat{\mu}_k) \lesssim \frac{\|h_T\|^2 H_k^2}{m}(1-\rho_k). Thus, high gradient coherence provides a principled basis for reducing attribution computation using a few representatives.

Regarding your example: ACE does not enforce uniform allocation across clusters. When target-relevant examples are concentrated in one cluster, utility-weighted allocation prioritizes that cluster while maintaining some coverage of other regions.

We acknowledge that clustering may not always improve selection quality. ACE is particularly beneficial when training data contain redundant learning signals across multiple target-relevant regions, balancing relevance, diversity, and attribution cost. As shown in our diversity experiments, ACE achieves broader target-set coverage rather than concentrating on a few dominant regions, potentially enabling more balanced gradient updates and downstream improvements.

Our goal is not to universally replace sample-level selection, but to lift the often-overlooked relationships among training examples to scale attribution-based data selection to larger datasets and models in pre training , beyond the small-scale, fine-grained optimization settings of much prior work.

W2. Selection Budget

Following your suggestion, we extend our experiments beyond the original 10% budget:

BudgetLESS AvgCluster-LESS AvgImprovement
5%55.4657.83+2.37
10%58.4159.35+0.94
20%58.4859.45+0.97
50%58.3758.76+0.39

ACE consistently outperforms sample-level LESS across all four budgets, demonstrating that its effectiveness is not limited to 10%. The highest observed performance occurs at 20%, while selecting more data does not necessarily improve performance.

We also examine the target validation trajectory during extended training:

StepTarget lossGradient norm
03.8326150.71
5002.7467131.59
1,0002.6517104.27
2,0003.1738170.47
4,0003.2277171.83

Both metrics reach their observed minima at step 1,000, which corresponds to approximately 20% of the data budget. This is consistent with our budget experiments, where Cluster-LESS also achieves its highest performance at 20%. These results suggest that the target validation loss and gradient norm may provide a more principled approach for identifying a suitable selection budget.

To Reviewer v2oa (2/2)

Q1. Why Not Directly Use Proxy-model Utility?

Our results suggest that small-model utility scores exhibit a certain degree of transferability to larger models, but remain a low-cost approximation. Using gradients from the actual model being trained provides more accurate and precise alignment with the target objective.

Candidate–candidate gradient similarity describes the internal structure of training examples, whereas candidate–target alignment additionally depends on the validation gradient and the current optimization state of the target model.

Therefore, even when candidate geometry is preserved across models, their target-conditioned utility rankings need not agree.

This motivates our separation:

  • Proxy model: discovers reusable candidate gradient structure.
  • Target model: computes utility aligned with its actual training objective and state.
  • Theorem: guarantees unbiased representative estimation with controlled variance when target-model intra-cluster coherence is sufficiently high.

To investigate this further, we conduct experiments that preserve the original clustering structure while replacing only the utility-scoring model with Qwen2.5-0.5B, Qwen2.5-1.5B, or Qwen2.5-3B. We also evaluate a baseline that selects training data directly using the utility scores from Qwen2.5-0.5B in a single selection stage.

Utility-scoring modelMMLUBBHTyDiQAAvg ↑
Qwen2.5-0.5B (one-shot selection)60.3851.3961.5557.77
Qwen2.5-0.5B61.9653.6160.4858.68
Qwen2.5-1.5B62.0452.9661.5458.85
Qwen2.5-3B60.3557.2259.4759.02
Original Cluster-LESS (Mistral-7B)60.8156.0261.2259.35

Proxy-only one-shot selection underperforms the cluster-based configurations, while larger proxy scoring models progressively approach the performance obtained using Mistral-7B for attribution.

Moreover, utility scores computed by the proxy and target models exhibit weak positive Spearman correlations (0.051–0.151), while their selected sample sets show limited Jaccard overlap (6.02%–11.60%). These results suggest that utility information is transferable to some extent across models, but the agreement remains limited, highlighting the importance of using the actual target model for more accurate target-conditioned utility estimation.

Together, these findings support our central design: small models identify useful candidate structure, while the model being trained evaluates actual target-conditioned utility. The key requirement is sufficient intra-cluster gradient coherence under the target model, which controls the estimation variance in our theoretical analysis, rather than cross-model utility agreement.

We sincerely appreciate your thoughtful comments and would be happy to clarify any remaining concerns. We hope these additional results help provide a clearer understanding of ACE's motivation and effectiveness.

Reviewer_v2oaOpenReview ↗

Response to Author Rebuttal by Reviewer v2oa

Thank you for the detailed responses. My concerns in Q1 and W2 are addressed. However, the response still does not explain then cluster-level allocation improves selection quality and when it does not quantitively. I think this is the central question the paper should answer. Regarding the example in W1, the author's response confirms that ACE swould still allocate part of the budget to irrelevant clusters, suggesting that cluster-level diversity is not necessarily beneficial by itself. Considering the above question, I decide to maintain my score. I sincerely encourage the authors to characterize, theoretically or throught controlled experiments, the conditions under which cluster-level allocation helps or hurts. This would stronger the paper's claims and bring more insights to readers.

Reviewer_Q3MpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a core issue in data attribution methods for data selection and proposes a clustering-based solution to fix the issue. Most data attribution methods compute a score for each training sample, this method is both expensive and often gives examples that are directionally similar. The proposed ACE method clusters examples based on gradient similarity (using a small Qwen 0.5B) and uses a representative sample from each cluster for computing cluster utility score. Within each selected cluster, examples are sampled uniformly.

For post-training data selection, experiments on multiple data attribution methods (TracIn, Loss and LESS) show that the cluster-based method is provides significant efficiency gains while maintaining performance. Similar gains were shown on pretraining selection experiments. My main concerns are about the gradient-based clustering using Qwen-2.5-0.5B (see my questions in the weaknesses section).

Critical strengths
  • Compared to computing attribution score for each sample in the training data, the cluster-based method as expected shows significant efficiency gains for both post-training and pre-training.
  • This paper targets a core inefficiency in per-example based attribution, and shows clear gains by clustering and sampling data based on these clusters.
Critical weaknesses
  • I didn't find the details for how exactly Qwen-2.5-0.5B gradients are computed for clustering? One thing I am not clear about is how the gradients computed using Qwen correlate with the actual model that's being trained. What if these are not aligned? I'd like to see more discussion here. Additionally, the analysis in Figure 3 includes a comparison of Qwen-2.5-0.5B against Mistral-7B. Why only Mistral-7B?Possibly comparing different Qwen 2.5 model sizes for gradient-based clustering can be more useful.
  • As training progresses, do you recompute the gradients or do you just compute them once offline using Qwen and use them across all epochs?
Questions for authors
  • It would be nice to see more embedding based similarity methods in Figure 3 (e.g., embeddinggemma-2?)
  • In table 1, I'd recommend stating what K and m are. This became clear much later in the text.
  • Why uniform sampling from within a cluster?
3 comments
AuthorsOpenReview ↗

We clarify our clustering and sampling procedures. Additional experiments compare multiple embedding models and gradient proxies, demonstrating the effectiveness and robustness of gradient-based clustering.

To Reviewer Q3Mp:

We thank you for the positive assessment and constructive suggestions. We address each concern below.

W1. Gradient Clustering and Proxy-model Choice

Clustering procedure. We first compute per-example gradients using the proxy model, all apply random projection to 2,048 dimensions, and then perform clustering. For embeddings and hidden states, we cluster their original representations. We also evaluate combined multi-layer hidden-state features.

Importantly, we evaluate all resulting clusters using independently computed Mistral-7B gradients across training checkpoints, rather than relying on similarity measured in the proxy space. This directly examines whether the clusters preserve the target-model gradient coherence required by our theoretical analysis.

To further address your concern, we added experiments with a model from a different family but with a comparable parameter count to Qwen2.5-0.5B, as well as Qwen2.5-1.5B. We also evaluated different projection dimensions for Mistral-7B and Qwen2.5-0.5B.

Clustering representationDimensionIntra-similarity ↑
Qwen2.5-0.5B gradients2,0480.5267
Qwen2.5-0.5B gradients4,0960.5335
Qwen2.5-1.5B gradients2,0480.5038
Mistral-7B gradients2,0480.3365
Mistral-7B gradients16,3840.5815
SmolLM2-360M gradients2,0480.5865

Under the same 2,048-dimensional projection, Qwen2.5-0.5B achieves higher intra-cluster similarity than Qwen2.5-1.5B (0.5267 vs. 0.5038) and Mistral-7B (0.3365). However, increasing the projection dimension to 16,384 improves Mistral-7B's similarity to 0.5815, surpassing Qwen2.5-0.5B. Interestingly, SmolLM2-360M, which is even smaller than Qwen2.5-0.5B, achieves a higher similarity of 0.5865 at only 2,048 dimensions.

These findings suggest that aggressive dimensionality reduction may cause substantial information loss for larger models, potentially offsetting their advantages in representation quality. This may explain the weaker performance of Mistral-7B at 2,048 dimensions. However, extracting Mistral-7B gradients and performing high-dimensional clustering incur considerably higher computational costs. Therefore, we favor lightweight proxy gradients as a practical balance between clustering quality and efficiency.

W2. Are Gradients Recomputed?

ACE deliberately separates clustering from attribution. Proxy gradients and clusters are computed once offline and reused across selection rounds. Target-model attribution is computed according to the underlying attribution method and its update schedule, allowing cluster utilities to reflect the target training process without repeatedly reconstructing clusters.

Q1. Additional Embedding Baselines

Following your suggestion, we evaluate additional embedding models:

RepresentationIntra-similarity ↑
Multilingual E5-base0.2771
Nomic Embed v1.50.2078
EmbeddingGemma 20.2466

All three embedding representations exhibit substantially lower target-model gradient coherence than the lightweight gradient proxies.

We also conduct downstream comparisons:

RepresentationMMLUBBHTyDiQAAvg ↑
EmbeddingGemma 260.9855.1959.9258.70
Qwen2.5-0.5B gradients60.8156.0261.2259.35
SmolLM2-360M-Instruct gradients61.4559.1759.3559.99

These additional results show that small-model gradient clustering generalizes beyond Qwen and can outperform embedding-based clustering in our setting.

Q2. Notation

Thank you for pointing this out. We will explicitly define K (number of clusters) and m (representatives per cluster) in Table 1.

Q3. Why Uniform Sampling?

We adopt uniform representative sampling because it provides an unbiased estimator of cluster-mean utility, consistent with our theoretical guarantees on unbiasedness and variance control. This is precisely the property we aim to preserve.

For the final training-data selection, we additionally explore center-based and mixed sampling strategies, as suggested by you. These alternatives are applied only to training-sample selection, without changing the uniform representative sampling used for utility estimation.

We sincerely appreciate your constructive feedback and would be happy to address any further questions. We hope these additional experiments and clarifications help strengthen your assessment of our work.

thanks for addressing my questions, hopefully these new results and clarifications make it to the next version.

One quick follow-up about the static clustering. I understand that recomputing proxy gradients and clusters could be costly but its still not clear to me if these precomputed clusters stay aligned with the target model distribution. As an auxiliary analysis, it would be interesting to check if the proxy-gradient based clustering is consistent with the clusters computed using target model gradients, at various stages of training.

Follow-up Response to Reviewer Q3Mp

We sincerely thank the reviewer for this valuable follow-up suggestion.

First, our existing cross-checkpoint analysis shows that fixed proxy-derived clusters maintain intra-cluster gradient coherence as the target model evolves during training, which is the key property required by our theoretical variance guarantee.

Second, following your suggestion, we conducted an additional large-scale experiment using 16 H20 GPUs for a whole day. Specifically, we independently computed target-model gradients and reconstructed clusters at four Mistral-7B training checkpoints (0, 200, 400, and 600), using the same 270,679 training examples and clustering configuration (K = 4,230). We then compared each target-model partition with the original fixed proxy-derived partition.

Mistral checkpointNMI ↑AMI ↑
00.67750.6106
2000.74050.6755
4000.74480.6799
6000.74810.6803

The results show that substantial shared structure is preserved as training progresses. NMI measures the shared information between two cluster partitions, while AMI further adjusts for similarities that may occur by chance. Notably, NMI increases from 0.6775 to 0.7481, and AMI from 0.6106 to 0.6803. These results suggest that the fixed proxy-derived clusters remain largely aligned with the underlying structural relationships of the target model throughout training, reflecting stable higher-level semantic or learning-signal structures.

These findings further support ACE's design: the reusable proxy-derived clusters remain largely aligned with the target model's gradient structure throughout training, providing the intra-cluster coherence needed for theoretically controlled utility estimation.

We will incorporate both our previous projection-dimension analysis and this new cross-checkpoint clustering analysis into Section 4.2 of the revised manuscript.

We sincerely appreciate your constructive suggestion. We carefully conducted this additional large-scale analysis to directly address your concern, and we hope these new results provide stronger evidence for the effectiveness of static clustering and support a more favorable assessment of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response— Summary of Reviewer Feedback and Additional Results

Reviewer Feedback

We are grateful that Reviewer cMUv explicitly acknowledged that our additional experiments addressed their concerns and maintained their positive assessment. Reviewer Q3Mp also acknowledged our clarifications, and we subsequently completed an additional cross-checkpoint analysis in response to their follow-up suggestion.

Importantly, Reviewer v2oa acknowledged that our responses addressed Q1 and W2, while requesting a more quantitative characterization of when cluster-level allocation helps or hurts. We devoted substantial additional effort to this remaining concern, providing both new theoretical conditions and controlled data-mixture experiments.

We sincerely appreciate the reviewers' constructive feedback, which has helped us strengthen the theoretical understanding, empirical validation, and practical significance of ACE. We will incorporate these additional analyses and results into the revised manuscript.

Additional Analyses and Experiments

  1. Proxy Model Robustness. Evaluated different proxy models, showing that lightweight models from different families can produce effective clusters, with SmolLM2-360M achieving higher gradient coherence (0.5865) and downstream performance (59.99) than our original Qwen proxy. (Reviewers Q3Mp(W1), cMUv(W2/Q2))

  2. Gradient Projection Analysis. Investigated different projection dimensions, finding that aggressive compression can substantially reduce clustering quality, especially for larger models (Mistral-7B: 0.3365 → 0.5815 with increased dimensions). (Reviewers Q3Mp(W1), cMUv(W2/Q2))

  3. Additional Embedding Baselines. Evaluated E5, Nomic, and EmbeddingGemma, finding that gradient-based clustering provides stronger target-model gradient coherence than these embedding-based alternatives. (Reviewer Q3Mp(Q1))

  4. Proxy–Target Clustering Alignment Across Training. Reconstructed clusters using target-model gradients at four training checkpoints and compared them with fixed proxy clusters, finding substantial structural agreement (NMI: 0.6775–0.7481; AMI: 0.6106–0.6803), supporting the alignment of static proxy clustering during training. (Reviewer Q3Mp(Follow-up))

  5. Decoupling Clustering and Attribution. Analyzed cross-model utility correlations and alternative utility-scoring models, showing that proxy-derived clusters can capture reusable data structure while target-model attribution provides more effective target-conditioned selection (59.35 vs. 57.77 for proxy-only selection). (Reviewer v2oa(Q1))

  6. Systematic Selection Budget Analysis. Evaluated selection budgets from 5% to 50%, finding consistent improvements over LESS and the highest performance at 20%, consistent with systematicly approach of the observed minima in target validation loss and gradient norm. (Reviewer v2oa(W2))

  7. Theoretical Applicability Conditions. Derived a two-level selection guarantee connecting inter-cluster utility gaps, estimation error, temperature, intra-cluster gradient coherence, and utility variance, providing quantitative conditions and lower bound for reliable selection and explaining its limitations. (Reviewer v2oa(W1, Follow-up))

  8. Broader applicability Across Data Distributions. Conducted controlled single-target and multiple-target experiments across different proportions of irrelevant data. ACE outperforms LESS in multiple-target settings, with gains up to +1.46 points, while extreme distributions reveal when its advantage diminishes. (Reviewer v2oa(Follow-up))

  9. Representative Estimation Ablation. Compared representative-based estimation with exact cluster means, showing nearly identical downstream performance (59.35 vs. 59.63) at substantially lower computational cost. (Reviewer cMUv(W1/Q1))

  10. Cluster Allocation Ablation. Compared utility-weighted and uniform cluster allocation, demonstrating that target-conditioned allocation substantially improves downstream performance (59.35 vs. 56.93). (Reviewer cMUv(W1/Q1))

  11. Detailed Runtime Breakdown. Separately measured preprocessing, attribution, selection, and training costs, identifying attribution computation as the main source of savings and demonstrating a 24.3× end-to-end speedup over LESS. (Reviewer cMUv(W3/Q3))

  12. Matched Total Compute. Compared ACE against Random and TSDS under equal total runtime, with ACE achieving higher downstream performance (59.35 vs. 59.15 and 58.76) despite fewer training steps. (Reviewer cMUv(W4/Q4))

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.