- Rating
- 1
- Confidence
- 4(+2)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Cached Diverse Influence Selection (CDIS), an efficient influence-based data selection framework that aims to reduce the computational overhead of LESS. The key observation is that gradient-based influence rankings remain relatively stable across nearby training checkpoints, making full gradient recomputation potentially unnecessary. CDIS reuses cached training gradients, selectively refreshes high-ranked candidates, and applies affine calibration to approximate updated influence scores. It further incorporates source- and length-based stratified selection to maintain data diversity. Experiments on Qwen3-4B and Llama-3.2-1B demonstrate that CDIS achieves high selection overlap with full recomputation while reducing the gradient computation cost by approximately 3.5×. Downstream experiments also show improvements over random selection, although some performance degradation remains compared with full recomputation.
Critical strengths
-
Clear motivation: The paper identifies an important computational bottleneck in gradient-based data selection methods such as LESS. Reusing historical gradient features is an intuitive and practically relevant approach to reducing repeated gradient computation.
-
Practical efficiency: CDIS provides a relatively simple and easy-to-integrate solution without requiring additional models. The proposed partial refresh and calibration mechanisms significantly reduce gradient computation costs while maintaining high selection fidelity.
-
Useful empirical analysis: The paper investigates gradient ranking stability, refresh budgets, checkpoint distances, and selection diversity. The experiments provide useful insights into the trade-off between computational efficiency and selection quality.
Critical weaknesses
-
W1. Limited experimental evaluation: Although the paper evaluates two model families, two candidate pools, and GSM8K/MMLU as selection targets, the experimental coverage remains relatively limited, particularly regarding downstream performance across different tasks and model scales. More importantly, the comparisons primarily focus on LESS-related variants baselines. Several representative data selection methods, as well as other relevant methods discussed in the Related Work section, are not included in the main comparisons. Broader comparisons with both gradient-based and non-gradient-based methods, together with evaluations on more diverse benchmarks and larger models, would provide stronger evidence for the effectiveness and generalizability of CDIS.
-
W2. Limited novelty and heuristic design choices: The proposed method mainly extends LESS through cached gradient reuse, partial refresh, and affine calibration. Although these components improve computational efficiency, the overall technical contribution appears relatively incremental. Several design choices, particularly the top-ranked refresh strategy and refresh fraction, remain heuristic. It is unclear whether this degradation is an unavoidable consequence of stale gradient approximation or results from a suboptimal refresh budget allocation strategy. Further comparisons with alternative refresh strategies, such as uncertainty-based or selection-boundary-aware refresh, would strengthen the methodological justification.
-
W3. Presentation and readability: The overall presentation could be improved, particularly in the experimental section, where results are scattered across multiple tables, figures, and appendix sections, making it difficult to identify and compare the key findings. Additionally, the AI Use Statement, Reproducibility Statement, and Ethics Statement are currently placed after the appendix, rather than before the references and appendix as indicated by the ICLR template and author guidelines.
Questions for authors
-
Q1. Could the authors extend the evaluation to larger models, more diverse downstream benchmarks, and additional representative data selection baselines? Comparisons under matched selection budgets and computational costs would help establish the generalizability and practical advantages of CDIS.
-
Q2. Could the authors further clarify the main technical novelty of CDIS beyond incremental improvements to LESS? In particular, how does the proposed partial refresh mechanism differ from existing gradient caching or selective re-computation strategies?
3 comments
W1 / Q1: larger models, more benchmarks, representative baselines at matched budget.
- Models. Qwen3-4B and Llama-3.2-1B with their own gradients and selections, and (new) Qwen3-8B-Base with its own warmup, gradient features, and selections. On Qwen3-8B the refresh recovers the top-k subset exactly at p = 0.3, per-stratum allocation keeps 0.956 of the stratified subset, the gradient stage is 3.48 times faster, and downstream CDIS is -0.4 [-1.1, 0.4] answer-level points from full recomputation (three training seeds).
- Benchmarks. Downstream on full GSM8K, full ASDiv, and (new) the full MMLU test set with MMLU-target selections (14,042 items, 5-shot, two seeds), where CDIS is +0.10 [-0.04, 0.24] points from full recomputation.
- Baselines at the same selection budget (k = 1,000), full GSM8K, three seeds, strict / answer-level:
| Method | Strict | Answer-level |
|---|---|---|
| Plain random | 0.389 | 0.789 |
| Length-matched random | 0.323 | 0.752 |
| Unconstrained influence top-k | 0.263 | 0.801 |
| BIDS-style balanced influence, full recompute (new) | 0.562 | 0.792 |
| BIDS-style, CDIS per-stratum (new) | 0.544 | 0.795 |
| RDS, representation-based, non-gradient (new) | 0.793 | 0.824 |
| Stratified influence, full recompute | 0.553 | 0.801 |
| Stratified influence, CDIS per-stratum | 0.542 | 0.805 |
Label-free cluster quotas (Appendix O), random refresh at matched compute (Appendix J), and checkpoint skipping (Appendix I) are also reported.
On this testbed, where the base models already answer about 80% of GSM8K items, RDS selects mostly Dolly examples (97 to 99%), whose responses end cleanly, and its models end their answers on 96% of items, which gives it the highest scores here. The choice of selector is orthogonal to our contribution: for gradient-based selection, CDIS reproduces the subset and the downstream result of full recomputation at a third of the gradient cost, on every model and benchmark we tested (general response, part 2).
Thank you for the additional experiments and detailed explanations, which have addressed some of my concerns. However, I still have reservations about the paper overall.
-
- Although RDS and BIDS-style comparisons have been added, much of the evaluation still concerns variants of the LESS-style pipeline. This does not sufficiently establish the method’s broader practical advantages. The main text also lacks a comprehensive comparison; the core experimental evidence should be presented there rather than requiring reviewers to reconstruct it from the appendix.
-
- I encourage the authors to consult related data selection studies and evaluate a broader range of capabilities beyond the current mathematics and knowledge benchmarks.
-
- Results at other ratios, such as 1%, 5%, and 20%, are missing, as is the particularly important full-data training baseline using 100% of the candidate pool. Full gradient recomputation followed by subset selection does not substitute for training on all available data.
-
- In my view, an important goal of data selection is to identify a smaller subset that matches or exceeds full-data training performance while reducing overall runtime. This perspective is offered for consideration. Although some cost accounting is provided, a comprehensive comparison covering the entire process—from selection through training—against full-data training and other selection methods is still missing.
I share Reviewer LDSr’s concerns in their follow-up comment regarding the method’s practical significance and the clarity of the writing. I hope the authors will substantially improve both the presentation and the experimental evidence. Given the remaining concerns, I will maintain my current score. Thank you.
We thank the reviewer for the review and the clear list of requests. We ran each requested comparison and revised the presentation.
W3: template and presentation. The AI use, reproducibility, and ethics statements now appear after the main text and before the references, as the ICLR template requires. Table 2 is now the single main comparison (recomputation cost, strict and answer-level downstream accuracy, paired CIs of every baseline against CDIS), and Section 4 separates the refresh from the selection rule.
W2: is the downstream degradation unavoidable or caused by the refresh allocation? It was caused mostly by the allocation. The submitted downstream runs refreshed the top pN of the whole pool. Allocating the same budget within each stratum of the selection rule raises CDIS from 0.510 to 0.542 strict exact match on full GSM8K (+3.2 [1.9, 4.4]), against 0.553 for full recomputation (-1.1 [-2.2, 0.0]); at the answer level CDIS reaches 0.805 against 0.801 (+0.4 [-0.4, 1.1]).
W2: alternative refresh strategies (uncertainty- or boundary-aware). New Appendix L compares strategies at equal budget (mean over eight settings, top-1,000 overlap, k/N = 0.1):
The boundary-aware band, which the restated Proposition 1 justifies, is best at small budgets; refreshing where drift was largest before is below both; all non-random strategies agree by p = 0.3. For the stratified rule, the per-stratum band reaches 0.912 stratified overlap at p = 0.1 against 0.839 for global top refresh. The adaptive rule sets p from observed residuals, so the budget is no longer a fixed heuristic.
W2 / Q2: novelty beyond LESS; difference from gradient caching and selective recomputation. Caching features is standard (LESS, TRAK, and LoGra keep datastores). What existing work does not provide is a rule for which cached features to recompute when the checkpoint changes, with a guarantee. Lazy greedy evaluation reuses stale gains because they are exact upper bounds under submodularity; influence scores have no such bound between checkpoints, so selective recomputation needs (i) a calibration of stale to current scores, (ii) a sufficient condition on the refresh set (Proposition 1), (iii) an observable estimate of the margin (the adaptive rule), (iv) an allocation of the budget for constrained selection rules, and (v) a check for intervals where reuse is unsafe. These five components are the contribution, and each is evaluated in the paper.