acceptodds
Under review as a conference paper at ICLR 2027

Refreshing Less, Selecting Better: Cache-Efficient Diverse Influence Selection for LLM

Abstract

Gradient-based data selection methods such as LESS score each candidate example by the alignment between its gradient and the gradient of a target validation set, and the scoring stage dominates their cost because per-example gradient features must be recomputed whenever the model checkpoint changes. Across three selection seeds, two model families, two candidate pools, and two target tasks, gradient features cached at a post-warmup checkpoint and paired with fresh validation gradients preserve the ranking 40 optimizer steps later with Spearman correlation between 0.952 and 0.991, while the top-10% subset they induce misses 10 to 22% of the examples that full recomputation selects. We therefore propose Cached Diverse Influence Selection (CDIS), which recomputes gradient features for the top-ranked fraction of candidates under the stale scores, fits an affine calibration on the recomputed examples, and selects the final subset under source and length quotas. A refresh fraction at or above the selection fraction recovers the exact top- subset whenever the calibrated stale scores have bounded error, and the measured budget curves follow this rule: at the recovered top- subset coincides with full recomputation in every setting, allocating the same budget per stratum recovers the stratified subset at 0.92 to 1.00, and the wall-clock of the gradient stage drops by a factor of 3.5 to 3.6. Iterating the cache over four checkpoints keeps top- overlap at 0.98 or higher at 1.9 gradient features per example against 4 for full recomputation, and the agreement between stale and recomputed scores on the refreshed examples provides a free check for unsafe reuse. Downstream, unconstrained top- selection on influence scores collapses to a single data source and scores 13 points below random selection on GSM8K. CDIS scores 12 points above random selection with paired confidence intervals that exclude zero and trails full recomputation by 4.3 points, one training-run standard deviation, at 3.4 times lower selection cost.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
data selection, influence functions, instruction tuning, post-training, efficient methods, gradient-based selection, caching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Z1hHOpenReview ↗
Rating
1
Confidence
4(+2)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Cached Diverse Influence Selection (CDIS), an efficient influence-based data selection framework that aims to reduce the computational overhead of LESS. The key observation is that gradient-based influence rankings remain relatively stable across nearby training checkpoints, making full gradient recomputation potentially unnecessary. CDIS reuses cached training gradients, selectively refreshes high-ranked candidates, and applies affine calibration to approximate updated influence scores. It further incorporates source- and length-based stratified selection to maintain data diversity. Experiments on Qwen3-4B and Llama-3.2-1B demonstrate that CDIS achieves high selection overlap with full recomputation while reducing the gradient computation cost by approximately 3.5×. Downstream experiments also show improvements over random selection, although some performance degradation remains compared with full recomputation.

Critical strengths
  • Clear motivation: The paper identifies an important computational bottleneck in gradient-based data selection methods such as LESS. Reusing historical gradient features is an intuitive and practically relevant approach to reducing repeated gradient computation.

  • Practical efficiency: CDIS provides a relatively simple and easy-to-integrate solution without requiring additional models. The proposed partial refresh and calibration mechanisms significantly reduce gradient computation costs while maintaining high selection fidelity.

  • Useful empirical analysis: The paper investigates gradient ranking stability, refresh budgets, checkpoint distances, and selection diversity. The experiments provide useful insights into the trade-off between computational efficiency and selection quality.

Critical weaknesses
  • W1. Limited experimental evaluation: Although the paper evaluates two model families, two candidate pools, and GSM8K/MMLU as selection targets, the experimental coverage remains relatively limited, particularly regarding downstream performance across different tasks and model scales. More importantly, the comparisons primarily focus on LESS-related variants baselines. Several representative data selection methods, as well as other relevant methods discussed in the Related Work section, are not included in the main comparisons. Broader comparisons with both gradient-based and non-gradient-based methods, together with evaluations on more diverse benchmarks and larger models, would provide stronger evidence for the effectiveness and generalizability of CDIS.

  • W2. Limited novelty and heuristic design choices: The proposed method mainly extends LESS through cached gradient reuse, partial refresh, and affine calibration. Although these components improve computational efficiency, the overall technical contribution appears relatively incremental. Several design choices, particularly the top-ranked refresh strategy and refresh fraction, remain heuristic. It is unclear whether this degradation is an unavoidable consequence of stale gradient approximation or results from a suboptimal refresh budget allocation strategy. Further comparisons with alternative refresh strategies, such as uncertainty-based or selection-boundary-aware refresh, would strengthen the methodological justification.

  • W3. Presentation and readability: The overall presentation could be improved, particularly in the experimental section, where results are scattered across multiple tables, figures, and appendix sections, making it difficult to identify and compare the key findings. Additionally, the AI Use Statement, Reproducibility Statement, and Ethics Statement are currently placed after the appendix, rather than before the references and appendix as indicated by the ICLR template and author guidelines.

Questions for authors
  • Q1. Could the authors extend the evaluation to larger models, more diverse downstream benchmarks, and additional representative data selection baselines? Comparisons under matched selection budgets and computational costs would help establish the generalizability and practical advantages of CDIS.

  • Q2. Could the authors further clarify the main technical novelty of CDIS beyond incremental improvements to LESS? In particular, how does the proposed partial refresh mechanism differ from existing gradient caching or selective re-computation strategies?

3 comments

We thank the reviewer for the review and the clear list of requests. We ran each requested comparison and revised the presentation.

W3: template and presentation. The AI use, reproducibility, and ethics statements now appear after the main text and before the references, as the ICLR template requires. Table 2 is now the single main comparison (recomputation cost, strict and answer-level downstream accuracy, paired CIs of every baseline against CDIS), and Section 4 separates the refresh from the selection rule.

W2: is the downstream degradation unavoidable or caused by the refresh allocation? It was caused mostly by the allocation. The submitted downstream runs refreshed the top pN of the whole pool. Allocating the same budget within each stratum of the selection rule raises CDIS from 0.510 to 0.542 strict exact match on full GSM8K (+3.2 [1.9, 4.4]), against 0.553 for full recomputation (-1.1 [-2.2, 0.0]); at the answer level CDIS reaches 0.805 against 0.801 (+0.4 [-0.4, 1.1]).

W2: alternative refresh strategies (uncertainty- or boundary-aware). New Appendix L compares strategies at equal budget (mean over eight settings, top-1,000 overlap, k/N = 0.1):

ptop stale (ours)boundary bandprevious drift (uncertainty proxy)random
0.050.8550.9360.8760.858
0.100.8890.9750.9150.863
0.200.9940.9970.9670.873
0.301.0001.0000.9890.886

The boundary-aware band, which the restated Proposition 1 justifies, is best at small budgets; refreshing where drift was largest before is below both; all non-random strategies agree by p = 0.3. For the stratified rule, the per-stratum band reaches 0.912 stratified overlap at p = 0.1 against 0.839 for global top refresh. The adaptive rule sets p from observed residuals, so the budget is no longer a fixed heuristic.

W2 / Q2: novelty beyond LESS; difference from gradient caching and selective recomputation. Caching features is standard (LESS, TRAK, and LoGra keep datastores). What existing work does not provide is a rule for which cached features to recompute when the checkpoint changes, with a guarantee. Lazy greedy evaluation reuses stale gains because they are exact upper bounds under submodularity; influence scores have no such bound between checkpoints, so selective recomputation needs (i) a calibration of stale to current scores, (ii) a sufficient condition on the refresh set (Proposition 1), (iii) an observable estimate of the margin (the adaptive rule), (iv) an allocation of the budget for constrained selection rules, and (v) a check for intervals where reuse is unsafe. These five components are the contribution, and each is evaluated in the paper.

W1 / Q1: larger models, more benchmarks, representative baselines at matched budget.

  • Models. Qwen3-4B and Llama-3.2-1B with their own gradients and selections, and (new) Qwen3-8B-Base with its own warmup, gradient features, and selections. On Qwen3-8B the refresh recovers the top-k subset exactly at p = 0.3, per-stratum allocation keeps 0.956 of the stratified subset, the gradient stage is 3.48 times faster, and downstream CDIS is -0.4 [-1.1, 0.4] answer-level points from full recomputation (three training seeds).
  • Benchmarks. Downstream on full GSM8K, full ASDiv, and (new) the full MMLU test set with MMLU-target selections (14,042 items, 5-shot, two seeds), where CDIS is +0.10 [-0.04, 0.24] points from full recomputation.
  • Baselines at the same selection budget (k = 1,000), full GSM8K, three seeds, strict / answer-level:
MethodStrictAnswer-level
Plain random0.3890.789
Length-matched random0.3230.752
Unconstrained influence top-k0.2630.801
BIDS-style balanced influence, full recompute (new)0.5620.792
BIDS-style, CDIS per-stratum (new)0.5440.795
RDS, representation-based, non-gradient (new)0.7930.824
Stratified influence, full recompute0.5530.801
Stratified influence, CDIS per-stratum0.5420.805

Label-free cluster quotas (Appendix O), random refresh at matched compute (Appendix J), and checkpoint skipping (Appendix I) are also reported.

On this testbed, where the base models already answer about 80% of GSM8K items, RDS selects mostly Dolly examples (97 to 99%), whose responses end cleanly, and its models end their answers on 96% of items, which gives it the highest scores here. The choice of selector is orthogonal to our contribution: for gradient-based selection, CDIS reproduces the subset and the downstream result of full recomputation at a third of the gradient cost, on every model and benchmark we tested (general response, part 2).

Thank you for the additional experiments and detailed explanations, which have addressed some of my concerns. However, I still have reservations about the paper overall.

    1. Although RDS and BIDS-style comparisons have been added, much of the evaluation still concerns variants of the LESS-style pipeline. This does not sufficiently establish the method’s broader practical advantages. The main text also lacks a comprehensive comparison; the core experimental evidence should be presented there rather than requiring reviewers to reconstruct it from the appendix.
    1. I encourage the authors to consult related data selection studies and evaluate a broader range of capabilities beyond the current mathematics and knowledge benchmarks.
    1. Results at other ratios, such as 1%, 5%, and 20%, are missing, as is the particularly important full-data training baseline using 100% of the candidate pool. Full gradient recomputation followed by subset selection does not substitute for training on all available data.
    1. In my view, an important goal of data selection is to identify a smaller subset that matches or exceeds full-data training performance while reducing overall runtime. This perspective is offered for consideration. Although some cost accounting is provided, a comprehensive comparison covering the entire process—from selection through training—against full-data training and other selection methods is still missing.

I share Reviewer LDSr’s concerns in their follow-up comment regarding the method’s practical significance and the clarity of the writing. I hope the authors will substantially improve both the presentation and the experimental evidence. Given the remaining concerns, I will maintain my current score. Thank you.

Reviewer_LDSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to reduce the cost of influence-based data selection when gradient features need to be recomputed across multiple model checkpoints. It proposes CDIS, which reuses cached candidate gradient features, recomputes only a top-ranked fraction of candidates, calibrates stale scores using the refreshed examples, and combines the resulting scores with a stratified selection rule. The paper shows that CDIS maintains high selection fidelity relative to full gradient recomputation while substantially reducing gradient-computation cost. With stratified selection, CDIS also preserves most of the downstream performance of full recomputation on GSM8K and ASDiv at much lower selection cost.

Critical strengths
  1. The motivation makes sense to some extent, and the angle is novel.
Critical weaknesses

The first and third points are the most important concerns for me.

  1. Overall, my main concern is that the setting feels too narrow and somewhat niche. For example, the method seems mainly applicable to the LESS-style approaches, so I view it more as partially addressing a specific issue in LESS-style methods rather than a broadly important setting. I do not find the setting itself particularly exciting.

  2. In terms of the method, I do not find the theoretical analysis very useful, because it depends on an unknown quantity, the maximum error, which makes the result not very informative in practice. In contrast, I am more curious about why an affine transformation is a reasonable calibration choice. I think this part needs more justification or additional experiments.

  3. In terms of the experiments, a large portion of the paper focuses on overlap and alignment analyses. I care more about downstream task performance. Although the paper does report downstream results, I think these should be treated as the main evaluation metric, with more experiments and analysis devoted to them. The paper places too much emphasis on alignment, which I do not view as the most important aspect.

  4. (Minor) The writing is not always easy to follow, and several figures are difficult to read or poorly formatted. For example, the y-axis in Figure 1 appears incomplete, Figure 2 is hard to read because of its size, and the legend in Figure 3(b) overlaps with the plot. These presentation issues make the paper feel less polished overall.

Questions for authors
  1. Why is an affine calibration appropriate here? What is the justification for this assumption? Is there additional empirical evidence showing that the relationship between stale and current scores is approximately affine, or comparisons with other calibration choices?

  2. What is the exact role of stratified selection in the paper? My understanding is that the core problem studied by the paper is how to efficiently refresh stale gradient features and scores across checkpoints, while stratification is more of a practical consideration for converting influence scores into a useful training subset. Is this interpretation correct? If so, I feel that the current presentation somewhat entangles these two issues in both the method and experimental sections. Would it be cleaner to separate the core refresh/staleness problem from the choice of downstream selection rule more explicitly?

  3. Can some examples remain stale for a very long time? Since the refresh set is chosen greedily based on the stale scores, is it possible that some examples are only evaluated with their true gradient at the beginning and are never refreshed again? If an example had a low score a long time ago but its true utility later increases substantially as the model changes, how would the current method discover it? More generally, what prevents low-ranked stale examples from being permanently overlooked?

Additional comments (just for reference): The setting also reminds me of online data selection, where the model changes over training and the utility of individual data points may therefore also change over time. A similar question arises there: whether every data point needs to be re-evaluated at every epoch or step, or whether evaluation can be performed more sparsely.

For example, [1] evaluates the difficulty of only a subset of problems at each step and estimates the difficulty of other problems using similarity. I wonder whether a related idea could also be useful here. Instead of applying the same affine calibration to all unrefreshed examples, one possibility may be to estimate their current scores based on their similarity to examples in the refresh set. I think the authors may find ideas from this area relevant to the current problem. The authors certainly do not need to cite [1]; I only mention it as a potentially useful reference for this direction. :)

[1] Sun, Yifan, et al. “Improving Data Efficiency for LLM Reinforcement Fine-Tuning through Difficulty-Targeted Online Data Selection and Rollout Replay.” Advances in Neural Information Processing Systems 38.

2 comments

We thank the reviewer for the detailed review and the concrete suggestions. We followed each of them, including the similarity-based estimate of unrefreshed scores.

W3 (most important): downstream results should be the main metric.

We agree and restructured the evaluation around downstream quality. Table 2 is now the main result: every method on full GSM8K and full ASDiv (three seeds each), with paired bootstrap CIs, under the strict exact match of the LESS evaluator and under answer-level exact match (Appendix F explains the difference). The key change is the allocation of the refresh budget: refreshing the top p fraction within each stratum of the selection rule, instead of the top pN of the whole pool, makes CDIS match full recomputation under both metrics on GSM8K (-1.1 [-2.2, 0.0] strict, +0.4 [-0.4, 1.1] answer-level) and ASDiv (-0.6 [-1.4, 0.2], -0.1 [-0.5, 0.2]). The 4.3-point gap in the submission came mostly from global allocation, which misses examples of the low-scoring source; the selection-level analysis predicted this. The same holds on Qwen3-8B with its own selections and three training seeds (-0.4 [-1.1, 0.4] answer-level; the strict gap of 3.0 points is within the training-seed spread), on MMLU with MMLU-target selections (+0.10 [-0.04, 0.24]), and for the BIDS-style rule (per-stratum allocation within 0.3 answer-level points of its full recomputation).

W1 (most important): the setting is narrow.

The refresh applies to any score that pairs a cached per-example feature with a checkpoint-dependent target vector: LESS and its proxy-model variants, TracIn-style checkpoint sums, TRAK and LoGra datastores queried at a new checkpoint, and gradient-matching coresets (CRAIG, GradMatch) that re-select every epoch. In all of them the per-example feature is the expensive part, and the question of which features to recompute when the model moves is the same. The revision states this scope in Sections 2 and 4 and contrasts it with lazy greedy evaluation, where stale gains are exact upper bounds; influence scores have no such bound, which is why calibration, a margin, and a check are needed. The scoring stage dominates the cost of these pipelines, and CDIS reduces it by 3.4 to 3.6 times on three models (1B, 4B, 8B).

W4: figures. Fixed: Figure 1 y-axis label, Figure 2 at full text width, Figure 3(b) with direct line labels and no legend over the data, and the Figure 4 labels.

First, I suspect that the response intended for me may have been accidentally posted under another reviewer. No worries, I read the response under the other reviewer’s thread. :)

I have to say that, although I tried very carefully to read and understand your response, I found it genuinely exhausting to follow, in much the same way as I did when reading the original paper. Many parts are not very intuitive, the response is packed with specialized terminology, and I honestly got a strong impression of AI-generated or AI-polished writing. I apologize if this is not the case.

I understood most of the points, and I appreciate the additional experiments. I would encourage the authors to include these experiments in the paper. To be honest, after reading the response, I no longer think most of my technical questions are particularly serious concerns.

Overall, however, I still have substantial concerns about both the practical significance of the method and the writing. Regarding the former, I still find the problem somewhat niche. Despite the rebuttal to W1, my impression remains that the paper focuses on solving a relatively specific issue within this particular family of methods, and I do not find the broader practical importance especially convincing. Regarding the writing, I still find the paper very difficult to read. The accumulation of specialized terminology, statements that often lack intuitive explanation, and dense presentation of experimental numbers make the paper feel very unfriendly to the reader. To be candid, this style also gives me a strong “AI slop” impression, and again I apologize if the paper was not written with AI assistance.

Finally, the amount of revision described in the response now seems to cover a very large portion of the original PDF. I understand that many of these changes may individually be relatively small, but taken together, the scale of the proposed revision does not leave a particularly positive impression on me.

Reviewer_qo88OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Cached Diverse Influence Selection (CDIS), an efficient approach to gradient-based data selection that reduces redundant gradient computations across model checkpoints The paper provides a theoretical sufficient condition for exact top-k recovery and evaluates the proposed approach across different model families, candidate pools, target tasks, and checkpoint intervals.

Critical strengths
  1. Comprehensive and carefully designed experimental evaluation.

The experimental section is a major strength of this work. The authors evaluate CDIS across multiple random seeds, two model families, different candidate pools, and two target tasks. The experiments also investigate checkpoint distances, repeated cache updates, refresh budget allocation, and alternative refresh strategies. These evaluations provide convincing empirical evidence for the effectiveness of partial gradient refresh in the considered setting.

  1. A simple and practically motivation.

The proposed method addresses a meaningful computational bottleneck in influence-based data selection. Its main idea is straightforward and potentially compatible with existing gradient-based selection pipelines.

Critical weaknesses
  1. Clarification of the refresh budget selection.

The authors use a refresh fraction of (p=0.3) for a selection fraction of (k/N=0.1), which works well across the reported settings. It would be helpful to clarify how practitioners should interpret the additional refresh margin when applying the method to different selection fractions or checkpoint intervals. A brief discussion based on the existing results would improve the practical guidance.

Questions for authors

Could the authors further clarify how the drift margin should be determined in practice? Section 4.2 provides a qualitative guideline, while Section 6.1 motivates the choice of p=0.3p=0.3 empirically. It would be helpful to understand how the two are connected when choosing the refresh budget.

1 comment

We thank the reviewer for the positive assessment and for the precise question, which led to a clearer link between the theory and the choice of p.

W1 / Q1: How should the drift margin be chosen, and how do Section 4.2 and Section 6.1 connect?

The revision makes this connection explicit (Section 4.2, paragraph "Choosing p" in Section 6.1, and Table 14).

  1. Proposition 1 now states that the refresh set must cover every candidate whose calibrated stale score lies within the maximum calibration error eps of the selection threshold. For a refresh set chosen by stale rank, this means p equals the selection fraction k/N plus the fraction of candidates within eps below the threshold.
  2. Evaluating eps after the fact gives the required p directly: 0.30, 0.36, and 0.30 for the three Qwen3-4B seeds and 0.25 for Llama-3.2-1B at k/N = 0.1. This is where the measured budget curves saturate, so the empirical choice p = 0.3 in Section 6.1 is the proposition evaluated at the observed error. On Qwen3-8B the condition asks for 0.43 while p = 0.3 is already exact, so the condition is sufficient and can be conservative.
  3. Since eps is not observed at selection time, we derive an adaptive rule from quantities that are: start at p = k/N, estimate the margin as the 0.95 quantile of the absolute residuals on the refreshed candidates, and grow the refresh set until it covers every candidate within that margin of the current threshold. It needs no tuning per setting:
Intervalk/Nchosen ptop-1,000 overlap
post-warmup, 8 settings0.050.09 to 0.15at least 0.982
post-warmup, 8 settings0.100.16 to 0.240.987 to 1.000
post-warmup, 8 settings0.200.27 to 0.39at least 0.998
long distance (80 to 160/240/320)0.100.32 to 0.380.994 to 0.999
early origin (40 to 80/120)0.100.45 to 0.500.991 to 0.998

The margin grows with the selection fraction and with the drift, and the rule spends more on exactly the intervals that the online check flags as unstable. A per-stratum version for stratified rules chooses p = 0.28 to 0.48 with stratified overlap 0.987 to 1.000.

Practical guidance (Sections 6.1 and 6.2): on stable post-warmup intervals, use p = k/N plus a margin of about 0.1 to 0.2, or run the adaptive rule; when the agreement between stale and recomputed scores on the refresh set falls below about 0.75, recompute all features.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): summary of the revision

We thank all reviewers for their careful and constructive reviews. We revised the paper substantially and ran the requested experiments. New and revised text is shown in blue in the revised PDF. The main changes are:

  1. Downstream quality is now the main evaluation (Table 2, Section 6.3). We report every method on full GSM8K (3 seeds x 1,319 items) and full ASDiv (3 x 2,232) with paired bootstrap CIs, under two metrics: the strict exact match of the LESS evaluator, as in the submission, and answer-level exact match, which scores the answer before the model starts a new few-shot item. The difference between the two is whether the fine-tuned model ends its answer, which the strict protocol also scores (new Appendix F).
  2. Per-stratum allocation of the refresh budget is now the default of CDIS for stratified rules (Section 4.2). With it, CDIS at p = 0.3 matches full recomputation under both metrics on Qwen3-4B: -1.1 [-2.2, 0.0] (strict) and +0.4 [-0.4, 1.1] (answer-level) points on GSM8K, and -0.6 [-1.4, 0.2] and -0.1 [-0.5, 0.2] on ASDiv. The global allocation used in the submission trails it by 3.2 [1.9, 4.4] strict points, which the selection-level analysis predicted.
  3. Theory that yields a usable rule (Section 4.2). Proposition 1 now only requires the refresh set to cover a band around the selection threshold. This generalizes the condition: p >= k/N is needed only when the refresh set is chosen by stale rank, and a refresh set centered on the threshold can be smaller. From it we derive an adaptive rule that sets p from the residuals observed on the refreshed examples.
  4. Refresh strategies and calibration maps (new Appendix L): boundary-aware band, per-stratum variants, previous-drift (an uncertainty proxy), random refresh; identity, affine, quadratic, isotonic, per-source, and nearest-neighbour calibration.
  5. New models, benchmarks, and baselines: Qwen3-8B-Base with its own warmup, features, and selections (fidelity in Table 1, downstream with three training seeds in Appendix F); MMLU-target selections evaluated on the full MMLU test set (Table 7); the BIDS-style balanced-influence baseline on three seeds (Table 21); and the non-gradient representation-based (RDS) baseline of LESS (Table 6).
  6. Presentation: statements moved before the references as the template requires; refresh and selection rule separated in Section 4; Figure 1 y-axis, Figure 2 size, Figure 3(b) legend, and Figure 4 labels fixed; related work on reusing stale scores (lazy greedy evaluation, TracIn/TRAK/LoGra datastores, CRAIG/GradMatch, online selection) added in Section 2.
AuthorsOpenReview ↗

General response (2/2): CDIS matches full recomputation on every model and benchmark

Across all new experiments, the central claim of the paper holds: given a gradient-based selector, CDIS reproduces its selected subset and its downstream result at a third of the gradient cost.

SettingCDIS minus full recomputation [95% CI], points
Qwen3-4B, GSM8K, strict / answer-level-1.1 [-2.2, 0.0] / +0.4 [-0.4, 1.1]
Qwen3-4B, ASDiv, strict / answer-level-0.6 [-1.4, 0.2] / -0.1 [-0.5, 0.2]
Qwen3-4B, MMLU (MMLU-target selection)+0.10 [-0.04, 0.24]
Qwen3-8B, GSM8K, answer-level-0.4 [-1.1, 0.4]
Qwen3-8B, GSM8K, strict-3.0 [-4.2, -1.8], within the training-seed spread of full recomputation (0.216 to 0.306)
Selection fidelity, 8 settings, top-k at p = 0.3exact recovery in every setting

The new baselines also clarify what drives downstream accuracy on this testbed. Both base models already answer about 80% of GSM8K items, so the selected subset changes accuracy mainly through whether the fine-tuned model ends its answer after the solution, which the strict protocol of LESS scores: the stratified rule raises strict exact match by 16.4 [14.8, 18.0] points over random selection (70% against 51% of responses end after the answer) and answer-level accuracy by 1.3 [0.2, 2.3] points (2.0 [1.0, 3.0] on Qwen3-8B). The same effect explains why the non-gradient RDS baseline, whose subset is mostly Dolly examples with cleanly ending responses, scores well here (Table 6). Which selector suits a given task is a separate question from the one CDIS answers: CDIS makes gradient-based selection cheaper and preserves the selected subset under every selection rule we tested (overlap with full recomputation 0.92 to 1.00 for top-k, stratified quotas, BIDS-style normalization, and cluster quotas).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.