acceptodds
Under review as a conference paper at ICLR 2027

DRIFT: Data Selection for LLM Instruction Tuning via On-Policy Attribution

10 views

Abstract

Data selection for instruction tuning is usually studied for efficiency, reaching strong performance with a fraction of the training data. We ask whether data selection can instead raise the performance ceiling, producing a better model than training on all the available data. To test this, we take a large language model (LLM) that has already completed supervised fine-tuning (SFT) on its full corpus and train it further on examples selected from that same corpus. Any gain therefore lifts the model above the ceiling set by full-data training. Existing selection methods yield little or no gain in this setting. We propose DRIFT (Data Refinement via On-Policy Influence Functions for Supervised Fine-Tuning), a data attribution method built on influence functions. DRIFT samples the model's own responses to validation queries and weights them by correctness to form a validation objective. It ranks the examples of the original corpus by their estimated influence on this objective, and the model then continues SFT on the top-ranked subset. This design is motivated by the proximity gap in influence function theory: we hypothesize that these on-policy responses are better validation targets than external references in terms of locality. DRIFT also corrects influence scores for their dependence on gradient norm within each validation task. DRIFT outperforms all evaluated data selection baselines on Olmo3-7B-Instruct-SFT and OpenR1-Distill-7B, raising average accuracy across eight benchmarks from 39.17 to 40.14 and from 57.23 to 58.66, respectively, with gains on benchmarks not used for attribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data selection, instruction tuning, supervised fine-tuning, influence functions, data attribution, on-policy, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Uu6TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DRIFT for selecting instruction data to improve LLM performance. Unlike traditional instruction data selection methods, where selection is performed off-policy and the candidate data are not fine-tuned by the LLM, DRIFT performs on-policy selection by first fine-tuning the data with the LLM and then performing selection. The main idea is to use the model’s on-policy rollouts with weights to define a proximal validation objective for influence scoring. The experiments were conducted on the Dolci-Instruct-SFT dataset using the OLMo3-7B-Instruct-SFT and OpenR1-Distill-7B models.

Critical strengths
  1. Almost all existing data selection methods are off-policy. This paper provides a different perspective and shows that on-policy selection can also be beneficial for instruction data selection.

  2. The authors also show that debiasing the influence scores (correcting gradient-norm bias) improves model robustness.

Critical weaknesses
  1. From my understanding, Table 1 reports continual-SFT results. However, most instruction data selection methods do not require full-corpus fine-tuning. It is important to also report the performance of the original selection methods. For each method, comparison against “base” matters, but it is also important to analyze performance variation with and without full-corpus fine-tuning.

  2. The selection cost of DRIFT is high because it requires full-corpus fine-tuning before selection. Therefore, a comparison with selection methods that do not require full-corpus fine-tuning is necessary to demonstrate the benefits of DRIFT.

Questions for authors
  1. It looks surprising that in Table 1, random selection performs worse than Base. Can the authors provide an explanation for this case?

  2. How is LESS implemented in your setting? Since LESS also uses a corpus (a small fraction) to fine-tune the LLM, is the version in Table 1 obtained simply by replacing that small fraction with the whole corpus, without changing the other steps in LESS?

  3. Can authors explain what each task evaluator is and if LLM-as-judge used for the evaluation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Uu6T (1/2)

Thank you for the constructive review. We added the comparison you asked for and answer each question below.

W1. Selection methods without full-corpus fine-tuning

The subset each method selected for Table 1 is used to train the pre-SFT model (Olmo-3-7B before instruction tuning) with the same 500-step schedule, one run per method. BM25, DSIR, and Qurating do not depend on the model, so for them this is exactly their original setting.

All (%)Without full-corpus SFT (subset, from pre-SFT model)With full-corpus SFT (Table 1)Difference
SFT checkpoint, no further trainingn/a39.17
Uniform35.1039.04+3.94
DSIR26.7234.37+7.65
Qurating37.0539.26+2.21
BM2536.6439.54+2.90
LESS35.9737.63+1.66
RDS36.2339.08+2.85
IF (GraSS)34.8938.66+3.77
DRIFT35.2240.14+4.92

The left column is the efficiency setting these methods were designed for. Every result there is below the full-SFT checkpoint, the best one by 2.1 points, and every method ends 1.7 to 7.6 points higher when it starts from that checkpoint. The ranking also differs. Qurating and BM25 lead in the efficiency setting and add little on top of full SFT, where DRIFT leads. Our question is whether selection can raise the ceiling of full-data training, which is why Table 1 compares all methods from the SFT checkpoint.

W2. Cost of DRIFT

The comparison with methods that do not require full-corpus fine-tuning is the table under W1. Used that way they are cheaper and reach at most 37.05, against 39.17 for the full-SFT checkpoint and 40.14 after DRIFT.

Within Table 1, full-corpus SFT is the shared starting point of every row, here the publicly released checkpoint, so the cost DRIFT adds is its selection stage. "On-policy" refers to this stage, where the validation targets are responses sampled from the current model. Appendix A.5 reports the cost in hours on one eight-GPU node.

SelectionContinued SFTTotalAll
BM254.26.610.839.54
IF (GraSS)12.56.619.138.66
RDS19 to 226.626 to 2939.08
DRIFT20.56.627.140.14
BM25, 2.5× data and steps4.216.520.739.66
DRIFT, 4 rollouts per query (rollout time est.)13.56.620.139.87

DRIFT's cost is in the range of the other gradient- and representation-based selectors, and twelve of its 20.5 selection hours are corpus gradient features that are reusable across validation sets. At a matched wall-clock budget, a cheaper selector does not close the gap by training longer.

AuthorsOpenReview ↗

Response to Reviewer Uu6T (2/2)

Q1. Random below Base

Random continues training on batches drawn uniformly from the full corpus. Its difference from Base is −0.13, within its standard deviation across seeds of 0.15, so it is better read as no change (ZebraLogic and OlympiadBench go up, LiveCodeBench and GPQA-D go down). This is what we expect from a checkpoint that is at or close to the ceiling of training on this corpus, and it is the reason we start from it. More training on the same distribution does not move the model, while DRIFT's selection under the identical schedule does (+1.10 over Random).

Q2. LESS implementation

Not exactly, since we do not run a separate warm-up. LESS warms the model up on a fraction of the candidate data before extracting gradients, and here gradients are extracted directly from the starting checkpoint, which has already been trained on the whole corpus. The scoring rule is that of LESS, with reference responses for validation gradients, cosine similarity, and the maximum over validation subtasks as an example's score. Two things are adapted for scale and shared with IF (GraSS) and DRIFT. Gradients are taken over full parameters and compressed with the GraSS sparse projection, where LESS uses LoRA gradients with a dense random projection. And we use plain CE gradients from a single checkpoint, where LESS aggregates Adam-preconditioned gradients over several warm-up checkpoints. We will label the row "LESS (adapted)", list these differences in Appendix A.2, and correct the warm-up wording there and in the cost table.

Q3. Evaluators

No LLM judge is used anywhere. All eight benchmarks are scored by the programmatic checkers of OLMES and LightEval.

BenchmarkCheckerDecoding
BBHanswer extraction, exact matchgreedy
MATHanswer extraction, mathematical equivalencegreedy
MMLU-Prochoice-letter extraction and matchgreedy
MBPP+execution against EvalPlus tests8 samples
LiveCodeBenchexecution against test cases4 samples
ZebraLogicfull-grid match with the solution4 samples
OlympiadBenchanswer extraction and match4 samples
GPQA Diamondchoice-letter extraction and match16 samples

Sampled benchmarks use temperature 0.6 and top-p 0.95, and the score is the mean accuracy over samples, not best-of-n. The correctness labels of the validation rollouts used for attribution also come from programmatic checkers. The full evaluation configuration will be released with the code.

Thank you for including the comparison I requested. It addresses most of my concerns, though I still have some reservations about DRIFT’s practical advantage over BM25. When training starts from the pre-SFT model, BM25 outperforms DRIFT by 1.4 points. After full-corpus SFT, DRIFT leads by 0.6 points, while data selection takes 20.5 hours for DRIFT compared with 4.2 hours for BM25. That is a modest gain given the additional cost. However, I think selecting data for continued training after a model has already seen the full corpus maybe a worthwhile problem, and DRIFT shows a benefit in that setting. I’m therefore keeping my score at 3.

Thank you for the thoughtful follow-up. We appreciate your assessment and will make the performance–cost trade-off with BM25 clearer in the revision.

Reviewer_NkoUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper researches the question of whether data selection would be able to raise performance above the level of full-data SFT and proposes an approach called DRIFT. It takes a model that has finished SFT and continues training it on a selected subset of the same corpus. DRIFT samples the model's own responses to validation queries and weights them by correctness. The approach ranks examples by influence on the particular objective and does a per-task correction for gradient-norm dependency.

The motivation of this paper comes from a proximity-gap argument from influence-function theory and related research. On Olmo3-7B-Instruct-SFT, average accuracy over 8 benchmarks rises from 39.17 to 40.14. On OpenR1-Distill-7B it rises from 57.23 to 58.66. DRIFT beats all evaluated selection baselines. Validation includes three seeds on OLMO-3 (one run per method on OpenR1), ablations on signed rollouts and the norm correction, and a signed-BM25 control.

Critical strengths

The paper asks an interesting question of whether examples the model has already used for training can still improve it.

Existing selection approaches give little or no gain here, so the setting is informative and interesting to data-centric research. Signed incorrect rollouts help over positive-only rollouts. Replacing influence scoring with signed BM25 does not reproduce the improvement. The appendix also states where the implementation departs from the theory.

Critical weaknesses

1, The ceiling claim is not clearly tested. Every method studied in the paper (pls correct me if there is any misunderstanding), including Random, continues from the checkpoint on a 10% subset. No control continues on the full corpus under the same token budget and schedule.

  1. The implementation of this algorithm uses temperature and top-p sampling and a token-mean loss. Meanwhile, the paper's appendix says these choices break the exact expected-correctness gradient. It also says the corrected score is not an inner product with one fixed validation gradient. Centroid-norm and drift plots are indirect. The data cannot separate locality from reward signing, hard-negative contrast, or the nonlinear norm transform. The influence-function and proximity-gap framing needs evidence that the score predicts actual local validation change.
Questions for authors

2 main questions:

  1. how does DRIFT compare with uniform sampling from the full original corpus across independent seeds? If the two match, will the ceiling claim be withdrawn? (the same checkpoint, continuation budget, and learning-rate schedule).
  2. On a tractable held-out slice, do the raw and corrected scores predict realized validation-objective change after local retraining better than matched reference-based scores? If they do not, will the paper present DRIFT as an empirical signed-rollout heuristic?
4 comments
AuthorsOpenReview ↗

Response to Reviewer NkoU (1/2)

Thank you for two precise questions. The control in Q1 is the Random row of Table 1, and we ran the local-retraining test in Q2.

W1, Q1. Uniform continuation on the full corpus

The Random row of Table 1 is this control. It draws batches uniformly from the entire corpus of 1.92M examples, from the same checkpoint, with the same 500 steps, batch size, and learning-rate schedule as DRIFT. This budget covers 0.107 epochs of the corpus, so no example is repeated and the run is equivalent to one pass over a fresh uniform 10% sample, which is how Appendix A.2 describes it. We will rename the row "Uniform (full corpus)" to make this explicit.

All, three seedsMean ± SD
SFT checkpoint39.17
Uniform (full corpus)38.87, 39.15, 39.0939.04 ± 0.15
DRIFT40.12, 40.27, 40.0240.14 ± 0.12

The two do not match. The gap is 1.10 points with no overlap between seeds, and uniform continuation leaves the checkpoint where it was. The ceiling claim therefore stands. Selection lifts the model above what full-data SFT reached, and above continued uniform training under the same budget.

On seeds, the rows above are the three training seeds per method behind Table 1. Two further sources of variation are covered elsewhere in the paper. Re-running selection with new rollout and projection seeds gives subsets with Jaccard overlap 0.69 to 0.73 with the original, and their downstream scores are 40.03, 40.19, and 40.08 (Appendix A.3). The eight other signed on-policy configurations in Tables 2 and 4 are eight further subsets with three seeds each, with means between 39.4 and 40.0. None of these runs is below 39.26, which is above every Uniform run and above the starting checkpoint.

AuthorsOpenReview ↗

Response to Reviewer NkoU (2/2)

W2, Q2. Whether the score predicts realized local validation change

We ran this test with raw and corrected scores against matched reference-based scores.

Protocol. From the SFT checkpoint we fix 4,096 candidate examples and split validation queries into disjoint sets A and B (116 queries and 1,280 rollouts each, over the four validation tasks). All scores and correction fits use A only. The reference-based comparator uses the same A queries with their reference responses and the same candidate gradients, projection, task aggregation, and correction. We form twelve subsets of 160 examples, the top and bottom of each of the four scores and of the on-policy-minus-reference rank gap (raw and corrected), matched in source and length composition and fixed before any outcome is measured. Each subset is trained for 20 steps from the SFT checkpoint. We then measure ΔJ_B, the change on held-out B of the implemented validation objective (signed, token-mean), where a positive value means correct rollouts become more likely and incorrect ones less likely.

ScoreΔJ_B, top-ranked subset (×10⁻³)ΔJ_B, bottom-ranked subset (×10⁻³)Spearman ρ, predicted vs. realized, 12 interventions [95% CI]
On-policy, raw−0.53−4.30+0.65 [0.59, 0.76]
On-policy, corrected (DRIFT)+0.92−4.64+0.75 [0.67, 0.83]
Reference, raw−4.70−0.40−0.81 [−0.85, −0.73]
Reference, corrected−4.46+1.26−0.87 [−0.91, −0.79]

Intervals are from 5,000 bootstrap draws over the queries of B.

On-policy scores order the interventions in the direction of the realized held-out change, both raw and corrected. Reference-based scores order them in reverse at this checkpoint, and the subsets they rank highest lower the objective about as much as the subsets that on-policy scores rank lowest. This agrees with the downstream results, where reference-based selection ends below the starting checkpoint (38.62 and 38.66) and on-policy selection above it. The norm correction is not what produces the difference, since the on-policy score is already predictive without it and the reference score stays reversed under the identical transform.

Separating the components. Tables 2 and 4 vary them one at a time.

Change, all else fixedAll
Reference response → one correct on-policy response (no negatives)38.62 → 39.53
Add one incorrect response with weight −139.53 → 39.97
All rollouts, incorrect ones weighted +1 instead of −140.14 → 39.03
Norm correction off → on, full objective39.43 → 40.14

Correct on-policy targets help before any negatives enter. The sign carries the rest, since adding incorrect responses with weight −1 helps and giving them weight +1 removes the gain. The norm correction adds on top at a fixed objective. This is the reading in Section 5.1, where on-policy targets keep the selection local and the correctness signs determine which local changes are useful.

Framing. The score predicts realized local change, so we keep the influence-function framing and the proximity-gap motivation. The test is run on the implemented score, with temperature and top-p sampling, the token-mean loss, and the norm correction all in place, so the departures from the exact gradient noted in Appendix D do not remove its predictive value. We will add this experiment to the paper as direct evidence that the on-policy score tracks local retraining, ahead of the drift and centroid diagnostics.

Thanks. In this case, it seems that Uniform and DRIFT are not statistically separable.

AuthorsOpenReview ↗

Re: Statistical separation of Uniform and DRIFT (reply to Reviewer NkoU)

Thank you for the reply. We respectfully disagree. The numbers in our response show that the two are clearly separated.

AllThree seedsMean ± SD
SFT checkpoint39.17
Uniform (full corpus)38.87, 39.15, 39.0939.04 ± 0.15
DRIFT40.12, 40.27, 40.0240.14 ± 0.12
  • The difference in means is 1.10 points, about eight pooled standard deviations. The lowest DRIFT seed is 0.87 points above the highest Uniform seed.
  • Welch's t-test gives t = 9.9 and p < 0.001 (two-sided). The 95% confidence interval of the difference is [0.79, 1.41].
  • A test without distributional assumptions agrees. With the three DRIFT runs on re-selected subsets from Appendix A.3 (40.03, 40.19, 40.08), all six DRIFT runs are above all three Uniform runs. If the two methods were interchangeable, this ordering would occur with probability 1/84 (one-sided p = 0.012).
  • DRIFT is higher than Uniform on each of the eight benchmarks in Table 1. The pair that is not separable is Uniform and the starting checkpoint (39.04 ± 0.15 vs. 39.17). If the comment refers to this pair, we agree, and it is what the control is meant to show. Continued uniform training leaves the model where it was, and selection moves it.

If the concern is a different one, such as a specific test or number of seeds you would like to see, please let us know and we will address it directly.

Reviewer_nPgtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

An interesting perspective, though its validity remains questionable.

This paper examines whether selecting instruction-tuning data can improve performance relative to full-corpus training, particularly after full supervised fine-tuning (SFT), by continuing training on selected subsets. The paper proposes DRIFT, an influence-function-based method that uses the model’s on-policy responses as validation targets and corrects gradient-norm bias. It outperforms baselines on two 7B models and increases average accuracy. I find this comparative perspective novel, yet I question whether such a setting exists in practice.

Critical strengths
  1. The work addresses a highly timely problem: modern large language models often require instruction tuning for downstream tasks, and data selection can improve both the efficiency and effectiveness of instruction tuning.

  2. The experimental perspective is comprehensive. The appendix includes extensive experimental details, additional analyses, and comparisons, strengthening the reliability of the results.

  3. Figures and tables are well-formatted and easy for readers to follow.

Critical weaknesses
  1. The primary concern is whether the proposed application scenario is practical or necessary. (1) Why perform SFT on the full dataset first and then retrain on a selected subset? The introduction fails to convincingly establish the realism of this setup. The standard paradigm compares training on selected subsets directly from the base LLM with full-dataset training, which aligns with practical constraints. The motivation for this two-stage training remains unclear throughout the paper. (2) The experimental results do not strongly support the necessity of this pipeline. As shown in Table 1, the performance gap between the Base model and DRIFT is small, with gains of only 1–2%. DRIFT performs worse than the Base on two benchmarks (MBPP+ and GPQA). (3) A critical comparison is missing: is this two-stage training superior to single-stage training? For instance, does full SFT followed by DRIFT fine-tuning outperform baselines trained directly on the original LLM using the selected subset of the training dataset?

  2. The evaluation is limited to two 7B models, which is insufficient to establish the method's broad applicability. Experiments on smaller (e.g., 1B) and larger (14B, 30B) models should be included to strengthen the conclusions.

  3. The introduction is poorly structured. (1) Some paragraphs lack clarity on their purpose, such as the section discussing “A second issue is gradient norm bias.” (2) The Olmo3-7B model is introduced without prior context. It should be properly framed (e.g., “We use Olmo3-7B as one of our studied LLMs”) and accompanied by a citation.

Questions for authors
  1. Why do we need to perform SFT on the full dataset first and then retrain on a selected subset?

  2. Does the proposed method DRIFT still work on smaller or larger models?

4 comments
AuthorsOpenReview ↗

Response to Reviewer nPgt (1/2)

Thank you for the detailed review. Your two main requests, a single-stage comparison and a smaller model, are both addressed with new experiments.

W1(1), Q1. Motivation for the setting

Data selection methods are designed for efficiency, and the common practice is to compare a small selected subset with a random subset, both trained under the same small budget. For SFT this evaluation is detached from practice. SFT is not resource-limited and models are trained on all available data, so what data curation should deliver here is a higher ceiling, a better model than full-data training gives.

Our setting tests this directly. We start from a model that has completed full training and is therefore at or close to that ceiling ("Base" in Table 1), and ask whether selected data can lift it further. Any gain is a gain over full-data training. It is also a stricter test than the efficiency setting, since a method has to improve a fully trained model and beating a random subset is not enough. Existing selectors add at most 0.37 points here and several fall below the starting checkpoint, while DRIFT adds 0.97 (Table 1).

We will state this motivation at the start of the introduction and rename "Base" to "SFT checkpoint".

W1(3). Two-stage versus single-stage training

We ran this comparison. The subset each method selected for Table 1 is used to train the pre-SFT model (Olmo-3-7B before instruction tuning) with the same 500-step schedule, one run per method.

All (%)Single-stage: subset, from pre-SFT modelTwo-stage: subset, from SFT checkpoint (Table 1)
SFT checkpoint, no further trainingn/a39.17
Uniform35.1039.04
DSIR26.7234.37
Qurating37.0539.26
BM2536.6439.54
LESS35.9737.63
RDS36.2339.08
IF (GraSS)34.8938.66
DRIFT35.2240.14

Full SFT followed by DRIFT outperforms every baseline trained directly on the pre-SFT model with its selected subset, by at least 3.1 points. Every single-stage result is also below the full-data SFT checkpoint itself, the best one by 2.1 points.

AuthorsOpenReview ↗

Response to Reviewer nPgt (2/2)

W1(2). Size of the gain

The starting checkpoint is at or close to the ceiling of full-data training, so the bar is whether a method improves it at all.

Continued training from the SFT checkpointChange in All
Uniform−0.13
DSIR, LESS, IF (GraSS), RDS−4.80 to −0.09
Qurating+0.09
BM25 (best existing selector)+0.37
DRIFT+0.97

DRIFT's gain is 2.6 times that of the best existing selector, and each DRIFT seed (40.02 to 40.27) is above every seed of every baseline (at most 39.70). The gain comes entirely from re-selecting data the model has already been trained on, and it carries over to the four benchmarks never used for attribution (+1.02).

On the two benchmarks you mention, MBPP+ is unchanged (56.40 vs. 56.42), and the GPQA-D difference of 0.44 is within its standard deviation across seeds of 0.92 on a 198-question benchmark. The other six benchmarks improve by 0.72 to 2.72 points.

W2, Q2. Smaller and larger models

We ran the full pipeline on OLMo-2-1B-SFT with its own SFT corpus (Tulu 3), recomputing rollouts, gradients, and selection at 1B. Both arms continue from the released SFT checkpoint with identical settings.

OLMo2-1B-SFT, single run per methodAll
SFT checkpoint13.07
Random 10%13.09
DRIFT13.33

In this single run, DRIFT improves the 1B checkpoint while random selection leaves it unchanged, the same pattern as at 7B. We will add this experiment to the appendix. For 14B to 30B models, the per-example gradient pass over the full corpus was beyond our compute within the rebuttal period.

W3. Presentation

We will restructure the introduction in the order motivation, setting, limitation of reference targets, method, results. Gradient-norm bias will be introduced as the second of DRIFT's two components, the validation objective and the scoring of examples against it, with details moved to Section 3. Olmo3-7B-Instruct-SFT will be introduced as the main model we study, with its citation, at first mention.

Thanks for the authors’ response. (1) However, I feel the authors may not have fully addressed the focus of my first question. I intended to ask: under what conditions is it necessary to fine-tune on the full dataset first and then select a subset for a second round of fine-tuning? Is this training paradigm proposed by the authors, or is it supported by prior literature? (2) For W2, Question 2, the authors provided experiments on small models, but results for larger models are missing. This addresses part of my concern.

AuthorsOpenReview ↗

Follow-up response to Reviewer nPgt

Thank you for clarifying your question.

(1) When the full-data round comes first, and whether the schedule is ours

We start from the completed full-data SFT checkpoint because we want to test whether a model can still improve by revisiting examples it has already been trained on, that is, whether selection can lift it above what full-data training reached. The setting therefore applies whenever a full-data SFT checkpoint and its training corpus are available. OLMo3 is a concrete case, as both its SFT checkpoint and its SFT corpus are public. We do not claim that this schedule is necessary when starting from a pretrained model, and our experiments do not establish that.

There is also a technical reason to score examples at the checkpoint we intend to improve. Classical influence analysis considers perturbations around a stationary solution. At a checkpoint that has not finished training, continued optimization on unchanged data already moves the model, and retraining then reflects this movement as well as the effect of the data change [1]. This motivates computing attribution after the full SFT schedule, where we treat the completed checkpoint as approximately stationary (Section 2.2).

Each component has precedent. In SFT, Humane et al. [2] fine-tune on the full LIMO dataset for ten epochs and compute influence at the resulting checkpoint to rank its training examples. In pretraining, Llama 3 [3], Blakeney et al. [4], and OLMo 2 [5] change the data mixture late in training. Our experiment combines these elements in a specific way. It scores individual examples with the current SFT model's own correctness-weighted responses, continues from that same checkpoint on the selected examples, and asks whether the result improves on the original full-data model.

For someone who holds the SFT checkpoint, the comparison on OLMo3-7B is as follows.

Eight-benchmark averageAll
Full-data SFT checkpoint39.17
Best run trained directly from the pre-SFT model (Qurating subset, 500 updates)37.05
Uniform continuation from the SFT checkpoint (500 updates)39.04
DRIFT continuation from the SFT checkpoint (500 updates)40.14

We will make this use case and its relation to prior work explicit in the introduction.

(2) Larger models

We agree that the 1B experiment answers only part of your question on scale. Our results cover 1B and 7B, and we have none for 14B or 30B. At 7B, extracting and projecting per-example gradients for the full corpus took about 12 hours on eight A800 GPUs. We will state this limit in the paper.

[1] Bae et al. If Influence Functions are the Answer, Then What is the Question? NeurIPS 2022, Section 4.3.

[2] Humane et al. Influence Functions for Efficient Data Selection in Reasoning, Section 3.1.

[3] Grattafiori et al. The Llama 3 Herd of Models, Section 3.1.3.

[4] Blakeney et al. Does your data spark joy? Performance gains from domain upsampling at the end of training.

[5] Team OLMo. 2 OLMo 2 Furious, Section 2.4.2.

Comments on the paper

2 comments

Hi all,

Reviewers: Please respond to the authors' rebuttal if you haven't already. If you feel that your concerns have not been sufficiently addressed, please state this clearly so the authors have an opportunity to respond.

AuthorsOpenReview ↗

Summary of the Discussion

We thank the reviewers and the Area Chair. Since the discussion is spread over several threads, we summarize here what the paper does and where each reviewer's concerns stand in our view.

The paper asks whether data selection can make a model better than training on all of its data, instead of making training cheaper. To test this, we take a model that has finished instruction tuning on its full corpus and keep training it on examples chosen from that same corpus. The model has already seen every example, so more training alone should not help, and in our experiments it does not. Most existing selection methods do not help either, and several make the model worse. This is a harder test than the usual comparison against random subsets. All three reviews note the novelty or interest of this perspective.

DRIFT passes this test. It lets the model answer a set of validation questions many times, marks each answer as right or wrong, and uses influence functions to find the training examples that would make the right answers more likely and the wrong ones less likely. Earlier influence-based selection uses reference answers as the target instead. On Olmo3-7B, DRIFT raises the eight-benchmark average from 39.17 to 40.14, including on four benchmarks never used for selection. The best existing method reaches 39.54, and continued training on the full corpus stays at 39.04.

Reviewer Uu6T asked how the selection methods perform without full-corpus fine-tuning and how DRIFT's cost compares with them, and requested details on the Random row, the LESS baseline, and the evaluators. We ran the requested comparison. Trained on their selected subsets from the pre-SFT model, none of the eight methods reaches the fully trained model, and the best is about two points below it. The remaining questions were answered with the cost breakdown and implementation details. We have not received a follow-up and believe these points are addressed.

Reviewer nPgt raised the same question about the setting, together with model scale and presentation. The comparison above answers whether two-stage training outperforms single-stage training. In a follow-up, the reviewer asked when the full-data round is needed and whether the protocol has precedent. We replied that the setting is the improvement of an already trained model, and pointed to prior work that computes influence after fitting the full dataset. On scale, we added a 1B experiment, which the reviewer said addresses part of the concern. Results above 7B are a limitation that the paper states.

Reviewer NkoU asked for two checks. The first was continued training on the full corpus under the same budget. This is the Random row of Table 1, and it gives no gain. The reviewer's follow-up says that this control and DRIFT are not statistically separable, but their runs do not overlap (38.87 to 39.15 against 40.02 to 40.27). The second was whether our scores predict what happens after actual retraining. In a new held-out experiment they do, while scores computed from reference answers predict the opposite direction. This supports the central idea of the paper, and the reviewer has not commented on it.

What remains open is how far the result extends to larger models, and the result itself held up under each check the reviewers proposed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.