- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DRIFT for selecting instruction data to improve LLM performance. Unlike traditional instruction data selection methods, where selection is performed off-policy and the candidate data are not fine-tuned by the LLM, DRIFT performs on-policy selection by first fine-tuning the data with the LLM and then performing selection. The main idea is to use the model’s on-policy rollouts with weights to define a proximal validation objective for influence scoring. The experiments were conducted on the Dolci-Instruct-SFT dataset using the OLMo3-7B-Instruct-SFT and OpenR1-Distill-7B models.
Critical strengths
-
Almost all existing data selection methods are off-policy. This paper provides a different perspective and shows that on-policy selection can also be beneficial for instruction data selection.
-
The authors also show that debiasing the influence scores (correcting gradient-norm bias) improves model robustness.
Critical weaknesses
-
From my understanding, Table 1 reports continual-SFT results. However, most instruction data selection methods do not require full-corpus fine-tuning. It is important to also report the performance of the original selection methods. For each method, comparison against “base” matters, but it is also important to analyze performance variation with and without full-corpus fine-tuning.
-
The selection cost of DRIFT is high because it requires full-corpus fine-tuning before selection. Therefore, a comparison with selection methods that do not require full-corpus fine-tuning is necessary to demonstrate the benefits of DRIFT.
Questions for authors
-
It looks surprising that in Table 1, random selection performs worse than Base. Can the authors provide an explanation for this case?
-
How is LESS implemented in your setting? Since LESS also uses a corpus (a small fraction) to fine-tune the LLM, is the version in Table 1 obtained simply by replacing that small fraction with the whole corpus, without changing the other steps in LESS?
-
Can authors explain what each task evaluator is and if LLM-as-judge used for the evaluation.
4 comments
Response to Reviewer Uu6T (2/2)
Q1. Random below Base
Random continues training on batches drawn uniformly from the full corpus. Its difference from Base is −0.13, within its standard deviation across seeds of 0.15, so it is better read as no change (ZebraLogic and OlympiadBench go up, LiveCodeBench and GPQA-D go down). This is what we expect from a checkpoint that is at or close to the ceiling of training on this corpus, and it is the reason we start from it. More training on the same distribution does not move the model, while DRIFT's selection under the identical schedule does (+1.10 over Random).
Q2. LESS implementation
Not exactly, since we do not run a separate warm-up. LESS warms the model up on a fraction of the candidate data before extracting gradients, and here gradients are extracted directly from the starting checkpoint, which has already been trained on the whole corpus. The scoring rule is that of LESS, with reference responses for validation gradients, cosine similarity, and the maximum over validation subtasks as an example's score. Two things are adapted for scale and shared with IF (GraSS) and DRIFT. Gradients are taken over full parameters and compressed with the GraSS sparse projection, where LESS uses LoRA gradients with a dense random projection. And we use plain CE gradients from a single checkpoint, where LESS aggregates Adam-preconditioned gradients over several warm-up checkpoints. We will label the row "LESS (adapted)", list these differences in Appendix A.2, and correct the warm-up wording there and in the cost table.
Q3. Evaluators
No LLM judge is used anywhere. All eight benchmarks are scored by the programmatic checkers of OLMES and LightEval.
| Benchmark | Checker | Decoding |
|---|---|---|
| BBH | answer extraction, exact match | greedy |
| MATH | answer extraction, mathematical equivalence | greedy |
| MMLU-Pro | choice-letter extraction and match | greedy |
| MBPP+ | execution against EvalPlus tests | 8 samples |
| LiveCodeBench | execution against test cases | 4 samples |
| ZebraLogic | full-grid match with the solution | 4 samples |
| OlympiadBench | answer extraction and match | 4 samples |
| GPQA Diamond | choice-letter extraction and match | 16 samples |
Sampled benchmarks use temperature 0.6 and top-p 0.95, and the score is the mean accuracy over samples, not best-of-n. The correctness labels of the validation rollouts used for attribution also come from programmatic checkers. The full evaluation configuration will be released with the code.
Thank you for including the comparison I requested. It addresses most of my concerns, though I still have some reservations about DRIFT’s practical advantage over BM25. When training starts from the pre-SFT model, BM25 outperforms DRIFT by 1.4 points. After full-corpus SFT, DRIFT leads by 0.6 points, while data selection takes 20.5 hours for DRIFT compared with 4.2 hours for BM25. That is a modest gain given the additional cost. However, I think selecting data for continued training after a model has already seen the full corpus maybe a worthwhile problem, and DRIFT shows a benefit in that setting. I’m therefore keeping my score at 3.
Thank you for the thoughtful follow-up. We appreciate your assessment and will make the performance–cost trade-off with BM25 clearer in the revision.
Response to Reviewer Uu6T (1/2)
Thank you for the constructive review. We added the comparison you asked for and answer each question below.
W1. Selection methods without full-corpus fine-tuning
The subset each method selected for Table 1 is used to train the pre-SFT model (Olmo-3-7B before instruction tuning) with the same 500-step schedule, one run per method. BM25, DSIR, and Qurating do not depend on the model, so for them this is exactly their original setting.
The left column is the efficiency setting these methods were designed for. Every result there is below the full-SFT checkpoint, the best one by 2.1 points, and every method ends 1.7 to 7.6 points higher when it starts from that checkpoint. The ranking also differs. Qurating and BM25 lead in the efficiency setting and add little on top of full SFT, where DRIFT leads. Our question is whether selection can raise the ceiling of full-data training, which is why Table 1 compares all methods from the SFT checkpoint.
W2. Cost of DRIFT
The comparison with methods that do not require full-corpus fine-tuning is the table under W1. Used that way they are cheaper and reach at most 37.05, against 39.17 for the full-SFT checkpoint and 40.14 after DRIFT.
Within Table 1, full-corpus SFT is the shared starting point of every row, here the publicly released checkpoint, so the cost DRIFT adds is its selection stage. "On-policy" refers to this stage, where the validation targets are responses sampled from the current model. Appendix A.5 reports the cost in hours on one eight-GPU node.
DRIFT's cost is in the range of the other gradient- and representation-based selectors, and twelve of its 20.5 selection hours are corpus gradient features that are reusable across validation sets. At a matched wall-clock budget, a cheaper selector does not close the gap by training longer.