- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper optimizes training datasets by iteratively changing how many times each example appears. It computes metagradients through a modified training procedure, uses these gradients to update discrete data counts, and then trains the final model on the resulting dataset with the original training routine. The method combines recursive checkpointing, interventions intended to make metagradients useful, a continuous insertion surrogate, and block updates to data counts. Experiments cover CLIP pretraining on DataComp and Gemma-2B instruction tuning.
Critical strengths
The paper demonstrates a concrete application of differentiation through substantial training suffixes to dataset optimization. Computing these gradients and repeatedly updating a large candidate dataset is nontrivial, and the DataComp results are encouraging: 22% vs. 18.2% for the reported strongest baseline at small scale, and 40.2% vs. 38.8% at medium scale. These results make the approach worth investigating, although the selection protocols and costs need closer comparison.
Critical weaknesses
-
The contribution relative to closely related work is not sufficiently established. The basic formulation: changing training-example weights to improve a target loss through the resulting training trajectory is well established. Beyond classical influence functions and TracIn, HyDRA explicitly studies hypergradients with respect to data weights, SOURCE approximates unrolled differentiation for data attribution, and trajectory-specific influence methods account for when a sample is used during training. PDS and DataRater go further and use training-dynamics or metagradient information for data selection. The paper cites several of these lines of work, but the discussion and experiments do not adequately separate its contribution from them. In particular, comparison with LESS does not establish the benefit over methods that already propagate influence through subsequent training. The scalability contribution of REPLAY also needs clearer positioning against existing checkpointing and efficient bilevel-differentiation methods, including MixFlow-MG. I would like to know whether the gain comes from a more informative gradient estimate, a longer differentiation horizon, repeated dataset updates, or the modified training procedure. A controlled comparison that changes the scoring method while keeping the outer dataset-update loop fixed would be more informative than comparing complete pipelines alone.
-
The cost-benefit tradeoff and the scope of the selected data remain unclear. Each dataset update requires another training run, followed by replay and differentiation, candidate scoring, and evaluation. These costs are additional to final model training. The paper provides a complexity discussion, but the reported final-task improvements do not by themselves show whether this is a worthwhile use of the total compute budget. An end-to-end accounting is needed, including the search for smooth training settings and the repeated dataset updates.
-
The intended use also depends strongly on the target data. This is reasonable for targeted instruction tuning, but broader claims about improving pretraining data require evidence that the selected dataset remains useful beyond the objectives used to select it. DataComp uses target examples from evaluation-task distributions, while the instruction-tuning experiments optimize hundreds of thousands of counts using very small target sets. The paper itself observes overfitting in the latter setting.
-
The metagradient is exact only for the continuous insertion surrogate. Its usefulness for finite changes to integer counts, followed by training with the original routine, is an empirical question. The positive results are encouraging, but the current evidence does not establish how reliably this transfer works across training configs. The instruction-tuning improvements are also reported without variation across training runs.
Questions for authors
-
Can you compare long-horizon metagradients with short-horizon or approximate influence scores inside the same iterative dataset-update procedure? This would help determine whether the expensive gradient computation is necessary for the gains.
-
What are the total GPU-hours and peak memory requirements for selection, including smoothness diagnostics, repeated training, replay, and candidate scoring? Under a matched total compute budget, how much advantage remains over a cheaper selection method?
-
How well does a selected dataset transfer to a different model or training configuration, and to tasks not used in its optimization? Such evidence would help establish whether the selection cost can be amortized through reuse.