acceptodds
Under review as a conference paper at ICLR 2027

Optimizing Training Data via Metagradient Descent

Abstract

Curating a training dataset from a mass of unfiltered data is a necessary and influential step in any large-scale machine learning pipeline. We frame dataset selection as an optimization problem over per-example data counts and solve it with metagradient descent (MGD). At a high level, our algorithm operates as follows: we start with a randomly chosen set of training data, then iteratively update the dataset selection using metagradients with respect to importance weights placed on each training datapoint. To do this at scale, we develop: (1) REPLAY, an algorithm and implementation for computing scalable metagradients using recursive checkpointing and efficient differentiation of individual training steps; (2) a diagnostic for *metasmoothness*, allowing us to modify the training routine to obtain useful gradients; (3) a continuous surrogate for optimizing data counts; and (4) a block-SGD-inspired algorithm for improving data optimization. Combining these four contributions, MGD reaches average scores of 22% on DataComp-small and 40.2% on DataComp-medium, and improves over LESS and full-data training for Gemma-2B instruction tuning on BBH and MMLU. These results show that repeated gradient-based dataset updates can improve both pre-training and instruction-tuning data.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
metagradient, optimization, data selection, datacomp, data poisoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kzRMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper optimizes training datasets by iteratively changing how many times each example appears. It computes metagradients through a modified training procedure, uses these gradients to update discrete data counts, and then trains the final model on the resulting dataset with the original training routine. The method combines recursive checkpointing, interventions intended to make metagradients useful, a continuous insertion surrogate, and block updates to data counts. Experiments cover CLIP pretraining on DataComp and Gemma-2B instruction tuning.

Critical strengths

The paper demonstrates a concrete application of differentiation through substantial training suffixes to dataset optimization. Computing these gradients and repeatedly updating a large candidate dataset is nontrivial, and the DataComp results are encouraging: 22% vs. 18.2% for the reported strongest baseline at small scale, and 40.2% vs. 38.8% at medium scale. These results make the approach worth investigating, although the selection protocols and costs need closer comparison.

Critical weaknesses
  • The contribution relative to closely related work is not sufficiently established. The basic formulation: changing training-example weights to improve a target loss through the resulting training trajectory is well established. Beyond classical influence functions and TracIn, HyDRA explicitly studies hypergradients with respect to data weights, SOURCE approximates unrolled differentiation for data attribution, and trajectory-specific influence methods account for when a sample is used during training. PDS and DataRater go further and use training-dynamics or metagradient information for data selection. The paper cites several of these lines of work, but the discussion and experiments do not adequately separate its contribution from them. In particular, comparison with LESS does not establish the benefit over methods that already propagate influence through subsequent training. The scalability contribution of REPLAY also needs clearer positioning against existing checkpointing and efficient bilevel-differentiation methods, including MixFlow-MG. I would like to know whether the gain comes from a more informative gradient estimate, a longer differentiation horizon, repeated dataset updates, or the modified training procedure. A controlled comparison that changes the scoring method while keeping the outer dataset-update loop fixed would be more informative than comparing complete pipelines alone.

  • The cost-benefit tradeoff and the scope of the selected data remain unclear. Each dataset update requires another training run, followed by replay and differentiation, candidate scoring, and evaluation. These costs are additional to final model training. The paper provides a complexity discussion, but the reported final-task improvements do not by themselves show whether this is a worthwhile use of the total compute budget. An end-to-end accounting is needed, including the search for smooth training settings and the repeated dataset updates.

  • The intended use also depends strongly on the target data. This is reasonable for targeted instruction tuning, but broader claims about improving pretraining data require evidence that the selected dataset remains useful beyond the objectives used to select it. DataComp uses target examples from evaluation-task distributions, while the instruction-tuning experiments optimize hundreds of thousands of counts using very small target sets. The paper itself observes overfitting in the latter setting.

  • The metagradient is exact only for the continuous insertion surrogate. Its usefulness for finite changes to integer counts, followed by training with the original routine, is an empirical question. The positive results are encouraging, but the current evidence does not establish how reliably this transfer works across training configs. The instruction-tuning improvements are also reported without variation across training runs.

Questions for authors
  1. Can you compare long-horizon metagradients with short-horizon or approximate influence scores inside the same iterative dataset-update procedure? This would help determine whether the expensive gradient computation is necessary for the gains.

  2. What are the total GPU-hours and peak memory requirements for selection, including smoothness diagnostics, repeated training, replay, and candidate scoring? Under a matched total compute budget, how much advantage remains over a cheaper selection method?

  3. How well does a selected dataset transfer to a different model or training configuration, and to tasks not used in its optimization? Such evidence would help establish whether the selection cost can be amortized through reuse.

Reviewer_sHnqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates training-data selection as optimization over per-example integer counts and proposes metagradient descent (MGD) to update these counts iteratively. The method combines REPLAY for memory-efficient differentiation through training, a metasmoothness diagnostic, a continuous surrogate for data counts, and stochastic block updates. Selected data are evaluated using the original training routines. MGD achieves 22.0% and 40.2% on DataComp-small and medium, respectively, and improves Gemma-2B LoRA instruction tuning over full-data training and LESS on BBH and MMLU.

Critical strengths
  1. The paper provides a technically substantive implementation of differentiation through long training trajectories. Combining recursive checkpointing with efficient higher-order differentiation addresses a meaningful engineering barrier to applying metagradients to data selection.

  2. The DataComp results are encouraging: MGD improves over the reported strongest prior results by 3.8 percentage points at small scale and 1.4 points at medium scale. Transferring the selected data back to the original training routine strengthens the evidence that the benefits extend beyond the modified optimization setup.

Critical weaknesses
  1. Practical cost and scalability remain insufficiently established. Each dataset update requires a fresh training run, followed by trajectory replay and higher-order differentiation. REPLAY reduces memory requirements by adding recomputation, while the current implementation requires float32/tensorfloat32 operations and specialized higher-order support for custom kernels. However, the paper does not report sufficient end-to-end GPU-hours, peak memory, or performance–cost comparisons to establish when these expenses are justified. These limitations substantially weaken the practical significance of the proposed data-curation approach.

  2. Target-set overfitting undermines confidence in the reliability of the selected data. The instruction-tuning experiments optimize 270,679 data counts using only 23 BBH or 57 MMLU target examples, and Figure 6 shows a widening target–validation loss gap. Iterate selection also relies on small validation sets. Although benchmark test examples are held out, the reported gains over LESS—1.53 and 0.76 percentage points—lack estimates of variation across training runs.

Questions for authors
  1. What are the actual end-to-end selection costs for each experiment, including the number of outer updates, GPU-hours, hardware, and peak memory? How do these costs compare with LESS and simpler selection methods?

  2. What measurements identify the main scaling bottleneck as model size or differentiation horizon increases? In particular, how much of the demonstrated language-model feasibility depends on restricting training to LoRA parameters?

  3. Can data selected using a small proxy model transfer to larger models and different training budgets, or must MGD be rerun for each configuration?

Reviewer_j67NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Selecting a training dataset from a massive amount of data is a critical step in training a large-scale machine learning model. This paper considers data selection as an optimization problem over per-sample data counts and leverages matagradient to find the best candidate. In particular, data counts are first randomized, then iteratively modified using metagradients with respect to importance weights placed on each data point. To this end, this paper proposes REPLAY for computing scalable metagradients, metasmoothness, and a block-SGD-inspired algorithm for improving data optimization. The proposed method, MGD, empirically reaches not only good performance on DataComp but also on instruction tuning with Gemma-2B.

Critical strengths
  • This paper introduces a potential direction for directly optimizing the per-sample data counts based on metagradient descent.
  • The proposed method, MGD, demonstrates strong performance across tasks. MGD reaches an average score of 22% on DataComp-small, while it reaches 40.2% on DataComp-medium. In addition, on Gemma-2B instruction tuning, MGD achieves 36.7% on BBH and 42.5% on MMLU, improving over LESS and full-data training.
Critical weaknesses
  • This method required training the model several times on various per-sample data counts to find the best data counts. This makes data selection cost scale with the number of updates, which is infeasible for large-scale training.
  • As mentioned in the Limitations section, computing metagradients with REPLAY is excessively costly because metagradients require a backward pass through another backward pass, which takes about 2–3 times the operations of one backward pass. This cost may hinder its practical use in real-world settings.
Questions for authors
  1. Could you provide the wall-clock time for data selection of MGD compared to LESS?
  2. Given a new model and dataset, how do you find a proper value of kk?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.