acceptodds
Under review as a conference paper at ICLR 2027

Learning from Noisy, Diverse Sources with Dataweights

Abstract

Training specialized models increasingly requires learning from large, heterogeneous datasets whose examples vary in quality, domain relevance, and source of supervision. Dataweighting—assigning different weights to training examples—offers a general way to control how such data influences training, but learning effective weights at scale remains challenging. We develop a scalable approach to learning dataweights: given a small, high-quality target dataset, we use metagradients through the training process to efficiently optimize training-example weights directly for performance on the target domain. To scale beyond a fixed dataset and model, we learn parametric weighting rules that generalize to new and streaming examples and transfer from smaller models to substantially larger ones. We study reward modeling as a prototypical application, where preference data may be noisy, adversarial, synthetic, or mismatched with the target domain. Learned dataweights improve held-out performance, automatically adapt reward models to specialized domains, and allow mixtures of human and synthetic supervision to outperform either source alone. More generally, the objective used to learn dataweights need not match the loss used to train the model. We use this flexibility to improve downstream rankings while still training with the standard Bradley-Terry loss.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Data Attribution, Metagradients, Reward modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ubdyOpenReview ↗
Rating
2(−1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns per-example training weights from a small curated target set by running metagradient descent through segments of the training run. It also proposes a parametric version, a linear map on frozen text embeddings, that can score unseen examples and be transferred to larger models. On reward modeling with Arena human preferences, flagged bot data and Gemini synthetic labels, the authors report lower test loss under noise, poisoning, domain shift and human plus synthetic mixing, better results than TRAK, influence function and TracIn based selection, and better leaderboard ranking when a soft Spearman surrogate is used as the outer objective.

Critical strengths

S1: The objective is well defined and covers several settings. A single bilevel objective handles noise filtering, domain customization, mixing human and synthetic labels, and even a non-differentiable outer objective. The ranking experiment in Section 4.6 is interesting. Soft Spearman performs poorly as the training loss but helps when used only as the outer objective, improving from 0.541 to 0.625 on one split and from a median of 0.880 to 0.957 over seven splits.

S2: The comparison with attribution-based selection methods in Section 4.3 and Appendix D is done carefully. The number of optimizer steps is matched, every keep fraction has a random-drop control, the checkpoint selection set is separate, and four training pools are covered. The response length analysis in Figure 29 is also a valuable finding. When 10% of the data is kept, 80.6% of the pairs selected by TracIn have the shorter response as the chosen one, compared with only 40.3% in the full pool.

S3: The parametric weights are practical. As shown in Figures 5 and 6, with only 4,609 meta parameters they match the per-sample weights, can be applied to examples not seen during meta training, and transfer from 0.5B to larger reward models without recomputing metagradients.

Critical weaknesses

W1: Key baselines that use the same target set are missing, including but not limited to the following (a) A relevant baseline is density ratio weighting, using a classifier trained on the same frozen embeddings to distinguish target data from training data. This baseline is needed to tell how much of the gain actually requires metagradients. (b) The closest reweighting methods, such as Ren et al. (2018), Meta-Weight-Net (Shu et al., 2019) and DataRater (Calian et al., 2025), are discussed but not compared. As a result, the core claim of the paper, that exact metagradients over long training segments are better than short horizon approximations, is not tested. The instantaneous metagradient already shown in Figure 31 could serve as this ablation in the main text (c) In Figure 1(a), 1(b), 1(d) and Figure 4, the unweighted baseline does not use the 3,000 Expert target pairs used by dataweighting. A baseline that trains or fine-tunes directly on the target pairs appears only in the STEM experiment with a Human-33k base (Figure 2).

W2: The improvements are small relative to the noise, and some conclusions seem stronger than what the figures show. For example, on Expert, the gains in Figure 1(a) are 0.009, 0.005 and 0.003, and in Figure 1(b) they are 0.002, 0.002 and 0.000 at 10k, 60k and 100k. Only 30k and 45k show a clear gain. Figures 10, 15 and 16 also mention notable overfitting to the target set, but the paper does not explain how the number of outer iterations is chosen Second, the evaluation relies almost only on Bradley-Terry test loss. The weights are initialized at 1 and moved up or down by a fixed step, and the paper does not mention normalization, so the outer optimization may also change the effective learning rate or the scale of the rewards. This can lower log loss through calibration without improving ranking. In addition, a control that learns only a single global weight with the same metagradient is needed to show how much of the gain comes from per-example weighting. Finally, the scalability claims go beyond the evidence. The TL;DR says the method “scales for free to larger datasets and models”, but meta optimization is only run on the 0.5B model, and Appendix D shows that reaching the Pareto frontier on 40k to 77k examples takes 48.8 to 58.3 H200 GPU hours. The gain also becomes smaller when transferring to 7B. In Figure 6a, the test loss drops from about 0.503 to about 0.500 with per-sample weights and to about 0.491 with parametric weights. The theory in Section 5 studies exhaustive subset selection in a linear model, which the authors acknowledge is not the algorithm actually used

W3: There are several inconsistencies, and some figures are hard to read (a) Section 4.1 says that weighting restores performance in all cases except when the ratio of human to bot data is very high. However, in Figure 1(b) the gain is also almost zero at the lowest ratio, 10k human with 10k bot data, from 0.634 to 0.632. (b) The unweighted Spearman value on the single split in Table 1 is 0.541, well below the 25th percentile (0.717) of the seven splits in Table 2, and Figure 9 reports 0.65 on 92k training examples. The corresponding meta-optimized values are 0.625, 0.957 and 0.93. The paper does not explain how these setups relate to each other (c) Most of the appendix figures from Figure 10 to Figure 28 are segment-wise loss curves that look very similar, with little accompanying analysis, which makes them hard to read. Figure 1(c) and these curves distinguish outer iterations only by shades of blue, and the bar charts in Figures 1 and 6 also rely only on color. I suggest adding markers or hatching and using a colorblind friendly palette, or replacing many of the curves with tables for the key results, which would be clearer

Questions for authors

Q1 (see W1 ): Could the authors add the following baselines? (i) Density ratio weighting with a classifier trained on the same text-embedding-3-small features to separate target data from training data. (ii) Under the same compute, at least one short horizon reweighting method, such as L2RW, Meta-Weight-Net, or the instantaneous metagradient in Figure 31. (iii) Baselines that train or fine-tune directly on the target pairs in the Expert experiments.

Q2 (see W2 ): How many seeds are behind Figures 1 and 6, and what are the paired confidence intervals on the 547 Expert test pairs? Given the overfitting in Figures 10, 15 and 16, how is the reported outer iteration chosen? Is it chosen on the 100-example checkpoint selection set, on the target set, or by some other rule? This is a clarification question. Regarding metrics beyond loss, please report pairwise accuracy, and if possible RewardBench or best-of-N results. Please also state whether the weights are normalized, and add the control that learns only a single global weight.

Q3 (see W2 and W3): Please explain the three different sets of Spearman values in Table 1, Table 2 and Figure 9, and the small gain at the lowest human to bot ratio in Figure 1(b), which is inconsistent with the statement in Section 4.1. Meta optimization is only run at 0.5B and takes tens of GPU hours, so please clarify what “scales for free” refers to. Since Expert is a subset of Arena, please also confirm that the Expert target and test rows were deduplicated against Human-155k in all experiments, not only in the selection baseline protocol in Appendix D

Q4 (see W3 ): Most of the appendix figures from Figure 10 to Figure 28 are similar segment-wise loss curves with little discussion. I suggest keeping only the curves that support a specific conclusion, explaining in the text what each figure shows, and moving the rest into a summary table with seed statistics. For Figure 1(c) and the bar charts in Figures 1 and 6, please add markers or hatching and use a colorblind friendly palette, so that the different series can be distinguished without relying on color.

1 comment

Thank you for the detailed review. We answer W1–W3 together with the corresponding questions Q1–Q4 below.

Same-target baselines

Q1 (see W1 ): Could the authors add the following baselines? (i) Density ratio weighting with a classifier trained on the same text-embedding-3-small features to separate target data from training data. (ii) Under the same compute, at least one short horizon reweighting method, such as L2RW, Meta-Weight-Net, or the instantaneous metagradient in Figure 31. (iii) Baselines that train or fine-tune directly on the target pairs in the Expert experiments.

(i) We will add this baseline.

(ii) In Figure 31 the one-step metagradient is used for selection, so it is not exactly this ablation. We ran the direct comparison, changing only the segment length: after 4 outer steps (4 passes over the full data), one-step metagradients reach 0.687 test loss vs 0.643 for full-pass ones (0.697 unweighted), and they are less compute-efficient. We will include the figure in the camera-ready version.

(iii) Agreed, we will add this for Expert.

Effect sizes, calibration, stopping, and scaling

Q2 (see W2 ): How many seeds are behind Figures 1 and 6, and what are the paired confidence intervals on the 547 Expert test pairs? Given the overfitting in Figures 10, 15 and 16, how is the reported outer iteration chosen? Is it chosen on the 100-example checkpoint selection set, on the target set, or by some other rule? This is a clarification question. Regarding metrics beyond loss, please report pairwise accuracy, and if possible RewardBench or best-of-N results. Please also state whether the weights are normalized, and add the control that learns only a single global weight.

The gains on Expert with clean data are indeed small and we will add confidence intervals. This is the easiest setting for the unweighted baseline though, and the gaps on STEM and against the selection baselines are well above seed noise. Figure 1 uses a single seed, Figure 5 three seeds, and Figure 6 three sets of learned weights.

The weights are not normalized. Since the model is trained with AdamW, the overall scale of the loss has almost no effect, so a single global weight would behave like the unweighted baseline. The ranking results do not depend on the reward scale either. We will add pairwise accuracy.

Inconsistencies

Q3 (see W2 and W3): Please explain the three different sets of Spearman values in Table 1, Table 2 and Figure 9, and the small gain at the lowest human to bot ratio in Figure 1(b), which is inconsistent with the statement in Section 4.1. Meta optimization is only run at 0.5B and takes tens of GPU hours, so please clarify what “scales for free” refers to. Since Expert is a subset of Arena, please also confirm that the Expert target and test rows were deduplicated against Human-155k in all experiments, not only in the selection baseline protocol in Appendix D

We will fix the sentence in Section 4.1: the gain is indeed only clear at 30k and 45k. The small gain at 10k was an artifact of the number of inner training epochs: in the new run, dataweighting brings the test loss down to 0.622. The three sets of Spearman values come from different setups (one split, seven splits, and transfer to the full 92k pool), which we will clarify in the updated version.

By "for free" we mean that the meta-optimization cost is paid once at 0.5B and the weights are then reused for larger models without any additional metagradient computation.

The Expert target and test sets are disjoint from Human-155k.

Figures

Q4 (see W3 ): Most of the appendix figures from Figure 10 to Figure 28 are similar segment-wise loss curves with little discussion. I suggest keeping only the curves that support a specific conclusion, explaining in the text what each figure shows, and moving the rest into a summary table with seed statistics. For Figure 1(c) and the bar charts in Figures 1 and 6, please add markers or hatching and use a colorblind friendly palette, so that the different series can be distinguished without relying on color.

We will trim the appendix curves and make the figures colorblind-friendly.

Reviewer_4DtpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When working with large, mixed-quality datasets that contain noise and synthetic data, the standard approach is to design filters, choose mixing ratios by hand, or discard low-scoring samples. This paper instead learns data weights using a small, clean target set. It treats the training process as a differentiable function and uses metagradients to update each sample’s weight based on performance on the target set.

For reward modeling, this approach can downweight noisy or adversarial data, adapt to specific domains, and turn synthetic data from harmful to helpful. For ranking, it can also optimize a target objective that differs entirely from the training loss. Overall, the method shows strong results compared with data selection approaches.

Critical strengths
  1. The paper proposes a scalable way to learn data weights. Given a small, high-quality target set, it uses metagradients through the training process to efficiently optimize training sample weights for target-domain performance.

  2. Data weighting works well, outperforming data selection methods on STEM. The paper also includes experiments on synthetic data, parameterized data weights, and transfer to larger-scale settings.

Critical weaknesses
  1. Reproducibility is my main concern. The method involves many datasets, model experiments, and hyperparameters. However, the authors have not released code or provided enough supporting material to make reproduction straightforward. Reproducing the results would likely require substantial effort.

  2. The method replaces manual filter design with the construction of a small, clean target set. This still requires human effort. The Expert set is curated from Arena, while STEM labels come from qualified human raters or ground-truth verifiers. The human work has not disappeared; it has shifted from writing filtering rules to curating and labeling a smaller set of high-quality examples.

Questions for authors
  1. How sensitive is the method to the size of the target set? This analysis is missing. Expert uses 3,000 examples for meta-optimization, and STEM uses 6,796. How well would the method work with only 300 examples, or even fewer?
1 comment

Reproducibility

Reproducibility is my main concern. The method involves many datasets, model experiments, and hyperparameters. However, the authors have not released code or provided enough supporting material to make reproduction straightforward. Reproducing the results would likely require substantial effort.

Almost all the datasets we use are public (Arena human preferences, arena-expert-5k, etc.), and we will add direct links. We will also add the full set of hyperparameters for each experiment in the appendix.

Human effort moves to target curation

The method replaces manual filter design with the construction of a small, clean target set. This still requires human effort. The Expert set is curated from Arena, while STEM labels come from qualified human raters or ground-truth verifiers. The human work has not disappeared; it has shifted from writing filtering rules to curating and labeling a smaller set of high-quality examples.

We did not annotate anything ourselves: both target sets are existing public datasets, and they are small compared to the training pool (a few thousand vs 155k pairs). In additional runs we still see gains with as few as 100 target pairs (see below).

Target size of 300 or fewer

How sensitive is the method to the size of the target set? This analysis is missing. Expert uses 3,000 examples for meta-optimization, and STEM uses 6,796. How well would the method work with only 300 examples, or even fewer?

The "Pure B" curve in Figure 2 answers part of this: the gain is already flat at around 1.7k target pairs. We also ran smaller targets, and we still gains even with 100. We will add more target set sizes to the figure.

Reviewer_Epa2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a data weighting method which learns training data weights by differentiating through model training to improve performance on a small target dataset. It further proposes to learn parametric weighting rules that can be used to generalize to new unseen examples and transfer from smaller models to substantially larger ones.

Critical strengths

S1. This paper works on a practical problem.

S2. Transferring from small to big models is a practical need in data valuation / attribution.

Critical weaknesses

W1. Overall, optimizing data given a target clean set is a well-researched problem, and this work needs to position and compare better with existing work. For instance, learning data weights given a clean validation objective is explored by [1], while DataRater [2] explores parametric weighting and transfer across model sizes. The scalable metagradient computation seems also overlap with prior work [3]. Therefore, the contribution and new technical insight remain unclear to me.

W2. The generalizability of the approach is not sufficiently demonstrated in experiments. Although the method is presented as a general framework for weighting and learning heterogeneous data, all experiments are around reward model training (which is a preference learning setup). Other setups such as instruction tuning and code /math (generation, not preference) would help demonstrate the effectiveness of the work.

W3. Results such as Figures 15 and 16 seem to indicate severe overfitting (more steps monotonically increase test loss).

[1] Ren, Mengye, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. "Learning to reweight examples for robust deep learning." In International conference on machine learning, pp. 4334-4343. PMLR, 2018.

[2] Calian, Dan Andrei, Greg Farquhar, Iurii Kemaev, Luisa Zintgraf, Matteo Hessel, Jeremy Shar, Junhyuk Oh et al. "Datarater: Meta-learned dataset curation." Advances in Neural Information Processing Systems 38 (2026): 20060-20092.

[3] Engstrom, Logan, Andrew Ilyas, Benjamin Chen, Axel Feldmann, William Moses, and Aleksander Madry. "Optimizing ml training with metagradient descent." arXiv preprint arXiv:2503.13751 (2025).

Questions for authors

None

1 comment

Positioning relative to Ren et al., DataRater, and metagradient descent

W1. Overall, optimizing data given a target clean set is a well-researched problem, and this work needs to position and compare better with existing work. For instance, learning data weights given a clean validation objective is explored by [1], while DataRater [2] explores parametric weighting and transfer across model sizes. The scalable metagradient computation seems also overlap with prior work [3]. Therefore, the contribution and new technical insight remain unclear to me.

We agree the related work should make this clearer and will revise it. We build on [3] for the metagradient computation. The main difference with [1] and [2] is the horizon: they differentiate through one or a few training steps, while we differentiate through long training segments, which is a much closer approximation of the metagradient of the full training run. In a new experiment we ran, one-step metagradients barely improve over unweighted training (0.687 vs 0.697 test loss), whereas full-pass metagradients reach 0.643 for the same number of outer steps and with less compute, since the target gradient is computed once per pass instead of at every step.

Generality beyond preference learning

W2. The generalizability of the approach is not sufficiently demonstrated in experiments. Although the method is presented as a general framework for weighting and learning heterogeneous data, all experiments are around reward model training (which is a preference learning setup). Other setups such as instruction tuning and code /math (generation, not preference) would help demonstrate the effectiveness of the work.

We focused on reward modeling on purpose, as it brings together noisy labels, synthetic data, domain shift and a ranking metric in a single setting. Metagradients were already shown to work in other settings in [3], including CLIP pretraining and instruction tuning, among others.

Overfitting in Figures 15–16

W3. Results such as Figures 15 and 16 seem to indicate severe overfitting (more steps monotonically increase test loss).

These two runs were trained for many more outer steps than needed. The test loss is not increasing throughout: it only goes up in the later outer steps, when the weights start overfitting the target set, which early stopping avoids. We will clarify the stopping rule in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.