- Rating
- 2(−1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper learns per-example training weights from a small curated target set by running metagradient descent through segments of the training run. It also proposes a parametric version, a linear map on frozen text embeddings, that can score unseen examples and be transferred to larger models. On reward modeling with Arena human preferences, flagged bot data and Gemini synthetic labels, the authors report lower test loss under noise, poisoning, domain shift and human plus synthetic mixing, better results than TRAK, influence function and TracIn based selection, and better leaderboard ranking when a soft Spearman surrogate is used as the outer objective.
Critical strengths
S1: The objective is well defined and covers several settings. A single bilevel objective handles noise filtering, domain customization, mixing human and synthetic labels, and even a non-differentiable outer objective. The ranking experiment in Section 4.6 is interesting. Soft Spearman performs poorly as the training loss but helps when used only as the outer objective, improving from 0.541 to 0.625 on one split and from a median of 0.880 to 0.957 over seven splits.
S2: The comparison with attribution-based selection methods in Section 4.3 and Appendix D is done carefully. The number of optimizer steps is matched, every keep fraction has a random-drop control, the checkpoint selection set is separate, and four training pools are covered. The response length analysis in Figure 29 is also a valuable finding. When 10% of the data is kept, 80.6% of the pairs selected by TracIn have the shorter response as the chosen one, compared with only 40.3% in the full pool.
S3: The parametric weights are practical. As shown in Figures 5 and 6, with only 4,609 meta parameters they match the per-sample weights, can be applied to examples not seen during meta training, and transfer from 0.5B to larger reward models without recomputing metagradients.
Critical weaknesses
W1: Key baselines that use the same target set are missing, including but not limited to the following (a) A relevant baseline is density ratio weighting, using a classifier trained on the same frozen embeddings to distinguish target data from training data. This baseline is needed to tell how much of the gain actually requires metagradients. (b) The closest reweighting methods, such as Ren et al. (2018), Meta-Weight-Net (Shu et al., 2019) and DataRater (Calian et al., 2025), are discussed but not compared. As a result, the core claim of the paper, that exact metagradients over long training segments are better than short horizon approximations, is not tested. The instantaneous metagradient already shown in Figure 31 could serve as this ablation in the main text (c) In Figure 1(a), 1(b), 1(d) and Figure 4, the unweighted baseline does not use the 3,000 Expert target pairs used by dataweighting. A baseline that trains or fine-tunes directly on the target pairs appears only in the STEM experiment with a Human-33k base (Figure 2).
W2: The improvements are small relative to the noise, and some conclusions seem stronger than what the figures show. For example, on Expert, the gains in Figure 1(a) are 0.009, 0.005 and 0.003, and in Figure 1(b) they are 0.002, 0.002 and 0.000 at 10k, 60k and 100k. Only 30k and 45k show a clear gain. Figures 10, 15 and 16 also mention notable overfitting to the target set, but the paper does not explain how the number of outer iterations is chosen Second, the evaluation relies almost only on Bradley-Terry test loss. The weights are initialized at 1 and moved up or down by a fixed step, and the paper does not mention normalization, so the outer optimization may also change the effective learning rate or the scale of the rewards. This can lower log loss through calibration without improving ranking. In addition, a control that learns only a single global weight with the same metagradient is needed to show how much of the gain comes from per-example weighting. Finally, the scalability claims go beyond the evidence. The TL;DR says the method “scales for free to larger datasets and models”, but meta optimization is only run on the 0.5B model, and Appendix D shows that reaching the Pareto frontier on 40k to 77k examples takes 48.8 to 58.3 H200 GPU hours. The gain also becomes smaller when transferring to 7B. In Figure 6a, the test loss drops from about 0.503 to about 0.500 with per-sample weights and to about 0.491 with parametric weights. The theory in Section 5 studies exhaustive subset selection in a linear model, which the authors acknowledge is not the algorithm actually used
W3: There are several inconsistencies, and some figures are hard to read (a) Section 4.1 says that weighting restores performance in all cases except when the ratio of human to bot data is very high. However, in Figure 1(b) the gain is also almost zero at the lowest ratio, 10k human with 10k bot data, from 0.634 to 0.632. (b) The unweighted Spearman value on the single split in Table 1 is 0.541, well below the 25th percentile (0.717) of the seven splits in Table 2, and Figure 9 reports 0.65 on 92k training examples. The corresponding meta-optimized values are 0.625, 0.957 and 0.93. The paper does not explain how these setups relate to each other (c) Most of the appendix figures from Figure 10 to Figure 28 are segment-wise loss curves that look very similar, with little accompanying analysis, which makes them hard to read. Figure 1(c) and these curves distinguish outer iterations only by shades of blue, and the bar charts in Figures 1 and 6 also rely only on color. I suggest adding markers or hatching and using a colorblind friendly palette, or replacing many of the curves with tables for the key results, which would be clearer
Questions for authors
Q1 (see W1 ): Could the authors add the following baselines? (i) Density ratio weighting with a classifier trained on the same text-embedding-3-small features to separate target data from training data. (ii) Under the same compute, at least one short horizon reweighting method, such as L2RW, Meta-Weight-Net, or the instantaneous metagradient in Figure 31. (iii) Baselines that train or fine-tune directly on the target pairs in the Expert experiments.
Q2 (see W2 ): How many seeds are behind Figures 1 and 6, and what are the paired confidence intervals on the 547 Expert test pairs? Given the overfitting in Figures 10, 15 and 16, how is the reported outer iteration chosen? Is it chosen on the 100-example checkpoint selection set, on the target set, or by some other rule? This is a clarification question. Regarding metrics beyond loss, please report pairwise accuracy, and if possible RewardBench or best-of-N results. Please also state whether the weights are normalized, and add the control that learns only a single global weight.
Q3 (see W2 and W3): Please explain the three different sets of Spearman values in Table 1, Table 2 and Figure 9, and the small gain at the lowest human to bot ratio in Figure 1(b), which is inconsistent with the statement in Section 4.1. Meta optimization is only run at 0.5B and takes tens of GPU hours, so please clarify what “scales for free” refers to. Since Expert is a subset of Arena, please also confirm that the Expert target and test rows were deduplicated against Human-155k in all experiments, not only in the selection baseline protocol in Appendix D
Q4 (see W3 ): Most of the appendix figures from Figure 10 to Figure 28 are similar segment-wise loss curves with little discussion. I suggest keeping only the curves that support a specific conclusion, explaining in the text what each figure shows, and moving the rest into a summary table with seed statistics. For Figure 1(c) and the bar charts in Figures 1 and 6, please add markers or hatching and use a colorblind friendly palette, so that the different series can be distinguished without relying on color.
Thank you for the detailed review. We answer W1–W3 together with the corresponding questions Q1–Q4 below.
Same-target baselines
(i) We will add this baseline.
(ii) In Figure 31 the one-step metagradient is used for selection, so it is not exactly this ablation. We ran the direct comparison, changing only the segment length: after 4 outer steps (4 passes over the full data), one-step metagradients reach 0.687 test loss vs 0.643 for full-pass ones (0.697 unweighted), and they are less compute-efficient. We will include the figure in the camera-ready version.
(iii) Agreed, we will add this for Expert.
Effect sizes, calibration, stopping, and scaling
The gains on Expert with clean data are indeed small and we will add confidence intervals. This is the easiest setting for the unweighted baseline though, and the gaps on STEM and against the selection baselines are well above seed noise. Figure 1 uses a single seed, Figure 5 three seeds, and Figure 6 three sets of learned weights.
The weights are not normalized. Since the model is trained with AdamW, the overall scale of the loss has almost no effect, so a single global weight would behave like the unweighted baseline. The ranking results do not depend on the reward scale either. We will add pairwise accuracy.
Inconsistencies
We will fix the sentence in Section 4.1: the gain is indeed only clear at 30k and 45k. The small gain at 10k was an artifact of the number of inner training epochs: in the new run, dataweighting brings the test loss down to 0.622. The three sets of Spearman values come from different setups (one split, seven splits, and transfer to the full 92k pool), which we will clarify in the updated version.
By "for free" we mean that the meta-optimization cost is paid once at 0.5B and the weights are then reused for larger models without any additional metagradient computation.
The Expert target and test sets are disjoint from Human-155k.
Figures
We will trim the appendix curves and make the figures colorblind-friendly.