- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses the poor generalization of selection meta-networks in Meta-learning for Training-data Selection (MTS) for LLMs. The authors theoretically and empirically identify two flaws in existing bilevel objectives (like ScaleBiO): competitive weight suppression (driving most sample weights toward zero) and persistent reliance on easy-to-learn shortcut features. To resolve this, they propose TESS (Transferable Example Scoring and Selection), which sequentially trains a training-only model and a model further trained with validation data, and scores each example by the loss difference . Experiments on LLM safety and targeted instruction tuning demonstrate strong transfer across datasets, corpus sizes, and model scales.
Critical strengths
- Clear Theoretical Insight: Theorems 1 and 2 convincingly explain why normalized bilevel objectives collapse into winner-takes-most allocations and amplify shortcut learning, supported by clean synthetic simulations.
- Simple and Scalable Design: Decoupling bilevel optimization into sequential training with a pointwise MSE loss stabilizes training, cuts peak GPU memory by ~20%, and achieves a 3.46x speedup over ScaleBiO on Tulu V2.
- Strong Generalization: TESS is the first MTS framework to successfully transfer across disjoint datasets (Alpaca Dolly), from 50K subsets to 197K corpora, and from 0.5B to 7B models.
Critical weaknesses
- Static Approximation: Computing from uniformly weighted models ignores how dynamic reweighting alters optimization trajectories. Under extreme noise, and may degrade before pseudo-labels are extracted.
- Missing Main-Text Details: Key details regarding the meta-network architecture and sensitivity to the hyperparameter are deferred to the appendix. Minor typos exist (e.g., "defalt" on line 416).
Questions for authors
- How sensitive are the pseudo-labels to the training duration of and (e.g., if memorizes the training set)?
- How does PVM conceptually relate to reducible loss methods like RHO-Loss?
4 comments
Response to Reviewer HV1w
We thank reviewer HV1w for the diligent and thoughtful feedback. Below we provide our responses.
W1+Q1: How sensitive are the pseudo-labels to the training duration of U and W?
A1: The pseudo-labels remain effective during the most part of the training. However, during the very first stage and when losses have decreased to near zero values, pseudo-labels become less effective. Further, the data selection network has the ability to correct noisy labels. Even if the network is trained on noisy pseudo-labels, it can have higher accuracy than that of the pseudo-labels.
Using the same controlled experiment in Section 4.2, we use a 2:1 mixture of GSM8K and unsafe examples. We have ground-truth labels since we know which examples are unsafe. At each training checkpoint of U and W, we evaluate the pseudo-labels and train a selector on the corresponding pseudo-label set. We report selector accuracy on the training data and on the unseen test data containing unseen GSM8K examples and unsafe examples at the same 2:1 ratio.
| Steps | 10 | 20 | 40 | 60 | 80 | 150 | 200 | 250 | 300 | 350 | 400 | 450 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pseudo-label Acc | 5.24 | 29.51 | 77.33 | 89.51 | 90.40 | 90.72 | 91.73 | 90.22 | 86.40 | 81.78 | 76.71 | 70.84 |
| LLM U Training Loss Mean (×10⁻²) | 99.41 | 86.24 | 78.63 | 75.06 | 72.54 | 47.38 | 23.21 | 15.08 | 8.79 | 4.77 | 2.62 | 1.34 |
| LLM W Training Loss Mean (×10⁻²) | 141.08 | 120.71 | 103.76 | 95.52 | 89.30 | 67.44 | 46.71 | 40.17 | 34.24 | 28.34 | 23.13 | 17.35 |
| Selector Training Acc | 2.31 | 31.82 | 94.49 | 97.78 | 98.13 | 98.04 | 98.67 | 98.58 | 98.22 | 98.04 | 97.69 | 96.98 |
| Selector Generalization Acc | 2.31 | 32.98 | 92.98 | 97.24 | 97.51 | 97.60 | 98.13 | 98.04 | 97.60 | 97.33 | 97.07 | 96.09 |
The results show that pseudo-label accuracy rapidly improves during early training, reaching over 90% from steps 80 to 250, before gradually declining as training losses approach zero. Importantly, our method demonstrates strong robustness to noisy pseudo-labels. Even when pseudo-label accuracy drops to 70.84% at step 450, our learned selector still achieves 96.98% training accuracy and 96.09% generalization accuracy. These results highlight that our method is highly robust to the training duration of U and W.
W2: Missing Main-Text Details: Key details regarding the meta-network architecture and sensitivity to the hyperparameter are deferred to the appendix. Minor typos exist (e.g., "defalt" on line 416).
A2: Thank you for pointing this out. We have carefully checked the manuscript and corrected the typos. We added a short description of the meta-network to the main text (Line 405--407).
Q2: How does PVM conceptually relate to reducible loss methods like RHO-Loss?
A3: Reducible Holdout (RHO) Loss uses per-sample loss difference between the current model and a model trained on a holdout dataset, to estimate the potential value of training examples. If a data point has a larger loss difference, it has high current loss but low loss on the holdout-trained model. This tells us that there is still much room for the model to learn from this data point, so it can be prioritized.
Our formulation is similar to the RHO idea but considers distribution differences between the training and validation (i.e., holdout) datasets. We take the loss differences from two models, trained on the training set only, and trained on the training set and the validation set. When a data point has higher loss on than , this indicates that training on validation data improves generalization to this data point. Hence, it is more similar to the validation set rather than the training set. Therefore, we prioritize such data points in training.
There are a few differences between RHO and TESS. First, RHO uses an in-distribution holdout set but TESS uses an out-of-distribution validation set. Further, RHO-Loss directly computes reducible loss for online sample selection, whereas PVM uses these loss differences to train a selection network, enabling transferable example scoring on unseen data. Finally, our loss difference definition can also be derived from the bilevel optimization problem (See A1 to Reviewer hc2v and Appendix A).
Reviewer: your initial assessment leans positive, but please do still respond to the authors, as they have put in a lot of effort writing to you.
Thank you for the response and the additionally experiments. My main concerns are resolved.
Authors: please respond to this reviewer.