acceptodds
Under review as a conference paper at ICLR 2027

Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic)

Abstract

Data selection is critical for training large language models on massive and heterogeneous corpora. Meta-learning for Training-data Selection offers a principled alternative to heuristic scoring by learning data weights from a target validation objective, but existing methods face a trade-off between fine-grained valuation and transferability to unseen data. A natural solution is to replace per-sample weights with a selection network. However, we find that directly incorporating such a network into existing MTS objectives leads to unstable optimization and poor generalization, caused by weight suppression and persistent reliance on easy-to-learn features. To address these issues, we propose \longname (\shortname), a scalable data-selection framework built on a Pointwise Value Matching objective (PVM). Experiments on LLM safety and targeted instruction tuning demonstrate strong transfer across datasets, from subsets to full corpora, and from smaller to larger models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Data Selection, Meta Learning, Generalization, Large Language Models, Bilevel Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HV1wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the poor generalization of selection meta-networks in Meta-learning for Training-data Selection (MTS) for LLMs. The authors theoretically and empirically identify two flaws in existing bilevel objectives (like ScaleBiO): competitive weight suppression (driving most sample weights toward zero) and persistent reliance on easy-to-learn shortcut features. To resolve this, they propose TESS (Transferable Example Scoring and Selection), which sequentially trains a training-only model θU∗\theta_U^{\ast} and a model θW∗\theta_W^{\ast} further trained with validation data, and scores each example ii by the loss difference Δi∗=ℓi(θU∗)−ℓi(θW∗)\Delta_i^{\ast} = \ell_i(\theta_U^{\ast}) - \ell_i(\theta_W^{\ast}). Experiments on LLM safety and targeted instruction tuning demonstrate strong transfer across datasets, corpus sizes, and model scales.

Critical strengths
  • Clear Theoretical Insight: Theorems 1 and 2 convincingly explain why normalized bilevel objectives collapse into winner-takes-most allocations and amplify shortcut learning, supported by clean synthetic simulations.
  • Simple and Scalable Design: Decoupling bilevel optimization into sequential training with a pointwise MSE loss stabilizes training, cuts peak GPU memory by ~20%, and achieves a 3.46x speedup over ScaleBiO on Tulu V2.
  • Strong Generalization: TESS is the first MTS framework to successfully transfer across disjoint datasets (Alpaca ↔\leftrightarrow Dolly), from 50K subsets to 197K corpora, and from 0.5B to 7B models.
Critical weaknesses
  • Static Approximation: Computing Δi∗\Delta_i^{\ast} from uniformly weighted models ignores how dynamic reweighting alters optimization trajectories. Under extreme noise, θU∗\theta_U^{\ast} and θW∗\theta_W^{\ast} may degrade before pseudo-labels are extracted.
  • Missing Main-Text Details: Key details regarding the meta-network architecture sϕs_\phi and sensitivity to the hyperparameter α\alpha are deferred to the appendix. Minor typos exist (e.g., "defalt" on line 416).
Questions for authors
  1. How sensitive are the pseudo-labels Δi∗\Delta_i^{\ast} to the training duration of θU∗\theta_U^{\ast} and θW∗\theta_W^{\ast} (e.g., if θU∗\theta_U^{\ast} memorizes the training set)?
  2. How does PVM conceptually relate to reducible loss methods like RHO-Loss?
4 comments

Authors: please respond to this reviewer.

AuthorsOpenReview ↗

Response to Reviewer HV1w

We thank reviewer HV1w for the diligent and thoughtful feedback. Below we provide our responses.


W1+Q1: How sensitive are the pseudo-labels to the training duration of U and W?

A1: The pseudo-labels remain effective during the most part of the training. However, during the very first stage and when losses have decreased to near zero values, pseudo-labels become less effective. Further, the data selection network has the ability to correct noisy labels. Even if the network is trained on noisy pseudo-labels, it can have higher accuracy than that of the pseudo-labels.

Using the same controlled experiment in Section 4.2, we use a 2:1 mixture of GSM8K and unsafe examples. We have ground-truth labels since we know which examples are unsafe. At each training checkpoint of U and W, we evaluate the pseudo-labels and train a selector on the corresponding pseudo-label set. We report selector accuracy on the training data and on the unseen test data containing unseen GSM8K examples and unsafe examples at the same 2:1 ratio.

Steps1020406080150200250300350400450
Pseudo-label Acc5.2429.5177.3389.5190.4090.7291.7390.2286.4081.7876.7170.84
LLM U Training Loss Mean (×10⁻²)99.4186.2478.6375.0672.5447.3823.2115.088.794.772.621.34
LLM W Training Loss Mean (×10⁻²)141.08120.71103.7695.5289.3067.4446.7140.1734.2428.3423.1317.35
Selector Training Acc2.3131.8294.4997.7898.1398.0498.6798.5898.2298.0497.6996.98
Selector Generalization Acc2.3132.9892.9897.2497.5197.6098.1398.0497.6097.3397.0796.09

The results show that pseudo-label accuracy rapidly improves during early training, reaching over 90% from steps 80 to 250, before gradually declining as training losses approach zero. Importantly, our method demonstrates strong robustness to noisy pseudo-labels. Even when pseudo-label accuracy drops to 70.84% at step 450, our learned selector still achieves 96.98% training accuracy and 96.09% generalization accuracy. These results highlight that our method is highly robust to the training duration of U and W.


W2: Missing Main-Text Details: Key details regarding the meta-network architecture and sensitivity to the hyperparameter α\alpha are deferred to the appendix. Minor typos exist (e.g., "defalt" on line 416).

A2: Thank you for pointing this out. We have carefully checked the manuscript and corrected the typos. We added a short description of the meta-network to the main text (Line 405--407).


Q2: How does PVM conceptually relate to reducible loss methods like RHO-Loss?

A3: Reducible Holdout (RHO) Loss uses per-sample loss difference between the current model and a model trained on a holdout dataset, to estimate the potential value of training examples. If a data point has a larger loss difference, it has high current loss but low loss on the holdout-trained model. This tells us that there is still much room for the model to learn from this data point, so it can be prioritized.

Our formulation is similar to the RHO idea but considers distribution differences between the training and validation (i.e., holdout) datasets. We take the loss differences from two models, θU\theta_U trained on the training set only, and θW\theta_W trained on the training set and the validation set. When a data point has higher loss on θU\theta_U than θW\theta_W, this indicates that training on validation data improves generalization to this data point. Hence, it is more similar to the validation set rather than the training set. Therefore, we prioritize such data points in training.

There are a few differences between RHO and TESS. First, RHO uses an in-distribution holdout set but TESS uses an out-of-distribution validation set. Further, RHO-Loss directly computes reducible loss for online sample selection, whereas PVM uses these loss differences to train a selection network, enabling transferable example scoring on unseen data. Finally, our loss difference definition can also be derived from the bilevel optimization problem (See A1 to Reviewer hc2v and Appendix A).

Reviewer: your initial assessment leans positive, but please do still respond to the authors, as they have put in a lot of effort writing to you.

Thank you for the response and the additionally experiments. My main concerns are resolved.

Reviewer_hc2vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Transferable Example Scoring and Selection (TESS), a framework for learning a data-selection network meant to generalize to unseen examples. The authors use per-example loss differences between two differently trained language models as pseudo-labels, then train a scoring network using a pointwise regression objective. Experiments cover harmful-data identification and targeted instruction tuning, including cross-dataset and cross-model transfer. Overall, I think the paper is well-written, however, its proposed method is generally simplistic and lacks novelty and the experiments do not provide sufficient uncertainty quantification, calling into question is methodological depth.

Critical strengths
  • The empirical improvements are promising, especially in the cross-dataset transfer experiments.
  • The comparisons between the original ScaleBiO objective and the proposed regression loss provide useful evidence of optimization difficulties.
Critical weaknesses

1. Insufficient justification of the pseudo-labels.

The proposed pseudo-labels measure differences in per-example losses between two language models, but this is not equivalent to measuring the marginal utility of an example for improving validation performance as there is no guarantee that they reflect an intrinsic property of the data itself rather than being an artefact of some other model aspect. Such differences can arise from calibration, model quality, or other changes unrelated to the contribution of the example to training.

2. Limited methodological novelty.

Once the pseudo-labels are constructed, the main algorithmic contribution amounts to classical supervised regression using a mean-squared-error objective, thus providing little novelty and serving predominantly as an application paper. The paper does not sufficiently distinguish its contribution from existing influence estimation and data valuation methods, or establish why the proposed regression objective should be preferred over simpler alternatives. Generally, I believe that the target venue prioritises novelty and insight, and the authors may wish to re-submit to a more applications-driven venue.

3. Efficiency and generalization claims are insufficiently supported.

The method still requires generating pseudo-labels before training the scoring network. These cost of generating pseudo-labels still exists in the offline pre-training stage. My faith in the generalisability of the method is also limited due to a lack of uncertainty quantification for single-run results, ideally the authors would run sufficiently many seeds to statistically separate their method from the competing baselines.

4. Theoretical analysis is narrow.

I am perfectly aware that theoretical analysis often serves to inform or motivate the work and does not have to apply to precisely the setting tested empirically. However, for this work Theorem 1 is explicitly attributed to prior work, while Theorem 2 analyzes a simplified one-layer network with two features. This is a minor concern and I do not expect authors to do anything about it.

Questions for authors

1.Under what assumptions do the proposed loss-difference pseudo-labels approximate the actual utility of training examples?

2.What is the methodological contribution, novel aspect, or insight, beyond training a supervised regressor on model-generated utility estimates?

3.Does the computational cost comparison include both language-model training runs and pseudo-label generation?

4.How does the method perform under more substantial distribution shifts and across independent training seeds?

3 comments
AuthorsOpenReview ↗

Response to Reviewer hc2v (Part 1): Questions 1–2

We thank reviewer hc2v for the diligent and thoughtful feedback. Below we provide our responses.


Q1: Under what assumptions do the proposed loss-difference pseudo-labels approximate the actual utility of training examples?

A1: The pseudo-labels can be justified from two perspectives. First, it can be directly derived mathematically. Due to space constraints, we did not include a full derivation but show it here (and in Appendix A).

The original bi-level optimization problem is the following. In this problem, we learn the wi,∀iw_i, \forall i that stand for data utility. \boldsymbol{w}^{\ast} = \arg\min_{\boldsymbol w} \mathcal{L}\bigl(\boldsymbol\theta^{\ast}; \mathcal{D}_{\mathrm{val}}\bigr) = \arg\min_{\boldsymbol w} N_{\mathrm{val}}^{-1} \sum\nolimits_{i=1}^{N_{\mathrm{val}}} \ell_i(\boldsymbol\theta^{\ast}), \tag{1}

\text{s.t.}\quad \boldsymbol\theta^{\ast} = \arg\min_{\boldsymbol\theta} \mathcal{L}\bigl(\boldsymbol\theta,\boldsymbol w; \mathcal{D}_{\mathrm{train}}\bigr) = \arg\min_{\boldsymbol\theta} \lVert\boldsymbol w\rVert_1^{-1} \sum\nolimits_{i=1}^{N_{\mathrm{train}}} w_i \ell_i(\boldsymbol\theta). \tag{2}

Creating two copies of the model parameters θW\boldsymbol\theta_W and θU\boldsymbol\theta_U, and adding a constraint on θW=θU\boldsymbol\theta_W=\boldsymbol\theta_U, we derive

\boldsymbol{w}^{\ast} = \arg\min_{\boldsymbol w} \mathcal{L}\bigl(\boldsymbol\theta_W; \mathcal{D}_{\mathrm{val}}\bigr), \tag{3}

\text{s.t.}\quad \boldsymbol\theta_U = \arg\min_{\boldsymbol\theta} \mathcal{L}\bigl(\boldsymbol\theta,\boldsymbol w; \mathcal{D}_{\mathrm{train}}\bigr), \tag{4}

\text{and}\quad \boldsymbol\theta_W = \boldsymbol\theta_U. \tag{5}

We further relax θW=θU\boldsymbol\theta_W = \boldsymbol\theta_U into its necessary condition that they have equal losses, or L(θW,w;Dtrain)−L(θU,w;Dtrain)=0\mathcal{L}\bigl(\boldsymbol\theta_W,\boldsymbol w;\mathcal{D}_{\mathrm{train}}\bigr) - \mathcal{L}\bigl(\boldsymbol\theta_U,\boldsymbol w;\mathcal{D}_{\mathrm{train}}\bigr) = 0. Building the Lagrangian function with the penalty coefficient α>0\alpha >0, we derive a new minimax problem:

\min_{\boldsymbol w,\boldsymbol\theta_W} \max_{\boldsymbol\theta_U}\quad \mathcal L(\boldsymbol\theta_W;\mathcal D_{\mathrm{val}}) + \alpha \Big[ \mathcal L(\boldsymbol\theta_W,\boldsymbol w;\mathcal D_{\mathrm{train}}) - \mathcal L(\boldsymbol\theta_U,\boldsymbol w;\mathcal D_{\mathrm{train}}) \Big]. \tag{6}

If we consider the first and last problems as equivalent, then we are indeed learning data utility from Eq. (6).

Another way to justify the pseudo-labels is to relate them to the Reducible Holdout (RHO) Loss (Mindermann et al., Prioritized Training on Points that Are Learnable, Worth Learning, and Not Yet Learnt. 2022 ICML.), which uses per-sample loss difference between the current model and a model trained on a holdout dataset, to estimate the potential value of training examples. If a data point has a larger loss difference, it has high current loss but low loss on the holdout-trained model. This tells us that there is still much room for the model to learn from this data point, so it can be prioritized.

Our formulation is similar to the RHO idea but considers distribution differences between the training and validation (i.e., holdout) datasets. We take the loss differences from two models, θU\theta_U trained on the training set only, and θW\theta_W trained on the training set and the validation set. When a data point has higher loss on θU\theta_U than θW\theta_W, this indicates that training on validation data improves generalization to this data point. Hence, it is more similar to the validation set rather than the training set. Therefore, we prioritize such data points in training.


Q2: What is the methodological contribution, novel aspect, or insight, beyond training a supervised regressor on model-generated utility estimates?

A2: Our main contribution is a theory-driven application of known techniques to solve problems that nobody knew how to do. The techniques by themselves may look simple, but knowing that they can solve the problem is by no means simple. Some of the most influential ideas in deep learning, such as SGD, dropout, batch / layer / RMS normalization, residual connections, etc., are extremely simple. Simple but effective ideas are easy to adopt and influential; simplicity is an advantage rather than a disadvantage.

Learning to select training data is an important and actively studied problem in LLM data selection, yet achieving both scalability and transferability remains challenging despite prior efforts. The key insight of the paper is to show why TESS works through theoretical analysis and controlled experiments: First, the old loss causes the discrimination between good and bad data points to disappear as we train more. Second, the old loss favors only easy features whereas the new loss learns to use both easy and hard features.

AuthorsOpenReview ↗

Response to Reviewer hc2v (Part 2): Questions 3–4

Q3: Does the computational cost comparison include both language-model training runs and pseudo-label generation?

A3: We realized that we inadvertently omitted the 5.2 hours needed for the extraction of input features of the selector. The originally reported 13.4 hours included both LLM training runs, pseudo-label generation, MLP selector training, and pool scoring. On the 197K-sample dataset, TESS takes 18.6 hours in total, compared with the reported 59.8 hours for SBO$_w$ and 39.6 hours for SEAL$_w$, or 3.22× and 2.13× of TESS. We revised the manuscript (Line522--525).


Q4: How does the method perform under more substantial distribution shifts and across independent training seeds?

A4: Different seeds. With three seeds, we rerun the data selection experiment on Dolly only and the transfer experiment from Dolly to Alpaca on Qwen2.5-7B-Instruct. Each benchmark reports ASR (%) as mean ± sample standard deviation. The Avg. column reports the averages of means and standard deviations. As shown in the tables, TESS exceeds Random by 45.54% in Training and 5.25% in Generalization. In Generalization, TESS also exceeds SEAL and SBO by 15.63% and 15.18%, respectively. These results support the effectiveness and reproducibility of TESS across independent runs.

MethodDirectHarm4HarmBenchHEx-PHIAvg.
Setting: Training (Dolly)
Random50.5036.0032.4139.64
SEAL$_\phi$62.25 ± 3.0442.83 ± 2.2544.14 ± 7.2249.74 ± 4.17
SBO$_\phi$59.42 ± 4.0441.50 ± 3.6146.09 ± 2.9649.00 ± 3.54
TESS83.83 ± 4.7586.17 ± 3.1885.54 ± 6.1485.18 ± 4.69
Setting: Generalization (Dolly → Alpaca)
Random27.5011.0013.1017.20
SEAL$_\phi$10.50 ± 2.563.17 ± 0.766.78 ± 1.406.82 ± 1.57
SBO$_\phi$11.58 ± 3.523.00 ± 0.877.24 ± 1.257.27 ± 1.88
TESS36.08 ± 5.5114.83 ± 2.9316.44 ± 3.1122.45 ± 3.85

Different distribution shifts. To test whether TESS still identifies unsafe data under larger shifts, we introduce three increasingly more difficult scenarios. The training set is a 2:1 mixture of GSM8K and unsafe examples. The validation set contains 100 safe examples. There are different three test sets for the network to select from: (1) IID: unseen examples from GSM8K, (2) near-domain: examples from a different math dataset (AQuA), and (3) far-OOD-domain: examples from a completely different task, SAMSum dialogue summarization. We compare against Random, which randomly selects examples and achieves an expected accuracy of 33.33% given the 2:1 safe-to-unsafe ratio; Emb and Grad, which rank examples by their mean validation distance in representation and gradient space, respectively, and SBO$_\phi$, which trains a selector using the SBO objective. As shown in the table, TESS accuracy decreases as the distribution changes, but under the largest shift (SAMSum), it still outperforms the learnable selector and the strongest baseline (Grad) by 35.73% and 12.53%, respectively.

MethodSeen GSM8KUnseen GSM8KAQuASAMSumUnseen avg.
Emb56.0955.2945.6946.7549.24
Grad84.1884.8978.1373.4278.81
SBO$_\phi$61.5159.6452.9850.2254.28
TESS98.5897.9692.8085.9592.24

Thanks for the rebuttal and additional experiments, although the late response left limited opportunity for constructive discussion. While your clarifications are useful I do not believe they bring the work to the significance bar required for a major conference. I thus choose to maintain my original score.

Reviewer_grpEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AI was used to rephrase/polish wording. The reviewer wrote the original assessment from scratch and reviewed the final wording.

Original reviewer-written assessment supplied as input:

Flag For Ethics Review

No.

The harmful-data-selection experiments are framed as safety diagnostics. The reported evaluation by itself gives no specific reason to request a separate ethics review.

Critical Weaknesses

  1. The dynamics explained by the theory are narrower than the main presentation implies. Appendix B analyzes weights that are optimized independently and have no upper bound, whereas the implemented selector is a shared MLP with a sigmoid output, whose updates couple examples and whose outputs are bounded. Consequently, the concentration result does not directly establish the claimed selector dynamics. Appendix C likewise idealizes every activation derivative as the same positive constant, and derives the PVM recurrence under that comparison condition together with a step-size restriction. These are useful analytical models, but the conditions should be stated alongside Theorem 2 in the main text. Furthermore, defining an “easy” feature as one with larger covariance with the target conflates how predictive a feature is with how hard it is to extract, and a shrinking gradient gap does not on its own show that a transferable rather than spurious feature is being learned. To me, the controlled experiments are more convincing than the general causal explanation currently attached to them.

  2. Supporting the scalability claim requires the complete cost of scoring an unseen pool, as well as a comparison with the information already contained in the selector's inputs. According to Appendix D.3, features are built from representation and per-sample gradient embeddings, distances to the three nearest validation examples and their mean, loss, and gradient norm, with a keyword feature added for safety selection. Applying the small MLP to unseen data thus still entails model-derived feature extraction, which may be expensive, as well as comparisons with validation data. While the memory saving from sequential training is genuine, it does not in itself demonstrate cheap deployment at corpus scale. A scorer based on validation distance or simple features that uses these same inputs would also help separate the contribution of loss-gap regression from the relevance information supplied to the network. This is important because the instruction-tuning gains over the strongest heuristics are smaller than the large margins over the SBO/SEAL variants.

Questions For Authors

  1. Could Theorems 1 and 2 be restated with their conditions on independent weights, activation derivatives, and step size made explicit? Beyond the similarity of aggregate training curves, what evidence links those idealizations to the actual bounded shared MLP?
  2. In the 197K-example selection experiment, what does each stage cost separately: training the two LLMs, extracting representation and gradient features, computing validation distances, training the MLP, and scoring the pool? Which of these stages are included in the reported 13.4 hours?
  3. How does PVM compare with a simple predictor or ranking rule built on the same ten-dimensional features? Please also explain how signed loss-gap targets are represented or transformed given that the selector output is restricted to (0,1), and whether this affects the ranking of low-score harmful data.

Rating

2: Weak rejection.

I place this paper slightly below the acceptance threshold. PVM is simple, the controlled comparison of objectives is valuable, and the transfer results look promising. Nevertheless, the paper's central theoretical explanation is stated more broadly than its assumptions warrant, and the scalability evidence does not clearly distinguish selector training from the feature extraction required on unseen data. My assessment would improve materially if the theory were explicitly narrowed and a full-pipeline advantage over same-input baselines were established.

Soundness

2: I have minor or fixable concerns about the paper’s soundness.

Paper Summary

TESS is a data-selection framework in which Pointwise Value Matching takes the place of a normalized, ScaleBiO-style selector objective. Two language models are trained one after the other: the first on the training data, and the second with additional guidance from a target validation set. The difference in their per-example losses serves as pseudo-labels for a shared selector trained with squared error. The authors analyze competitive weight concentration and the preference for easy features, and then evaluate harmful-data identification, transfer between Alpaca and Dolly, transfer from small to large models, and selection from a larger instruction corpus. In targeted instruction tuning, TESS reaches an overall average of 26.46, compared with 23.91 for LESS and 24.51 for RDS+. These results support a practical regression-based selector, but both the scope of the theoretical explanation and the full cost of scoring unseen examples need clarification.

Presentation

2: Generally understandable, but requires substantial effort in places.

Critical Strengths

  1. Beyond the change to sequential training, the loss ablation offers useful evidence in favor of PVM. Table 3 contrasts the PVM, SBO, and KL objectives within the TESS framework while holding the broad training construction fixed. Relative to SBO on Qwen2.5-7B, PVM raises mean harmful-data-selection ASR from 37.56 to 59.16 in the training setting and from 28.89 to 53.51 under cross-dataset generalization. The comparison against directly ranking the pseudo-labels further suggests that learning a shared mapping can achieve more than reproducing their raw ordering.

  2. Several distinct transfer settings are evaluated, together with a useful positive downstream task. Alpaca–Dolly transfer, selection from a subset to the full corpus, and transfer from 0.5B to 7B each probe a different aspect of reusability. Targeted instruction tuning matters especially, since a higher attack success rate after harmful-data selection does not in itself demonstrate better selection of benign training data. That complementary evidence comes from the GSM8K and CodeX results, which show gains over both shared-network baselines and validation-relevance heuristics.

Informativeness

2: I learned something interesting or useful.

Critical strengths

TESS is a data-selection framework in which Pointwise Value Matching takes the place of a normalized, ScaleBiO-style selector objective. Two language models are trained one after the other: the first on the training data, and the second with additional guidance from a target validation set. The difference in their per-example losses serves as pseudo-labels for a shared selector trained with squared error. The authors analyze competitive weight concentration and the preference for easy features, and then evaluate harmful-data identification, transfer between Alpaca and Dolly, transfer from small to large models, and selection from a larger instruction corpus. In targeted instruction tuning, TESS reaches an overall average of 26.46, compared with 23.91 for LESS and 24.51 for RDS+. These results support a practical regression-based selector, but both the scope of the theoretical explanation and the full cost of scoring unseen examples need clarification.

Critical weaknesses
  1. Beyond the change to sequential training, the loss ablation offers useful evidence in favor of PVM. Table 3 contrasts the PVM, SBO, and KL objectives within the TESS framework while holding the broad training construction fixed. Relative to SBO on Qwen2.5-7B, PVM raises mean harmful-data-selection ASR from 37.56 to 59.16 in the training setting and from 28.89 to 53.51 under cross-dataset generalization. The comparison against directly ranking the pseudo-labels further suggests that learning a shared mapping can achieve more than reproducing their raw ordering.

  2. Several distinct transfer settings are evaluated, together with a useful positive downstream task. Alpaca–Dolly transfer, selection from a subset to the full corpus, and transfer from 0.5B to 7B each probe a different aspect of reusability. Targeted instruction tuning matters especially, since a higher attack success rate after harmful-data selection does not in itself demonstrate better selection of benign training data. That complementary evidence comes from the GSM8K and CodeX results, which show gains over both shared-network baselines and validation-relevance heuristics.

Questions for authors
  1. The dynamics explained by the theory are narrower than the main presentation implies. Appendix B analyzes weights that are optimized independently and have no upper bound, whereas the implemented selector is a shared MLP with a sigmoid output, whose updates couple examples and whose outputs are bounded. Consequently, the concentration result does not directly establish the claimed selector dynamics. Appendix C likewise idealizes every activation derivative as the same positive constant, and derives the PVM recurrence under that comparison condition together with a step-size restriction. These are useful analytical models, but the conditions should be stated alongside Theorem 2 in the main text. Furthermore, defining an “easy” feature as one with larger covariance with the target conflates how predictive a feature is with how hard it is to extract, and a shrinking gradient gap does not on its own show that a transferable rather than spurious feature is being learned. To me, the controlled experiments are more convincing than the general causal explanation currently attached to them.

  2. Supporting the scalability claim requires the complete cost of scoring an unseen pool, as well as a comparison with the information already contained in the selector's inputs. According to Appendix D.3, features are built from representation and per-sample gradient embeddings, distances to the three nearest validation examples and their mean, loss, and gradient norm, with a keyword feature added for safety selection. Applying the small MLP to unseen data thus still entails model-derived feature extraction, which may be expensive, as well as comparisons with validation data. While the memory saving from sequential training is genuine, it does not in itself demonstrate cheap deployment at corpus scale. A scorer based on validation distance or simple features that uses these same inputs would also help separate the contribution of loss-gap regression from the relevance information supplied to the network. This is important because the instruction-tuning gains over the strongest heuristics are smaller than the large margins over the SBO/SEAL variants.

3 comments
AuthorsOpenReview ↗

Response to Reviewer grpE (Part 1): Weakness 1 and Question 1

We thank reviewer grpE for the diligent and thoughtful feedback. Below we provide our responses.


Q1/W1: Could Theorems 1 and 2 be restated with their conditions on independent weights, activation derivatives, and step size made explicit? Beyond the similarity of aggregate training curves, what evidence links those idealizations to the actual bounded shared MLP?

A1: Thank you for pointing out the issues with the theorems. For Theorem 1, it is straightforward to extend to the case with the sigmoid activation function, which we added in the new version. At a high level, the sigmoid is monotonically increasing, so it changes neither the direction of dw/dt nor the existing conclusion at infinite time.

The original Theorem 2 is indeed problematic; we knew but did not have time to fix it. We removed the constant derivative assumption, which is wrong. The theorem now correctly characterizes the early-stage training behavior. We extended the analysis to late-stage training in a new Theorem 3 and related the behavior of \beta_h to generalization error under shortcut learning in a new Theorem 4.

We also restated the assumptions of theorems more explicitly. To preserve space and readability, we marked the statements of Theorems 2, 3, and 4 in the main paper as “informal”, and refer the reader to the appendix for formal mathematical details.


W1: Furthermore, defining an “easy” feature as one with larger covariance with the target conflates how predictive a feature is with how hard it is to extract, and a shrinking gradient gap does not on its own show that a transferable rather than spurious feature is being learned.

A2: We agree that the idea of “easy feature” is associated with both predicability and ease of extraction and we are only dealing with the first aspect. We specifically added a limitation statement (Line533--535) for that. In practice, the data selection network is a 3-layer MLP, and the input contains only 10 features, so the ability of the network to extract sophisticated features is limited.

We also agree that a shrinking gradient gap alone does not establish transferability. To fix this, we added Theorems 3 and 4 to analyze the behaviors of βh\beta_h and βe\beta_e at convergence and relate those to generalization performance.

AuthorsOpenReview ↗

Response to Reviewer grpE (Part 2): Weakness 2 and Questions 2–3

W2/Q2: In the 197K-example selection experiment, what does each stage cost separately: training the two LLMs, extracting representation and gradient features, computing validation distances, training the MLP, and scoring the pool? Which of these stages are included in the reported 13.4 hours?

A3: Thank you for catching that. We realized that we inadvertently omitted the 5.2 hours needed for the extraction of all features. The originally reported 13.4 hours included both LLM training runs, pseudo-label generation, MLP training, and pool scoring. On the 197K-sample dataset, TESS takes 18.6 hours in total, compared with the reported 59.8 hours for SBO$_w$ and 39.6 hours for SEAL$_w$, or 3.22× and 2.13× of TESS. We revised the manuscript (Line522--525). The most expensive step in feature extraction is to compute and store the LLM gradient features, which contain up to 7B dimensions. To simplify computation and storage, we project per-example gradients into 4,096 dimensions. To accelerate the projection, we use a sparse mapping in which each original dimension contributes to eight output buckets. For each example, we jointly extract the loss, gradient norm, projected gradient representation, and last-layer embedding in a single forward–backward pass, avoiding redundant model passes.


W2/Q3: Please also explain how signed loss-gap targets are represented or transformed given that the selector output is restricted to (0,1), and whether this affects the ranking of low-score harmful data. How does PVM compare with a simple predictor or ranking rule built on the same ten-dimensional features? A scorer based on validation distance or simple features that uses these same inputs would also help separate the contribution of loss-gap regression from the relevance information supplied to the network.

A4: Regarding the signed loss-gap targets, we apply a sigmoid to both the raw targets and the final-layer outputs of the selector network so that they share the same range, (0,1). Bounding the targets limits the influence of extreme loss gaps on training. The sigmoid is strictly increasing, so it preserves the target ordering, including which examples have the lowest scores. To examine its effect, we create a new baseline TESS w/o sigmoid, we remove the sigmoid from the selector's final layer and regress directly on the raw signed loss gaps. After the network is trained, we select the top-N data points ranked by the network as the new training dataset. Hence, it does not matter if the network predicts a negative value. To answer the question on a simple predictor or ranking rule, we constructed three additional baselines: two distance-based ranking heuristics (Grad and Emb) and one learned baseline (Linear). Grad and Emb rank examples by their mean distance to validation examples in gradient and representation space, respectively. Linear fits a linear regression model to predict the raw signed loss gaps. In the safe–unsafe data classification experiment (Section 4.2), we report accuracy on the seen and unseen splits.

MethodAcc_seenAcc_unseenAvg.
Grad84.1884.8984.54
Emb56.0955.2955.69
Linear74.9374.0474.49
TESS98.5897.9698.27
TESS w/o sigmoid98.5897.8798.23

In the GSM8K targeted instruction tuning experiment (Section 5.2), we select N training examples with each method and report GSM8K accuracy after fine-tuning.

MethodN = 5KN = 10KAvg.
Grad22.7422.2922.52
Emb16.4517.7417.10
Linear21.3020.5520.93
TESS25.4725.7025.59
TESS w/o sigmoid25.0225.1725.10

TESS outperforms the all simpler baselines, Grad, Emb, and Linear, on both tasks. These results show that learning an MLP from the same ten-dimensional features works better than the simple predictors and ranking rules we tested. TESS w/o sigmoid performs very similarly with TESS, showing that the sigmoid transformation has little effects on the results.

Reviewer: please respond to the authors.

Comments on the paper

2 comments

Reviewers: please respond to the authors and let them know if there are any remaining concerns.

Dear Reviewers HV1w, hc2v, and grpE,

Thank you for your thoughtful reviews and valuable suggestions. May we kindly ask whether our responses have addressed your concerns? We would greatly appreciate any further feedback and would be happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.