acceptodds
Under review as a conference paper at ICLR 2027

Good Sources or Good Matches: Disentangling Source Quality in Language Model Transfer

Abstract

Data-selection methods seek to predict which training sources will benefit a target, but prediction accuracy alone does not reveal whether they identify generally useful sources or target-specific matches. This distinction determines whether past transfer experiments can guide future selection or whether new target-specific evidence is needed. We investigate this question through adapter transfer on 15 datasets and four language models. We measure relative source usefulness through target-conditioned rankings and use HodgeRank to identify how much transfer asymmetry is explained by a common source ordering. This ordering corresponds to average source quality and explains most squared asymmetry in the decoder-only models studied; T5 shows more directional variation that a single ordering cannot explain. We then evaluate two ways to reuse measured transfer. Historical source quality guides selection for incoming targets from an established model library, outperforming the tested gradient references in the decoder-only settings. Transfer measured on a smaller pilot also supports source selection for Qwen3-8B and LLaMA, while target-model gradients remain stronger for T5. Further analysis shows that the pilot’s general source ordering predicts target-model asymmetry more accurately than its full directional structure. These results clarify what makes transfer evidence useful and provide practical ways to select source datasets using observations from previous targets or a smaller model.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
data selection, transfer learning, dataset valuation, source quality, cross-task transfer, HodgeRank, language model adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VxmROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors analyze the transferability between different training sources by separating generally useful and target-specific benefits. They use HodgeRank to decompose the influence to general source benefits and pair-specific asymmetry. They find that in decoder-only models (such as Qwen and LLaMA) the source-general benefits can explain most of transfer asymmetry. Based on that, they evaluate the strategy that selects data based on the general source ordering when obtaining a new training task. Empirically, they find that the strategy works well across different models. Besides, they also find that the quality of training sources evaluated by a small model can transfer to a larger model.

Critical strengths
  1. Data selection is a crucial problem in the training process of LLMs. The authors provide a new perspective that decomposes the advantages of training sources to generally useful and target specific benefits. The idea is reasonable and interesting.
  2. The empirical results are solid to verify the main insights in this paper. The experiments with different decoder-only models show that the generally useful benefits are more important than target-specific benefits.
  3. The paper is organized well and easy to follow. The logic flow is clear.
Critical weaknesses
  1. As shown in Table1, the residual component can not be ignored in the analysis. However, the strategy in the following section mainly select data based on source-general ranking. Is it possible to design adaptive strategies or coorprates two source better?
  2. The experiments in this paper seem to mainly focus on adapter training. However, I think it would be much better to verify the insights of data selection in more post-training paradigms including supervised fine-tuning with full parameters and reinforcement learning.
  3. The main comparisons in this paper are evaluated on methods based gradient similarity. Is it possible to provide comparisons with more baselines in the relation work section.
  4. I think more discussions on the comparison between decoder-only and encoder-decoder models are needed. The experiements show that there exist significant differences . However, it seems that the understanding of the difference is limited.
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Thank you for suggesting an adaptive strategy using the residual (1/3)

Thank you for the positive assessment and the constructive suggestions. We are glad you found the decomposition “reasonable and interesting”, the empirical results “solid”, and the paper “organized well and easy to follow”. We answer each weakness (W1–W4) below. Your weaknesses were not numbered, so we numbered them in order, and your questions referred to them.

Weaknesses

W1. “As shown in Table1, the residual component can not be ignored in the analysis. However, the strategy in the following section mainly select data based on source-general ranking. Is it possible to design adaptive strategies or coorprates two source better?”

Response. Thank you for this excellent suggestion, which led to the switch, now the best rule in the paper.

Splitting the residual. The revision splits the residual into mutual match, where two tasks help each other alike, and directional match (Table 1). Mutual match is at least as large as directional match on every model, and it concentrates within task groups, so the target's label names capture it. Same-group pairs have higher mutual match than cross-group pairs on all four models (Fig. 3b). The residual is also mostly real rather than training noise. Between Qwen3-1.7B seeds, mutual match correlates at 0.59 (rate 10⁻³, three seeds) and 0.78 (rate 2 × 10⁻⁴, two seeds), and directional match at 0.41 and 0.77, all above every one of 2,000 task relabelings (p < 0.001; App. C.5, Table C.3).

ModelWithin-group minus cross-group mutual matchPermutation p
Qwen3-1.7B+0.0980.020
Qwen3-8B+0.1470.0003
LLaMA-2-7B+0.2000.0006
Gemma-2-2B+0.2180.0002

The adaptive rule. The switch follows records when the source share of past tasks reaches a cutoff and otherwise falls back to a matched-value rule (Sec. 3, Alg. 2). The cutoff is set by leave-one-model-out validation (τ = 0.23–0.24). The fallback is pluggable, and the switch improves both fallbacks we test (Table 2).

RuleRegret@1 (%)Gain over randomWorst model
Records alone13.3 ± 2.7+13.1 ± 1.9+6.9 ± 3.7
Label overlap alone17.5 ± 3.6+9.5 ± 3.2+5.6 ± 3.0
Switch + label overlap10.4 ± 1.6+14.9 ± 2.2+9.2 ± 3.0
LESS-style alone17.3 ± 3.1+9.7 ± 2.7+6.6 ± 4.6
Switch + LESS-style12.1 ± 1.8+13.8 ± 2.1+7.0 ± 2.9

The switch with label overlap beats label overlap alone (p = 0.012), and the switch with LESS-style beats LESS-style alone by 4.1 points (p = 0.023). Across ten runs over models, seeds, and learning rates, the higher the share, the more records beat label match (Spearman 0.68, p = 0.034). Below the cutoff label match beats records by 3.8 points, and above it records win by 4.1 (Fig. 3a). On the five runs with a share below the cutoff, the switch beats records by 4.2 points (p = 0.046). Any cutoff from 0.14 to 0.42 keeps the switch at or above records, and on six runs never used to set the cutoff it matches or beats records on five (Sec. 4.4).

AuthorsOpenReview ↗

Thank you for raising other post-training paradigms (2/3)

Combining two sources (continued response to W1). If the two sources are the two parts of value, the switch is our answer, as above. If they are two datasets trained together, the same records also guide the mixture, with no target data and no extra training (new Appendix C.9). We trained Qwen3-1.7B on all 105 pairs of the 15 datasets, each mixture taking half of each dataset, and used these mixtures only to check predictions made from single-dataset records of two other seeds. Each task was held out in turn and removed from the records before any prediction.

  • General value picks good mixtures. Ranking pairs by the mean general value of their two datasets orders the mixtures correctly on all 15 held-out tasks. Its top ten pairs gain 8.5 points more than an average pair and 14.7 more than its bottom ten (14 of 15 tasks, p = 0.0002), ahead of LESS-style (+7.9) and similarity to the target (+6.4), which both see the held-out task (Table C.8).
  • Matched value finds complementary and redundant pairs. Two datasets whose matched value falls on the same tasks are redundant, and two whose matched value falls on different tasks are complementary. The ten most complementary pairs beat the better of their two datasets by 5.0 points more than the ten most redundant ones (12 of 15 tasks, p = 0.002). Correlating raw gains instead gives 0.9 points, and task groups cannot tell whether a mixture beats its better dataset (Table C.9).
  • The map goes beyond task labels. Records agree with task groups for Yelp with Amazon and DBpedia with Yahoo, and they also flag pairs across groups, such as QQP with MultiRC (redundant) and QQP with IMDB (complementary), which the trained mixtures confirm (Fig. C.4).

W2. “The experiments in this paper seem to mainly focus on adapter training. However, I think it would be much better to verify the insights of data selection in more post-training paradigms including supervised fine-tuning with full parameters and reinforcement learning.”

Response. Thank you for this suggestion. The method needs only the matrix of transfer results that training produces, so it applies unchanged to full fine-tuning and reinforcement learning, and the Conclusion names both as next steps. Within adapter training, the revision tests a change of training regime through the learning rate on Qwen3-1.7B (Table 3).

Rate (seeds)ShareLargestLabel overlapRecordsSwitchOracle
10⁻³ (3)40%−9.5 ± 2.2+11.1 ± 3.4+12.8 ± 2.6+12.8 ± 2.6+20.9 ± 2.6
5 × 10⁻⁴ (1)35%−1.5 ± 2.3+4.5 ± 3.9+4.7 ± 1.5+4.7 ± 1.5+17.3 ± 2.7
2 × 10⁻⁴ (2)15%+1.0 ± 1.9+3.3 ± 2.9+3.0 ± 1.3+3.3 ± 2.9+13.5 ± 2.4

Records beat random at every rate (p ≤ 0.015). Gentler training gives target match more weight, the share falls to 14–16% at 2 × 10⁻⁴, below the cutoff, and the switch moves to label overlap, the best rule at that rate. The share thus tracks a change of training regime and selects the right rule for it, which is the property a new paradigm would need.

Evidence from prior work. The method needs a data ranking measured in one training setup to hold in another, and prior work gives direct evidence that such rankings transfer. LESS (Xia et al., 2024) selects data with the gradients of LLaMA-2-7B, and the selection still helps when training LLaMA-2-13B and Mistral-7B. RegMix (Liu et al., 2025) finds that the ranking of data mixtures from 1M-parameter proxy models holds for 1B-parameter models, and DoReMi (Xie et al., 2023a) tunes domain weights on a 280M-parameter proxy that improve an 8B model. Our results add two such changes: records from Qwen3-1.7B rank datasets for Qwen3-8B, and from Qwen3-0.6B for Qwen3-1.7B (Sec. 4.5), and the share tracks a change of learning rate (Table 3). None of these tests full fine-tuning or reinforcement learning directly, so we state them as next steps rather than as claims.

References

  • Xia et al. (2024). LESS: Selecting Influential Data for Targeted Instruction Tuning.
  • Liu et al. (2025). RegMix: Data Mixture as Regression for Language Model Pre-training.
  • Xie et al. (2023a). DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining.
AuthorsOpenReview ↗

Thank you for your suggestions on baselines and model architectures (3/3)

W3. “The main comparisons in this paper are evaluated on methods based gradient similarity. Is it possible to provide comparisons with more baselines in the relation work section.”

Response. Thank you for this suggestion. The revision adds the methods from the related work as baselines: V-usable information, perplexity, IFD, Data-OOB, label overlap, description similarity, OTDD (the dataset distance behind LAVA), SEmb, LogME, LEEP, DSIR, and text similarity, alongside the gradient methods LESS-style and Grad Cos (Sec. 4, Table 2). Records beat every target-free rule (p < 0.001) and seven of the eight rules that see target examples, and the switch significantly beats all of them (p ≤ 0.024). The full numbers are in the headline table of the general response. We also compare records with simpler uses of the same past results, and ranking sources within each past target matters: averaging raw accuracy or win rates over past targets gains +10.0 and +8.7 points, below records at +13.1 (p = 0.024 and p < 0.001; App. C.5, Table C.4). The related work section adds data valuation, including Data Shapley (Ghorbani & Zou, 2019), influence functions (Koh & Liang, 2017), TracIn (Pruthi et al., 2020), Datamodels (Ilyas et al., 2022), LAVA (Just et al., 2023), OTDD (Alvarez-Melis & Fusi, 2020), Projektor (Kang et al., 2023, “Performance Scaling via Optimal Transport”, NeurIPS), and Data-OOB (Kwon & Zou, 2023) (Sec. 2). The same baselines, adapted to free-form answers, are compared on the new math library, where records rank datasets better than all 13 by NDCG and weighted τ (Table 2).

W4. “I think more discussions on the comparison between decoder-only and encoder-decoder models are needed. The experiements show that there exist significant differences . However, it seems that the understanding of the difference is limited.”

Response. Thank you for this point. The revision measures model differences with the decomposition (Table 1, as in KFUw Q3). The share varies among decoder models too, from 52.5% on Qwen3-1.7B to 10.4% on LLaMA-2-7B, and a low share comes with more mutual match. On LLaMA-2-7B, mutual match is 47.5% of ranking variation and general value only 10.4%. T5, the one encoder-decoder model, has a general ordering that differs from the decoder models, so pilot records do not carry over between T5 and them in either direction, while each model's own records still beat every baseline (App. C.4). The difference is thus one case of a measurable quantity. The share can be checked on any model before past records are trusted, and the switch uses label match where it is low.

References

  • Ghorbani & Zou (2019). Data Shapley: Equitable Valuation of Data for Machine Learning.
  • Koh & Liang (2017). Understanding Black-box Predictions via Influence Functions.
  • Pruthi et al. (2020). Estimating Training Data Influence by Tracing Gradient Descent.
  • Ilyas et al. (2022). Datamodels: Understanding Predictions with Data and Data with Predictions.
  • Just et al. (2023). LAVA: Data Valuation without Pre-Specified Learning Algorithms.
  • Alvarez-Melis & Fusi (2020). Geometric Dataset Distances via Optimal Transport.
  • Kang et al. (2023). Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources.
  • Kwon & Zou (2023). Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Value.
Reviewer_KFUwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors discussed how the the previously trained adaptor can be used to predict selection of data source. The experiment was conducted on 15 sources and 4 models, and transfer between models was also tested.

Critical strengths
  1. The prediction between sources is meaningful and can enhance efficiency when facing a large amount of data.
  2. The rank-based selection can help to normalize the performance under different tasks.
  3. In the experiment, several models benefit from the selection of data sources.
Critical weaknesses
  1. While the data source selection being meaningful, the setting in the paper seems less motivated. When you have the true label for a target, it is more direct to train the model with the traget source, rather than use others. In that sense, the value of source selection will be limited.
  2. The experiments are conducted on different pretrained models. These model are trained with different data, which might already contain the 15 dataset. The ranking-based results can not eliminate this data contamination. Moreover, the transfer results on different models seem to be weak. There are only two models exhibit positive results, and the generalization is not validated on broader model ranges or model sizes.
  3. The categories of tested dataset are limited, and the selected models seem mismatching. The authors focused on supervised tasks, which is more suitable with encoder-only models. The results on encoder-only models are missing. The dataset are all related to conventional sentence understanding, rather than the continual pretraining-related datasets which are more related to target data selection.
  4. The training budget across different sources may be different. One epoch per source does not ensure the same training tokens.
  5. There is no baseline comparison with state-of-the-art algorithms.
  6. The performance gaps are small compared to the standard error in Table 2.
Questions for authors
  1. Is this method effective for other domains such as math, code, financial data?
  2. How is the method performed compared with state-of-the-art algorithms?
  3. How do the architectures affect the performance of transfer?
8 comments
AuthorsOpenReview ↗

Thank you for helping us clarify the setting and motivation (1/6)

Thank you for the detailed review. We are glad you found that “the prediction between sources is meaningful and can enhance efficiency when facing a large amount of data” and that “the rank-based selection can help to normalize the performance under different tasks”. We answer each weakness (W1–W6) and question (Q1–Q3) below. Your points were not numbered, so we numbered them in order.

Weaknesses

W1. “While the data source selection being meaningful, the setting in the paper seems less motivated. When you have the true label for a target, it is more direct to train the model with the traget source, rather than use others. In that sense, the value of source selection will be limited.”

Response. Thank you for raising this point, which helped us state the setting more clearly. If target labels exist, the target data is used too, and the question becomes which other datasets to add, as in intermediate-task transfer (Pruksachatkun et al., 2020; Vu et al., 2020; Poth et al., 2021) and targeted data selection (Xia et al., 2024). Many decisions are also made before any target data exists. A team that buys, curates, or prices data must value it before the tasks it will serve are known (Agarwal et al., 2019; Sim et al., 2022). In expert libraries, one adapter is trained per dataset and one is chosen for each unseen task (Jang et al., 2023; Ostapenko et al., 2024), where a single well-chosen expert can outperform a model trained on hundreds of tasks (Jang et al., 2023). The revision states this setting in Sec. 1 and Sec. 4.

Valuing before the task is known is an open question in data valuation. Ghorbani et al. (2020) name the reliance on a known task as an outstanding limitation, and Agarwal et al. (2019) argue that past success says little about a new buyer's task. Our results answer it for this setting: much of a dataset's value carries over to new tasks, and even to task families never seen before (+9.0 ± 1.2 points over random, W3).

Labeled target examples are not needed for good selection. Records use no data from the new task, while the baselines below use 100 labeled target examples (Table 2).

RuleTarget dataGain over randomRecords ahead, pp
Records (ours)none+13.1 ± 1.9
LEEPlabeled+7.5 ± 2.6p=0.027p=0.027
Grad Coslabeled+6.1 ± 3.5p=0.034p=0.034
OTDDlabeled+5.8 ± 2.6p=0.008p=0.008
LogMElabeled+5.8 ± 1.8p=0.003p=0.003

Records are also ahead of LESS-style on average (+9.7 ± 2.7), though not significantly (p=0.22p=0.22). Adding only the target's label names, the switch reaches +14.9 ± 2.2 and significantly beats every rule that uses labeled examples, including LESS-style (p=0.019p=0.019). A dataset can therefore be valued once, before any task arrives, which is what buying, curating, and pricing data require.

Adding to target data. Our experiments train each source alone, which isolates its value for the target. When target data is also available, intermediate-task transfer trains on a source and then on the target, and prior work finds that the choice of source still changes target accuracy substantially, sometimes for the worse (Pruksachatkun et al., 2020; Vu et al., 2020; Poth et al., 2021). Ranking sources therefore remains useful there, and testing records in that setting is a natural extension.

References

  • Pruksachatkun et al. (2020). Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
  • Vu et al. (2020). Exploring and Predicting Transferability across NLP Tasks.
  • Poth et al. (2021). What to Pre-Train On? Efficient Intermediate Task Selection.
  • Xia et al. (2024). LESS: Selecting Influential Data for Targeted Instruction Tuning.
  • Agarwal et al. (2019). A Marketplace for Data: An Algorithmic Solution.
  • Sim et al. (2022). Data Valuation in Machine Learning: “Ingredients”, Strategies, and Open Challenges.
  • Jang et al. (2023). Exploring the Benefits of Training Expert Language Models over Instruction Tuning.
  • Ostapenko et al. (2024). Towards Modular LLMs by Building and Reusing a Library of LoRAs.
  • Ghorbani et al. (2020). A Distributional Framework for Data Valuation.
AuthorsOpenReview ↗

Thank you for raising contamination and cross-model transfer (2/6)

W2. “The experiments are conducted on different pretrained models. These model are trained with different data, which might already contain the 15 dataset. The ranking-based results can not eliminate this data contamination. Moreover, the transfer results on different models seem to be weak. There are only two models exhibit positive results, and the generalization is not validated on broader model ranges or model sizes.”

Response. Thank you for these points. Contamination can enter on the target side or on the source side, and the two cases differ.

Target-side exposure. If a model saw a target in pretraining, its accuracy on that target shifts for every candidate source alike, because all candidates are trained on the same base model and compared on the same target (Sec. 3). Ranking sources within each target removes this common shift, so the order of sources is largely unaffected. Different models are never compared on raw accuracy.

Source-side exposure. If a model saw a source dataset in pretraining, adapting on it may add less or more than on an unseen dataset, so the direction of the effect is uncertain. Two points limit its impact here. First, any such effect is part of what that dataset adds to that model, the quantity a user of the model needs, and every rule is scored against the same measured accuracies. Second, results hold where source exposure is ruled out. Six of the 28 math sources (OpenMathInstruct-2, AceMath, OpenThoughts, OpenR1-Math, Bespoke-Stratos, and LIMO) were released after Gemma-2-2B, so they cannot be in its pretraining data, and records still rank datasets better than every baseline on Gemma-2-2B (weighted τ\tau 0.49 against at most 0.28, Table C.7).

Records now beat random on every model, across three families and sizes from 1.7B to 8B parameters on text (Table 1, Secs. 4.2–4.3) and from 0.6B to 8B on math (Q1).

ModelParametersSource share (%)Records' gain over random
Qwen3-1.7B1.7B52.5 ± 11.5+13.5
Qwen3-8B8B44.4 ± 18.9+23.8
LLaMA-2-7B7B10.4 ± 8.1+6.9
Gemma-2-2B2B22.9 ± 11.5+8.3

The switch stays at least 9.2 points above random on every model. Over three Qwen3-1.7B seeds, records stay 13.5, 10.5, and 14.6 points above random (p≤0.003p\le0.003), and top-3 shortlists built on one seed beat random on the other seeds in all six directions (p≤0.025p\le0.025) (Sec. 4.3).

Transfer between models works within a family. Records from Qwen3-1.7B, with about a fifth of the parameters, rank datasets for Qwen3-8B with no Qwen3-8B training (Sec. 4.5, Fig. 4).

Rule for Qwen3-8BGain over randomPilot ahead by
Qwen3-1.7B pilot records (ours)+20.7 ± 1.7
LESS-style (Qwen3-8B gradients)+10.6 ± 1.910.1 (p=0.003p=0.003)
OTDD+5.4 ± 2.615.3 (p<0.001p<0.001)
Grad Cos (Qwen3-8B gradients)+5.4 ± 5.015.3 (p=0.019p=0.019)
SEmb+0.2 ± 2.820.5 (p<0.001p<0.001)
Largest dataset−10.9 ± 2.531.6 (p<0.001p<0.001)

Pilot records recover Qwen3-8B's rankings as well as Qwen3-8B's own records (within-target Spearman 0.57 for both), against 0.27 for OTDD and at most 0.18 for any other baseline, and lose only 5.8% against the best dataset, against at least 20% for every baseline. The datasets that carry general value differ across families (Fig. C.1), so the revision states that a pilot should come from the same family as the target model.

The pilot also works on math, and so do shorter runs (Sec. 4.5). Qwen3-1.7B general values correlate at 0.89 with Qwen3-8B's, and Qwen3-0.6B values at 0.88 with Qwen3-1.7B's. On catalogs of five candidates, pilot records gain 10.3 and 10.1 points over random, as much as each model's own records, against at most 4.4 for perplexity and the largest dataset (p≤0.001p\le0.001). Records from adapters stopped at a tenth of training match those of complete runs (gain 10.7 against 9.7, weighted τ\tau 0.43 against 0.46, both p≥0.12p\ge0.12), so records can cost a tenth of a full run per dataset.

AuthorsOpenReview ↗

Thank you for encouraging broader task and model coverage (3/6)

W3. “The categories of tested dataset are limited, and the selected models seem mismatching. The authors focused on supervised tasks, which is more suitable with encoder-only models. The results on encoder-only models are missing. The dataset are all related to conventional sentence understanding, rather than the continual pretraining-related datasets which are more related to target data selection.”

Response. Thank you for this suggestion. We use the 15-task benchmark of Wang et al. (2023), the standard benchmark for continual learning of large language models, which spans sentiment (Yelp, Amazon, IMDB, SST-2), topic (DBpedia, Yahoo Answers, AG News), inference (MNLI, RTE, CB), paraphrase (QQP), word sense (WiC), causal reasoning (COPA), and reading comprehension (MultiRC, BoolQ). Each task is posed as an instruction and scored by rank classification, the protocol T0 uses for unseen tasks with decoder models (Sanh et al., 2022), and the same protocol underlies expert libraries for decoder LLMs (Jang et al., 2023; Ostapenko et al., 2024). Decoder models are the ones such libraries and data selection methods such as LESS target, which is why we study them.

The variety across task types matters for the result. Records often pick a dataset of a different type that shares no label with the target, and these picks still help (Table C.5).

ModelPicks sharing no labelBeat randomMean gainLargest gain
Qwen3-1.7B10/159/10+12.7 ± 2.8MNLI → DBpedia, +26.0
Qwen3-8B10/1510/10+21.3 ± 2.0CB → COPA, +29.6
LLaMA-2-7B13/158/13+4.0 ± 3.6SST-2 → DBpedia, +40.3
Gemma-2-2B15/1512/15+8.3 ± 2.5COPA → IMDB, +25.7

The revision also tests task categories directly. For each target, we remove its whole task family (sentiment, inference, topic, or other) both from the past tasks that build records and from the candidates, so records must value data for a kind of task they have never seen (Sec. 4.3, App. C.5).

Rule, family held outQwen3-1.7BQwen3-8BLLaMA-2-7BGemma-2-2BMean gain over random
Records (ours)+7.6+20.2−1.1+9.2+9.0 ± 1.2
Label overlap+6.9+0.6+2.7+8.0+4.5 ± 2.5
Largest dataset−5.9−8.2−2.1−0.6−4.2 ± 1.7

Records still beat random on a new family (p<0.001p<0.001) and beat the largest dataset (p<0.001p<0.001). LLaMA-2-7B is the exception, and it is also the model with the lowest share, where the switch uses label overlap instead.

The math library (Q1) also goes beyond sentence understanding, with generative targets and sources that include raw web text (MathPile) and web-extracted question-answer pairs (WebInstructSub), the kind of corpora used for continual pretraining.

Encoder-only models. Source selection for encoder-only models is well studied by the works our baselines come from, which use BERT and RoBERTa (Pruksachatkun et al., 2020; Vu et al., 2020; Poth et al., 2021). We focus on decoder models, where expert libraries and data selection for LLMs operate and the question is less studied, and we include T5, an encoder-decoder model, in App. C.4. Encoder-only models and full continual-pretraining corpora are not tested; the decomposition needs only a matrix of transfer results, so it applies to them unchanged.

References

  • Wang et al. (2023). Orthogonal Subspace Learning for Language Model Continual Learning.
  • Sanh et al. (2022). Multitask Prompted Training Enables Zero-Shot Task Generalization.
  • Jang et al. (2023). Exploring the Benefits of Training Expert Language Models over Instruction Tuning.
  • Ostapenko et al. (2024). Towards Modular LLMs by Building and Reusing a Library of LoRAs.
  • Pruksachatkun et al. (2020). Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
  • Vu et al. (2020). Exploring and Predicting Transferability across NLP Tasks.
  • Poth et al. (2021). What to Pre-Train On? Efficient Intermediate Task Selection.
Reviewer_4hTaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces the problem of distinguishing and explaining the effect of source quality from its relevance (to the target) in model transfer. They formulate the problem by defining the source's general quality and how much it can explain the asymmetry in model transfer. They conduct extensive experiments using 15 NLP tasks over 4 models.

Critical strengths

The paper addresses a very interesting problem: separating the effect of source data quality from its usefulness/relevance to the target. Understanding this problem would yield not only better data selection but also the ability to assess dataset quality. The paper also introduces an interesting formulation to investigate this problem.

Critical weaknesses

The paper would benefit from reorganization and better framing; currently, it's difficult to follow and understand the work's main contributions. The experiments are limited to 4 language models, which isn't enough to draw general conclusions. For example, in Table 1, we clearly observe that using T5, the share of dataset quality is significantly lower than that of other models. The authors can also include additional baselines for dataset selection from the literature. I suggest including more literature related to data selection or works on data evaluation, such as https://arxiv.org/abs/2110.08420. There are issues with the references section of this paper. Most of the references are incorrect or have formatting issues.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Thank you for helping us improve the framing and broaden validation (1/2)

Thank you for the review and for finding that the paper “addresses a very interesting problem” with “an interesting formulation”. We answer each weakness (W1–W5) below. Your weaknesses were not numbered, so we numbered them in order.

Weaknesses

W1. “The paper would benefit from reorganization and better framing; currently, it's difficult to follow and understand the work's main contributions.”

Response. Thank you for this frank assessment, which led to the largest rewrite. The paper is now organized around one data valuation question, namely how much of a dataset's value carries over to new tasks and how much is matched to a target. Valuation methods assume value is tied to a known target, transfer studies disagree on whether sources or targets matter, and the part of value that carries over had not been separated and tested (Sec. 1, Sec. 2). The introduction states four contributions: a HodgeRank decomposition of data value into general and matched parts, the valuation rules derived from it (records and the switch), results across models and learning rates, and low-cost records from a small pilot model, shorter runs, and a few past tasks (Sec. 1). Sec. 3 derives each rule from one part of the decomposition, and Sec. 4 tests one claim per subsection.

  • 4.1 Setup. Text and math benchmarks, metrics, and one set of baselines for both.
  • 4.2 The general part of data value. A part of data value carries over across tasks, on every model.
  • 4.3 Valuing datasets for an incoming target. Records built from that part rank datasets better than the baselines, on both benchmarks and four models each, and also guide two-dataset mixtures.
  • 4.4 Choosing between general and matched value. The source share decides when to use records and when label match.
  • 4.5 Lowering the cost of records. A smaller pilot model, shorter runs, and scoring a new dataset on a few past tasks.

W2. “The experiments are limited to 4 language models, which isn't enough to draw general conclusions. For example, in Table 1, we clearly observe that using T5, the share of dataset quality is significantly lower than that of other models.”

Response. Thank you for this observation, which the revision builds on. We add Gemma-2-2B as a fourth decoder model, two extra Qwen3-1.7B seeds, a second Gemma-2-2B seed, and three learning rates, ten runs in all (Secs. 4.3–4.4). A low share is not specific to T5. It is 10.4% on LLaMA-2-7B and 22.9% on Gemma-2-2B, against 52.5% and 44.4% on the Qwen3 models (Table 1). The method therefore measures the share on each model and uses it to choose the rule (Sec. 4.4).

RunsRule chosen by the switchResult
Share above cutoffRecordsRecords beat label match by 4.1 points
Share below cutoff (five runs)Label overlapSwitch beats records by 4.2 points (p = 0.046)
Second Gemma-2-2B seed, share 14%Label overlap+10.1 over random (p = 0.007 against records)

Records still beat random on every decoder model, by 6.9 to 23.8 points, and T5 is analyzed in App. C.4. Beyond models, the revision adds a second main benchmark of 28 math datasets (Sec. 4, Table 2). There, the share is 0.34 and 0.48 on two Qwen3-1.7B seeds, and records rank datasets better than all 13 applicable baselines by NDCG and weighted τ on each of four models (Qwen3-0.6B, Qwen3-1.7B, Qwen3-8B, Gemma-2-2B; Table C.7).

A note on Table 1. The revised Table 1 uses a different measure from the submitted one. It splits the variation of within-target rankings, with tasks scored by rank classification, whereas the submitted table split the asymmetry between task pairs, scored by exact match. Its numbers therefore differ from the submitted table, where LLaMA-2-7B was 75.8%. Under the submitted measure, T5 remains the lowest (44.0%).

AuthorsOpenReview ↗

Thank you for your baseline, related-work, and reference suggestions (2/2)

W3. “The authors can also include additional baselines for dataset selection from the literature.”

Response. Thank you for this suggestion. The revision adds sixteen baselines in three groups (Table 2), listed in KFUw W5. Our rule is best in every group. On the new math benchmark, records rank datasets better than all 13 baselines that apply, including the largest dataset, on each of four models (Table 2, Table C.7).

GroupOur rule, gainBest baseline, gainOurs ahead, pp
No target dataRecords +13.1 ± 1.9Data-OOB +3.1 ± 2.2p<0.001p<0.001
Label names or descriptionSwitch +14.9 ± 2.2Description +10.7 ± 3.0p=0.024p=0.024
Target examplesSwitch + LESS-style +13.8 ± 2.1LESS-style +9.7 ± 2.7p=0.023p=0.023

W4. “I suggest including more literature related to data selection or works on data evaluation, such as https://arxiv.org/abs/2110.08420.”

Response. Thank you for the pointer. The suggested paper, V-usable information (Ethayarajh et al., 2022), is both cited and used as a baseline. It values a dataset by how much usable information its inputs hold about its own labels, and it gains −0.3 ± 2.4 points over random against +13.1 ± 1.9 for records (p<0.001p<0.001). The revision adds a data valuation section (Sec. 2) covering target-specific methods (Data Shapley, influence functions, TracIn, Datamodels, LESS, LAVA, OTDD, Projektor), target-free methods (Data-OOB and V-usable information), data markets, and intermediate-task selection (Pruksachatkun et al., 2020; Vu et al., 2020; Poth et al., 2021). It also covers transfer studies that use pairwise transfer results, to pick training tasks for zero-shot generalization (Zhou et al., 2023), to choose source tasks together with a few target examples (Kim et al., 2023), or to study donors and recipients in LoRA transfer (Dymkiewicz et al., 2025), as well as distributional Shapley (Ghorbani et al., 2020) and volume-based valuation (Xu et al., 2021). Dymkiewicz et al. come closest: the source explains 10% of the variance of raw gains in their study, against 41% for the target. Raw gains, however, are dominated by how much each target gains from any training, which cannot help choose among sources for one target. We therefore compare sources within each target and test the part that carries over against methods that see the target, which none of these works does. Measured this way, the part that carries over, with no target data, scores higher on average than every rule that sees target examples.

W5. “There are issues with the references section of this paper. Most of the references are incorrect or have formatting issues.”

Response. Thank you for pointing this out. Every entry of the bibliography was checked against its publisher record for authors, title, venue, year, and pages, and all entries now follow one format. The revision also cites each model and method at first mention, including Qwen3, LLaMA-2, Gemma-2, T5, and LoRA.

References

  • Ethayarajh et al. (2022). Understanding Dataset Difficulty with V-Usable Information.
  • Pruksachatkun et al. (2020). Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
  • Vu et al. (2020). Exploring and Predicting Transferability across NLP Tasks.
  • Poth et al. (2021). What to Pre-Train On? Efficient Intermediate Task Selection.
  • Zhou et al. (2023). Not All Tasks Are Born Equal: Understanding Zero-Shot Generalization.
  • Kim et al. (2023). TaskWeb: Selecting Better Source Tasks for Multi-Task NLP.
  • Dymkiewicz et al. (2025). Donors and Recipients: On Asymmetric Transfer across Tasks and Languages with Parameter-Efficient Fine-Tuning.
  • Ghorbani et al. (2020). A Distributional Framework for Data Valuation.
  • Xu et al. (2021). Validation Free and Replication Robust Volume-Based Data Valuation.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response: Headline results (3/3)

Headline result (Table 2). Each of the 15 tasks is held out in turn as the incoming target, and the other 14 are past datasets. Each rule ranks the candidates. Gain is the first pick's accuracy over a random pick in points; NDCG (Vu et al., 2020; Poth et al., 2021) and weighted Kendall's τ\tau (You et al., 2021) are standard measures of the whole ranking against target accuracy (higher is better). Text is averaged over Qwen3-1.7B, Qwen3-8B, LLaMA-2-7B, and Gemma-2-2B, math (new) over two training seeds of Qwen3-1.7B; ± is the standard error over targets. Empty cells mean not applicable: those rules need class labels.

Target dataRuleText: GainText: NDCGText: Weighted τ\tauMath: GainMath: NDCGMath: Weighted τ\tau
NoneRecords (ours)+13.1 ± 1.90.9520.38 ± 0.04+15.7 ± 2.70.9800.58 ± 0.10
NoneData-OOB+3.1 ± 2.20.9060.02 ± 0.06
NonePerplexity−2.4 ± 0.80.890−0.02 ± 0.05+15.8 ± 2.70.9410.32 ± 0.04
NoneLargest dataset−6.1 ± 2.00.885−0.14 ± 0.04+11.1 ± 1.90.9310.27 ± 0.04
Label namesSwitch + label overlap (ours)+14.9 ± 2.20.9590.42 ± 0.04+15.7 ± 2.70.9800.58 ± 0.10
DescriptionDescription similarity+10.7 ± 3.00.9450.25 ± 0.07+4.4 ± 1.70.9060.06 ± 0.04
Label namesLabel overlap+9.5 ± 3.20.9390.19 ± 0.06+3.2 ± 1.30.9170.13 ± 0.06
LabeledSwitch + LESS-style (ours)+13.8 ± 2.10.9520.37 ± 0.03+15.7 ± 2.70.9800.58 ± 0.10
LabeledLESS-style+9.7 ± 2.70.9330.17 ± 0.05−3.3 ± 2.60.887−0.09 ± 0.05
LabeledLEEP+7.5 ± 2.60.9400.23 ± 0.06
LabeledOTDD+5.8 ± 2.60.9270.22 ± 0.04+1.3 ± 3.10.9130.14 ± 0.05

Records, with no target data, beat every other target-free rule (p<0.001p<0.001 for each) and seven of the eight rules that see target examples on the first pick: OTDD (p=0.008p=0.008), SEmb (p=0.013p=0.013), Grad Cos (p=0.034p=0.034), DSIR (p=0.011p=0.011), LogME (p=0.003p=0.003), LEEP (p=0.027p=0.027), and text similarity (p=0.006p=0.006). Their lead over LESS-style is not significant (p=0.22p=0.22). On weighted τ\tau, they beat every baseline (p≤0.027p\le0.027). The switch has the highest gain, NDCG, and weighted τ\tau and significantly beats every baseline on all three (p≤0.024p\le0.024), including LESS-style (p=0.019p=0.019), description similarity (p=0.024p=0.024), label overlap (p=0.012p=0.012), and OTDD (p<0.001p<0.001). With LESS-style as its fallback, it beats LESS-style alone by 4.1 points (p=0.023p=0.023). In math, the share is above the cutoff, so both switches follow records, which beat every baseline on NDCG and weighted τ\tau (p≤0.004p\le0.004) and tie perplexity on the first pick (Q1 of KFUw).

Across models (Fig. 2). On both benchmarks, records rank datasets better than every baseline on all six model–benchmark pairs whose share is clearly above the cutoff, with weighted τ\tau of 0.49 to 0.63 against at most 0.32 for the strongest baseline on each pair (p≤0.008p\le0.008). On the two text pairs with the lowest shares, LLaMA-2-7B (0.11) and Gemma-2-2B (0.23), a baseline ranks better (LogME 0.41, description similarity 0.33), which is where the switch falls back to label overlap. The strongest baseline changes from pair to pair (Grad Cos, V-usable information, perplexity, size), so no single proxy is reliable.

References

  • Vu et al. (2020). Exploring and Predicting Transferability across NLP Tasks.
  • Poth et al. (2021). What to Pre-Train On? Efficient Intermediate Task Selection.
  • You et al. (2021). LogME: Practical Assessment of Pre-Trained Models for Transfer Learning.
AuthorsOpenReview ↗

General response (2/3): New evidence by concern

New evidence, by concern.

  • Motivation and setting. The paper states the expert-library setting of Jang et al. (2023) and Ostapenko et al. (2024) and the data acquisition settings it serves (Sec. 1, Sec. 4).
  • Baselines. Sixteen baselines from data valuation, data selection, transferability estimation, and selection of intermediate tasks, in three groups by the target data they need, all compared with paired sign-flip tests over 15 held-out targets (Sec. 4, Table 2, App. B).
  • Related work. A data valuation section covering target-specific methods, target-free methods including the suggested V-usable information, data markets, and transfer studies that use pairwise transfer results (Zhou et al., 2023; Kim et al., 2023; Dymkiewicz et al., 2025) (Sec. 2).
  • Math fine-tuning library (new). Twenty-eight math datasets, including large uncurated and long-answer corpora, with 15 accuracy-scored targets, trained on Qwen3-1.7B (two seeds), Qwen3-0.6B, Qwen3-8B, and Gemma-2-2B. On every model, records rank datasets better than all 13 applicable baselines by NDCG and weighted τ, including the largest dataset and perplexity (Sec. 4.3, Table 2, Fig. 2, Table C.7).
  • Combining datasets (new). All 105 two-dataset mixtures on Qwen3-1.7B, used only to check predictions from single-dataset records with each target held out. General value picks good mixtures, whose top ten beat the bottom ten by 14.7 points (p = 0.0002), ahead of LESS-style and similarity to the target, and matched value flags redundant pairs that task labels miss (Sec. 4.3, App. C.9, Tables C.8–C.9, Fig. C.4).
  • Cost of records (new). A smaller pilot of the same family gives records as good as the target model's own (Qwen3-1.7B for Qwen3-8B on text and math, Qwen3-0.6B for Qwen3-1.7B on math). Runs stopped at a tenth of training give records as good as complete runs, and a new dataset needs only three past tasks (Sec. 4.5).
  • Scope checks. Unseen task families, seed noise in the residual, simpler uses of past results, and incomplete records (Sec. 4.3, App. C.5).
  • Models, seeds, regimes. A fourth decoder model (Gemma-2-2B), two extra Qwen3-1.7B seeds, a second Gemma-2-2B seed, and three learning rates (Secs. 4.3–4.4, Table 3, App. C.1).
  • Adaptive rule. The switch, with a cutoff set by leave-one-model-out validation and tested on six runs never used to set it (Sec. 3, Sec. 4.4, Fig. 3).
  • Statistics. Standard errors on every average and paired tests on every comparison (Tables 1–3, Tables C.1–C.9).
  • Model differences. The decomposition splits matched value into mutual and directional parts on every model, and the T5 analysis tests pilot transfer across architectures (Table 1, Fig. C.1, App. C.4, Fig. C.3).
  • Training budget. Value against dataset size, acquisition under a data budget, and the number of training examples each rule spends before it finds a helpful dataset (Sec. 4.5, Table C.1, Fig. C.2).

References

  • Jang et al. (2023). Exploring the Benefits of Training Expert Language Models over Instruction Tuning.
  • Ostapenko et al. (2024). Towards Modular LLMs by Building and Reusing a Library of LoRAs.
  • Zhou et al. (2023). Not All Tasks Are Born Equal: Understanding Zero-Shot Generalization.
  • Kim et al. (2023). TaskWeb: Selecting Better Source Tasks for Multi-Task NLP.
  • Dymkiewicz et al. (2025). Donors and Recipients: On Asymmetric Transfer across Tasks and Languages with Parameter-Efficient Fine-Tuning.
AuthorsOpenReview ↗

General response (1/3)

General response

We thank all three reviewers for careful and constructive reviews. Reviewer VxmR found the decomposition of a dataset's value into general and target-specific parts “reasonable and interesting” and the paper “organized well and easy to follow”. Reviewer KFUw found that “the prediction between sources is meaningful and can enhance efficiency when facing a large amount of data”. Reviewer 4hTa found that the paper “addresses a very interesting problem” with “an interesting formulation”. The main concerns were the motivation of the setting, missing baselines and related work, the number of models and seeds, gaps relative to standard error, an adaptive rule that also uses the residual, combining datasets, and the difference between model architectures. We addressed each one with new experiments, all reported with standard errors and paired tests, and a rewritten paper.

The question. Data valuation defines a dataset's value against a known target, distributional Shapley names this reliance on a known task as an outstanding limitation (Ghorbani et al., 2020), and data markets assume that a dataset's past success says little about a new buyer's task (Agarwal et al., 2019). Transfer studies disagree: some find sources that help most targets (Pruksachatkun et al., 2020; Jang et al., 2023), others find that the target matters most (Poth et al., 2021; Dymkiewicz et al., 2025). Whether a dataset's value carries over to new tasks has been assumed rather than measured, yet it decides whether data can be valued once, before its tasks are known, or must be valued again with target data for every new task. The paper measures it.

The message in three steps.

  1. Two parts of value. A dataset's value for a new task has a general part, which helps many tasks, and a matched part, which helps only related tasks. HodgeRank on within-target comparisons separates the two, and the source share measures their balance on each model (Sec. 3, Table 1).
  2. Records. The general part can be measured once from past transfer results, called records, and used with no data from the new task. On text, records beat every target-free valuation rule and seven of the eight rules that see target examples, all but LESS-style; on math, they rank datasets better than all 13 applicable baselines (Sec. 4.3, Table 2).
  3. The switch. The share tells when the general part is enough. The switch follows records when the share is high and otherwise falls back to a matched-value rule, label overlap or LESS-style. On text it is the best rule overall and significantly beats every baseline; on math the share is high, so it follows records (Sec. 4.4).

References

  • Ghorbani et al. (2020). A Distributional Framework for Data Valuation.
  • Agarwal et al. (2019). A Marketplace for Data: An Algorithmic Solution.
  • Pruksachatkun et al. (2020). Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
  • Jang et al. (2023). Exploring the Benefits of Training Expert Language Models over Instruction Tuning.
  • Poth et al. (2021). What to Pre-Train On? Efficient Intermediate Task Selection.
  • Dymkiewicz et al. (2025). Donors and Recipients: On Asymmetric Transfer across Tasks and Languages with Parameter-Efficient Fine-Tuning.
AuthorsOpenReview ↗

A kind reminder before the discussion closes

Dear Reviewers,

Thank you again for your time and careful reviews. The discussion closes on Nov 18 (AoE), so we would like to kindly follow up on our responses. We would be grateful to hear whether they address your concerns.

In brief, the revision adds:

  • Baselines and statistics. Sixteen baselines, with standard errors and paired tests on every comparison. With no target data, records gain 13.1 points over random, and the switch significantly beats every baseline (p ≤ 0.024).
  • Adaptive rule. The switch follows records when the source share is high and otherwise falls back to a matched-value rule. It has the lowest regret (10.4%).
  • Scope. A second benchmark of 28 math datasets on four models, where records rank datasets better than all 13 applicable baselines, plus Gemma-2-2B, extra seeds and three learning rates.
  • Framing. The paper is rewritten around one data valuation question with four stated contributions, a data valuation section is added, and every reference was checked.

Please let us know if anything remains unclear, and we will be glad to clarify.

Best regards,

Authors of Submission 37225

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.