- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors analyze the transferability between different training sources by separating generally useful and target-specific benefits. They use HodgeRank to decompose the influence to general source benefits and pair-specific asymmetry. They find that in decoder-only models (such as Qwen and LLaMA) the source-general benefits can explain most of transfer asymmetry. Based on that, they evaluate the strategy that selects data based on the general source ordering when obtaining a new training task. Empirically, they find that the strategy works well across different models. Besides, they also find that the quality of training sources evaluated by a small model can transfer to a larger model.
Critical strengths
- Data selection is a crucial problem in the training process of LLMs. The authors provide a new perspective that decomposes the advantages of training sources to generally useful and target specific benefits. The idea is reasonable and interesting.
- The empirical results are solid to verify the main insights in this paper. The experiments with different decoder-only models show that the generally useful benefits are more important than target-specific benefits.
- The paper is organized well and easy to follow. The logic flow is clear.
Critical weaknesses
- As shown in Table1, the residual component can not be ignored in the analysis. However, the strategy in the following section mainly select data based on source-general ranking. Is it possible to design adaptive strategies or coorprates two source better?
- The experiments in this paper seem to mainly focus on adapter training. However, I think it would be much better to verify the insights of data selection in more post-training paradigms including supervised fine-tuning with full parameters and reinforcement learning.
- The main comparisons in this paper are evaluated on methods based gradient similarity. Is it possible to provide comparisons with more baselines in the relation work section.
- I think more discussions on the comparison between decoder-only and encoder-decoder models are needed. The experiements show that there exist significant differences . However, it seems that the understanding of the difference is limited.
Questions for authors
See Weaknesses.
3 comments
Thank you for raising other post-training paradigms (2/3)
Combining two sources (continued response to W1). If the two sources are the two parts of value, the switch is our answer, as above. If they are two datasets trained together, the same records also guide the mixture, with no target data and no extra training (new Appendix C.9). We trained Qwen3-1.7B on all 105 pairs of the 15 datasets, each mixture taking half of each dataset, and used these mixtures only to check predictions made from single-dataset records of two other seeds. Each task was held out in turn and removed from the records before any prediction.
- General value picks good mixtures. Ranking pairs by the mean general value of their two datasets orders the mixtures correctly on all 15 held-out tasks. Its top ten pairs gain 8.5 points more than an average pair and 14.7 more than its bottom ten (14 of 15 tasks, p = 0.0002), ahead of LESS-style (+7.9) and similarity to the target (+6.4), which both see the held-out task (Table C.8).
- Matched value finds complementary and redundant pairs. Two datasets whose matched value falls on the same tasks are redundant, and two whose matched value falls on different tasks are complementary. The ten most complementary pairs beat the better of their two datasets by 5.0 points more than the ten most redundant ones (12 of 15 tasks, p = 0.002). Correlating raw gains instead gives 0.9 points, and task groups cannot tell whether a mixture beats its better dataset (Table C.9).
- The map goes beyond task labels. Records agree with task groups for Yelp with Amazon and DBpedia with Yahoo, and they also flag pairs across groups, such as QQP with MultiRC (redundant) and QQP with IMDB (complementary), which the trained mixtures confirm (Fig. C.4).
W2. “The experiments in this paper seem to mainly focus on adapter training. However, I think it would be much better to verify the insights of data selection in more post-training paradigms including supervised fine-tuning with full parameters and reinforcement learning.”
Response. Thank you for this suggestion. The method needs only the matrix of transfer results that training produces, so it applies unchanged to full fine-tuning and reinforcement learning, and the Conclusion names both as next steps. Within adapter training, the revision tests a change of training regime through the learning rate on Qwen3-1.7B (Table 3).
| Rate (seeds) | Share | Largest | Label overlap | Records | Switch | Oracle |
|---|---|---|---|---|---|---|
| 10⁻³ (3) | 40% | −9.5 ± 2.2 | +11.1 ± 3.4 | +12.8 ± 2.6 | +12.8 ± 2.6 | +20.9 ± 2.6 |
| 5 × 10⁻⁴ (1) | 35% | −1.5 ± 2.3 | +4.5 ± 3.9 | +4.7 ± 1.5 | +4.7 ± 1.5 | +17.3 ± 2.7 |
| 2 × 10⁻⁴ (2) | 15% | +1.0 ± 1.9 | +3.3 ± 2.9 | +3.0 ± 1.3 | +3.3 ± 2.9 | +13.5 ± 2.4 |
Records beat random at every rate (p ≤ 0.015). Gentler training gives target match more weight, the share falls to 14–16% at 2 × 10⁻⁴, below the cutoff, and the switch moves to label overlap, the best rule at that rate. The share thus tracks a change of training regime and selects the right rule for it, which is the property a new paradigm would need.
Evidence from prior work. The method needs a data ranking measured in one training setup to hold in another, and prior work gives direct evidence that such rankings transfer. LESS (Xia et al., 2024) selects data with the gradients of LLaMA-2-7B, and the selection still helps when training LLaMA-2-13B and Mistral-7B. RegMix (Liu et al., 2025) finds that the ranking of data mixtures from 1M-parameter proxy models holds for 1B-parameter models, and DoReMi (Xie et al., 2023a) tunes domain weights on a 280M-parameter proxy that improve an 8B model. Our results add two such changes: records from Qwen3-1.7B rank datasets for Qwen3-8B, and from Qwen3-0.6B for Qwen3-1.7B (Sec. 4.5), and the share tracks a change of learning rate (Table 3). None of these tests full fine-tuning or reinforcement learning directly, so we state them as next steps rather than as claims.
References
- Xia et al. (2024). LESS: Selecting Influential Data for Targeted Instruction Tuning.
- Liu et al. (2025). RegMix: Data Mixture as Regression for Language Model Pre-training.
- Xie et al. (2023a). DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining.
Thank you for your suggestions on baselines and model architectures (3/3)
W3. “The main comparisons in this paper are evaluated on methods based gradient similarity. Is it possible to provide comparisons with more baselines in the relation work section.”
Response. Thank you for this suggestion. The revision adds the methods from the related work as baselines: V-usable information, perplexity, IFD, Data-OOB, label overlap, description similarity, OTDD (the dataset distance behind LAVA), SEmb, LogME, LEEP, DSIR, and text similarity, alongside the gradient methods LESS-style and Grad Cos (Sec. 4, Table 2). Records beat every target-free rule (p < 0.001) and seven of the eight rules that see target examples, and the switch significantly beats all of them (p ≤ 0.024). The full numbers are in the headline table of the general response. We also compare records with simpler uses of the same past results, and ranking sources within each past target matters: averaging raw accuracy or win rates over past targets gains +10.0 and +8.7 points, below records at +13.1 (p = 0.024 and p < 0.001; App. C.5, Table C.4). The related work section adds data valuation, including Data Shapley (Ghorbani & Zou, 2019), influence functions (Koh & Liang, 2017), TracIn (Pruthi et al., 2020), Datamodels (Ilyas et al., 2022), LAVA (Just et al., 2023), OTDD (Alvarez-Melis & Fusi, 2020), Projektor (Kang et al., 2023, “Performance Scaling via Optimal Transport”, NeurIPS), and Data-OOB (Kwon & Zou, 2023) (Sec. 2). The same baselines, adapted to free-form answers, are compared on the new math library, where records rank datasets better than all 13 by NDCG and weighted τ (Table 2).
W4. “I think more discussions on the comparison between decoder-only and encoder-decoder models are needed. The experiements show that there exist significant differences . However, it seems that the understanding of the difference is limited.”
Response. Thank you for this point. The revision measures model differences with the decomposition (Table 1, as in KFUw Q3). The share varies among decoder models too, from 52.5% on Qwen3-1.7B to 10.4% on LLaMA-2-7B, and a low share comes with more mutual match. On LLaMA-2-7B, mutual match is 47.5% of ranking variation and general value only 10.4%. T5, the one encoder-decoder model, has a general ordering that differs from the decoder models, so pilot records do not carry over between T5 and them in either direction, while each model's own records still beat every baseline (App. C.4). The difference is thus one case of a measurable quantity. The share can be checked on any model before past records are trusted, and the switch uses label match where it is low.
References
- Ghorbani & Zou (2019). Data Shapley: Equitable Valuation of Data for Machine Learning.
- Koh & Liang (2017). Understanding Black-box Predictions via Influence Functions.
- Pruthi et al. (2020). Estimating Training Data Influence by Tracing Gradient Descent.
- Ilyas et al. (2022). Datamodels: Understanding Predictions with Data and Data with Predictions.
- Just et al. (2023). LAVA: Data Valuation without Pre-Specified Learning Algorithms.
- Alvarez-Melis & Fusi (2020). Geometric Dataset Distances via Optimal Transport.
- Kang et al. (2023). Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources.
- Kwon & Zou (2023). Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Value.
Thank you for suggesting an adaptive strategy using the residual (1/3)
Thank you for the positive assessment and the constructive suggestions. We are glad you found the decomposition “reasonable and interesting”, the empirical results “solid”, and the paper “organized well and easy to follow”. We answer each weakness (W1–W4) below. Your weaknesses were not numbered, so we numbered them in order, and your questions referred to them.
Weaknesses
W1. “As shown in Table1, the residual component can not be ignored in the analysis. However, the strategy in the following section mainly select data based on source-general ranking. Is it possible to design adaptive strategies or coorprates two source better?”
Response. Thank you for this excellent suggestion, which led to the switch, now the best rule in the paper.
Splitting the residual. The revision splits the residual into mutual match, where two tasks help each other alike, and directional match (Table 1). Mutual match is at least as large as directional match on every model, and it concentrates within task groups, so the target's label names capture it. Same-group pairs have higher mutual match than cross-group pairs on all four models (Fig. 3b). The residual is also mostly real rather than training noise. Between Qwen3-1.7B seeds, mutual match correlates at 0.59 (rate 10⁻³, three seeds) and 0.78 (rate 2 × 10⁻⁴, two seeds), and directional match at 0.41 and 0.77, all above every one of 2,000 task relabelings (p < 0.001; App. C.5, Table C.3).
The adaptive rule. The switch follows records when the source share of past tasks reaches a cutoff and otherwise falls back to a matched-value rule (Sec. 3, Alg. 2). The cutoff is set by leave-one-model-out validation (τ = 0.23–0.24). The fallback is pluggable, and the switch improves both fallbacks we test (Table 2).
The switch with label overlap beats label overlap alone (p = 0.012), and the switch with LESS-style beats LESS-style alone by 4.1 points (p = 0.023). Across ten runs over models, seeds, and learning rates, the higher the share, the more records beat label match (Spearman 0.68, p = 0.034). Below the cutoff label match beats records by 3.8 points, and above it records win by 4.1 (Fig. 3a). On the five runs with a share below the cutoff, the switch beats records by 4.2 points (p = 0.046). Any cutoff from 0.14 to 0.42 keeps the switch at or above records, and on six runs never used to set the cutoff it matches or beats records on five (Sec. 4.4).