- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents D2S, a data selection framework that simultaneously considers static data-centric scoring and dynamic model-centric scoring. Specifically, it precomputed multiple static metrics (quality, difficulty, and diversity) and adaptively weights them with a reuse penalty to shortlist a candidate set. The candidate set is then re-ranked using model-aware scoring, which first distinguishes important and unimportant tokens and then scores them separately. Top-rank candidates are then selected to fine-tune the model for one epoch. The MaS rankings are also used update category allocation budgets and to re-optimize data-perspective attribute weights for subsequent rounds. The quantitative evaluation reports performance improvements and runtime savings over full-data training and existing data selection baselines.
Critical strengths
- The motivation is clear, and the paper is overall easy to read.
- The evaluation is extensive, covering different instruction-tuning pools, base models, downstream tasks, and baseline methods.
Critical weaknesses
-
The paper frames the combination of static data-centric scoring and dynamic model-centric scoring as an unexplored area in instruction tuning. However, this hybrid philosophy has been explored in several work, including MoDS [1], LEAD [2], and UDS [3]. The paper fails to distinguish itself from these work in both concept and evaluation.
-
Without the dual-perspective framing, the contributions narrow down to a token-partitioning heuristic and a Spearman-based weight grid search. However, the paper's own ablations in Table 4 show that the closed-loop feedback provides only a marginal +0.9 point gain, and a simple iterative random-candidate control (
w/ random selection) already reaches 55.9. The feedback mechanism is interesting but underexplored. -
The evaluation is confounded by different training strategy. D2S uses an iterative, multi-round loop, while all baseline methods uses a static single-stage configuration. The ablation study in Table 4 reveals that an iterative control (i.e., w/ random selection) already reaches an average score of 55.9 on Alpaca-GPT4 with LLaMA-3.1-8B, outperforming every external baseline in Table 1. Therefore, the observed performance advantage may be driven by the iterative multi-round training paradigm and schedule resets rather than the proposed dual-perspective selection algorithm.
[1] Qianlong Du, Chengqing Zong, and Jiajun Zhang. 2023. MoDS: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311.15653.
[2] Xiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, and Yuyu Luo. 2026. LEAD: Iterative data selection for efficient LLM instruction tuning. Proceedings of the VLDB Endowment (PVLDB), 19(3): 426–439.
[3] Heming Zou, Yixiu Mao, Yun Qu, Qi Wang, and Xiangyang Ji. 2026. Utility-diversity aware online batch selection for LLM supervised fine-tuning. In Proceedings of the 43rd International Conference on Machine Learning (ICML).
Questions for authors
-
If strong dynamic baselines (e.g., UDS) and a random selection baseline are trained using the exact same multi-round training schedule as D2S, how much of the performance advantage of D2S remains?
-
How does D2S conceptually and empirically differ from work such as MoDS, LEAD, and UDS?
6 comments
Author Response (2/3)
Response to W3 and Q1: If strong dynamic baselines (e.g., UDS) and a random selection baseline are trained using the exact same multi-round training schedule as D2S, how much of the performance advantage of D2S remains?
Thank you for pointing this out. D²S outperforms strong baselines and pure random selection using the same multi-round training schedule. In the following content, we will clarify the experimental settings and present comparisons with other iterative selection methods and pure random selection.
Clarification of UDS’s iterative selection procedure.
It is worth clarifying that UDS and D²S adopt fundamentally different iterative frameworks. UDS selects a small subset from each candidate batch (e.g., 4 samples) and repeats this process thousands of times. In contrast, D²S selects 3,000 training examples per round and trains for only a few rounds. This substantial difference in selection granularity and iteration frequency makes it difficult to directly adapt UDS to D²S's multi-round training schedule.
Clarification of the “w/ random selection” ablation.
The “w/ random selection” ablation in Table 4 only ablates the data perspective candidate selection, while the model perspective still keeps the MaS scores to select the final training samples. The “w/ random selection” ablation therefore includes model-aware selection and should not be interpreted as the effect of multi-round training alone.
Comparison with selection methods under multi-round training.
We reproduced UDS following its official implementation of iterative data selection and reported the results in Tables 1, 2, and 7 of the original paper. To extend our comparison to more multi-round methods, we compared D²S with the strongest dynamic baseline LEAD [2], using the same multi-round training schedule for both methods.
As shown in Table 2, D²S achieves an average score of 57.5, outperforming LEAD (54.3) and UDS (54.5) by 3.2 and 3.0 points, respectively. These results suggest that D²S’s gains come not only from multi-round training but also from dual-perspective data selection.
Table 2: Comparison with LEAD and UDS using LLaMA-3.1-8B on Alpaca-GPT4
| Method | BBH | GSM8K | MMLU | TQA | HumanEval | MBPP | Avg. |
|---|---|---|---|---|---|---|---|
| LEAD | 50.2 | 64.4 | 46.0 | 54.3 | |||
| UDS | 62.5 | 57.6 | 43.9 | ||||
| D²S | 66.4 | 58.6 | 65.5 | 58.2 | 45.1 | 51.0 | 57.5 |
Comparison with pure random selection.
To further examine whether multi-round training alone accounts for the gains, we additionally conducted an experiment that selects the final training samples entirely at random in each round. As shown in Table 3, this variant achieves an average score of 52.9, compared with 57.5 for D²S, a difference of 4.6 points. This result further supports the contribution of our dual-perspective data selection framework.
Table 3: Comparison between pure random multi-round training and D²S
| Method | BBH | GSM8K | MMLU | TQA | HumanEval | MBPP | Avg. |
|---|---|---|---|---|---|---|---|
| Pure random | |||||||
| D²S | 66.4 | 58.6 | 65.5 | 58.2 | 45.1 | 51.0 | 57.5 |
Author Response (3/3)
Response to W2: The variability of feedback mechanism.
Thank you for your suggestion. To assess the variability of D²S and the no-feedback variant, we conducted three independent training runs with different random seeds using LLaMA-3.1-8B + Alpaca-GPT4.
As shown in Table 4, the full D²S method achieves higher mean performance on all six benchmarks, with an average improvement of 1.2 ± 0.3 points, showing that the feedback mechanism provides a consistent performance advantage.
Table 4: Comparison between the full D²S method and the variant without feedback
| Method | BBH | GSM8K | MMLU | TruthfulQA | HumanEval | MBPP | Avg. |
|---|---|---|---|---|---|---|---|
| w/o feedback | |||||||
| D²S | 66.5 ± 0.1 | 58.0 ± 0.6 | 65.8 ± 0.3 | 58.6 ± 0.4 | 45.5 ± 1.3 | 50.1 ± 1.0 | 57.4 ± 0.1 |
| Paired improvement | +1.8 ± 0.7 | +2.3 ± 0.8 | +0.3 ± 0.2 | +1.8 ± 1.0 | +0.6 ± 1.6 | +0.7 ± 0.1 | +1.2 ± 0.3 |
Thank you again for your valuable suggestions for improving the paper. We hope the explanations and additional experiments above address your concerns.
References
[1] Qianlong Du, Chengqing Zong, and Jiajun Zhang. 2023. MoDS: Model-oriented Data Selection for Instruction Tuning. arXiv preprint arXiv:2311.15653.
[2] Xiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, and Yuyu Luo. 2025. LEAD: iterative data selection for efficient LLM instruction tuning. Proceedings of the VLDB Endowment.
[3] Heming Zou, Yixiu Mao, Yun Qu, Qi Wang, and Xiangyang Ji. 2026. Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning. Forty-third International Conference on Machine Learning.
[4] Yanjun Fu, Faisal Hamman, and Sanghamitra Dutta. 2025. T-SHIRT: Token-selective hierarchical data selection for instruction tuning. Advances in Neural Information Processing Systems.
Author Response (1/3)
Thank you for carefully reviewing our paper and for your valuable suggestions. In response to your concerns, we provide the following replies. The corresponding revisions are highlighted in blue in the revised paper.
Response to W1 and Q2: How does D2S conceptually and empirically differ from work such as MoDS, LEAD, and UDS?
Thank you for your comment. The novelty of D²S lies in the following three aspects:
Although MoDS, LEAD, and UDS incorporate certain data-related or model-related signals, they do not systematically model the data and model perspectives or explicitly capture their interactions as D²S does. Specifically, the differences are as follows.
Specific differences from MoDS, LEAD, and UDS.
Empirical comparison with MoDS, LEAD, and UDS.
In the original paper, we compared D²S with UDS. We have extended this comparison to include MoDS and LEAD, using LLaMA-3.1-8B on three datasets under the same experimental setting. As shown in Table 1, D²S achieves the highest average on all three datasets, with average improvements of 2.8, 3.1, and 3.2 points over MoDS, LEAD, and UDS, respectively. These results provide empirical evidence of the effectiveness of D²S, demonstrating the benefits of systematically modeling and dynamically coordinating the data and model perspectives for SFT data selection.
Table 1: Comparison with related methods using LLaMA-3.1-8B on Alpaca-GPT4, WizardLM, and CoT