- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies task-specific instruction data selection across different base models. Authors propose MARS, which uses a fixed external LLM annotator scores every example in the candidate pool once along 6 ability dimensions, together with a separate quality score. For each base model, response NLLs are computed on a small target reference set and converted into percentile ranks, which determine the per-bucket quotas, and the difficulty-weighted reference profiles. Candidates are assigned to ability buckets by their dominant ability and ranked within each bucket by the profile distance, with the quality score as the secondary key. Experiments cover 4 base models and 5 target tasks under a 5% selection budget.
Critical strengths
-
The problem setting is practically relevant. Since data selection outcomes are sensitive to the experimental setup, including the base model, repeating model-dependent selection for every new model is a real cost, and the division of labor between reusable candidate characterization and lightweight reference-based model calibration is easy to deploy.
-
The method is transparent and interpretable. Both the allocation and the profiles have closed forms.
-
The experiments span 4 model families under a unified training configuration, and the paper provides a selection-cost audit, the complete annotation prompt, and code.r
Critical weaknesses
-
The novelty is limited, and closely related work is neither discussed nor compared. The core idea of MARS has been explored extensively, e.g. [1-6]. Most of these methods require no target benchmark data, and [6] as well as CDT (cited only as a capability characterization framework), additionally support capability-targeted selection. None of [1–6] is discussed or used as a baseline. Besides, a control that uses the same annotator with simple scoring or matching (e.g., asking gpt-5.6-luna to rate task relevance and taking the top B) is also missing, so it cannot be ruled out that much of the gain stems from the knowledge of the strong external annotator rather than from the proposed calibration design.
-
Model adaptivity is the main claim of the paper, but the supporting evidence is weak and partly contradicts the motivation. While the MMLU shifts in Fig4 differ across models, the BBH and TyDiQA shifts are nearly identical across all four, suggesting that the NLL difficulty ranking is largely shared across models. The "w/o model calibration" variant in Table 8 uses no model information and therefore appears to yield a single subset shared by all four models, requiring one annotation pass and no GPU time; MARS improves over it by only 0.68 points on Gemma-3-4B and 1.30 points on OLMo-2-7B, without variance estimates.
Transfer of MARS's own subsets is evaluated only between Llama-3.1 and OLMo-2, whereas Table 2 compares MARS with recipient-specific calibration against LESS/NAIT subsets selected on Llama-3.1-8B, changing the method and the calibration model simultaneously. However, the baselines themselves do not support authors' premise: LESS subsets transferred from Llama-3.1-8B outperform LESS subsets selected natively on every recipient model, and the same holds for NAIT on OLMo-2-7B and Gemma-3-4B. This contradiction with the core motivation is not addressed.
-
The setting relies on a known target benchmark and gold-response references, and the ability dimensions appear entangled with the evaluated tasks. Although this setting has precedent, the limitation should be noted: The references come from the same benchmarks as the test sets, and open-ended tasks without gold responses are not supported.
-
Despite the literature and cognitive-science motivation, the 6 dimensions map almost one-to-one onto the 5 evaluated tasks (Semantic Retrieval - MMLU, Rule - BBH, Quantitative Reasoning - GSM8K, Code / Procedural Reasoning - MBPP, and Context Maintenance - TyDiQA). How the method performs on more numerous or more complex benchmarks that do not align with this taxonomy?
-
Statistical reliability and experimental scope are insufficient, and the comparison with base models is sidestepped. No training seeds or variance are reported, which is critical in data selection, where gains of 1.01–1.29 points over the strongest baseline may fall within run-to-run noise. Only a single 5% selection budget is evaluated, although SFT outcomes can differ substantially between low- and high-budget regimes, and the quota rounding and bucket-deficit filling in MARS may also behave differently across budgets. Furthermore, Qwen2.5-7B Base averages 70.96, higher than every fine-tuned result including MARS. Why does fine-tuning generally hurt in this case?
-
The efficiency advantage is a bit overstated. Full-pool computation is not eliminated but shifted to the external annotator. And the relationship between the 0.12-hour GPU-stage elapsed time and the 0.34 GPU-hours in Appendix B.5 should be clarified. Similar to W1, the cheapest model-dependent baseline is also missing, and the selection costs of MoNA and TSDS are not reported.
[1] DataMan: Data Manager for Pre-training Large Language Models.
[2] Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models.
[3] QuRating: Selecting High-Quality Data for Training Language Models.
[4] What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning.
[5] Improving Data Efficiency via Curating LLM-Driven Rating Systems.
[6] Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuning.
Questions for authors
Please see weaknesses.
16 comments
Same-Annotator and Capability-Targeted Comparisons (W1, Part 2/2)
3. Does the Strong External Annotator Explain the Gains?
To isolate the contribution of our selection design, we introduce three additional baselines:
- Quality Top-B: Selects candidates using quality scores from the same LLM annotator as MARS.
- Task Relevance Top-B: Uses the same annotator to score candidate relevance to the target task.
- CDT: Performs capability-targeted selection through cognition–domain–task label matching.
We also compare with Ability Distribution Matching and w/o Model Calibration, which reuse MARS's ability annotations without its complete model-calibrated selection mechanism. All methods use the same candidate pool, 5% budget, and downstream training configuration.
Table R1: Same-annotator and capability-targeted comparisons (5% budget). CDT and MARS multi-seed results are reported as mean ± sample SD over three matched training seeds (42, 43, 44). Direct-scoring baselines are single-run results.
(a) Llama-3.1-8B
| Method | MMLU | BBH | TyDiQA | GSM8K | MBPP | Avg. |
|---|---|---|---|---|---|---|
| Quality Top-B | 61.04 | 61.20 | 72.47 | 53.37 | 43.60 | 58.34 |
| Task Relevance Top-B | 64.41 | 63.52 | 72.46 | 63.61 | 48.60 | 62.52 |
| CDT (3 seeds) | 62.87 ± 0.79 | 63.27 ± 0.56 | 71.66 ± 0.34 | 64.34 ± 0.31 | 47.80 ± 0.53 | 61.99 ± 0.15 |
| MARS (seed 42) | 64.83 | 64.72 | 73.20 | 66.11 | 49.60 | 63.69 |
| MARS (3 seeds) | 64.72 ± 0.17 | 64.01 ± 1.39 | 72.76 ± 0.38 | 65.53 ± 0.50 | 48.40 ± 1.04 | 63.09 ± 0.57 |
(b) Gemma-3-4B-PT
| Method | MMLU | BBH | TyDiQA | GSM8K | MBPP | Avg. |
|---|---|---|---|---|---|---|
| Quality Top-B | 57.16 | 53.52 | 71.32 | 44.43 | 24.00 | 50.09 |
| Task Relevance Top-B | 57.96 | 55.93 | 73.82 | 50.49 | 44.60 | 56.56 |
| CDT (3 seeds) | 56.91 ± 0.66 | 56.05 ± 0.54 | 73.96 ± 0.46 | 49.91 ± 0.31 | 45.87 ± 0.46 | 56.54 ± 0.19 |
| MARS (seed 42) | 58.43 | 56.30 | 74.44 | 51.86 | 46.40 | 57.49 |
| MARS (3 seeds) | 58.32 ± 0.30 | 56.91 ± 0.60 | 74.74 ± 0.49 | 52.14 ± 0.83 | 46.47 ± 0.31 | 57.72 ± 0.27 |
Direct-scoring comparisons. Task Relevance Top-B substantially outperforms Quality Top-B, confirming that task relevance is a meaningful selection signal. Nevertheless, under the original single-run setting, MARS further improves over Task Relevance Top-B by 1.17 and 0.93 points on Llama and Gemma, respectively, outperforming it on all ten individual task comparisons.
Across all four evaluated models, MARS also exceeds Quality Top-B in all 20 model–task settings.
Ability-matching controls. Ability Distribution Matching achieves 59.79 and 55.87 on Llama and Gemma, respectively; w/o Model Calibration achieves 60.11 and 56.81. MARS reaches 63.69 and 57.49 under the same single-run evaluation, showing that ability annotations alone do not reproduce the full selection gains.
Multi-seed CDT comparison. Across three matched training seeds, MARS improves the five-task average from 61.99 ± 0.15 to 63.09 ± 0.57 on Llama (+1.10) and from 56.54 ± 0.19 to 57.72 ± 0.27 on Gemma (+1.18). All six model–seed comparisons show positive average gains.
Together, these results establish that MARS provides additional value beyond strong annotator knowledge, direct task relevance, and existing capability matching. The gains support the contribution of reference-calibrated ability allocation and instance-level matching to effective data selection.
Recipient-Specific Calibration Beyond Direct Subset Transfer (W2, Part 1/2)
We thank the reviewer for highlighting the strong cross-model transfer results. We agree that these findings deserve closer examination and provide an important baseline for evaluating model adaptivity.
1. Why Is Recipient-Specific Calibration Valuable When Direct Transfer Works?
As the reviewer observes, LESS subsets selected using Llama transfer better than their recipient-specific counterparts on all three recipient models, while NAIT shows similar behavior on OLMo and Gemma. These findings establish that recomputing selection signals for each model does not automatically improve performance. They also highlight the importance of comparing against strong transferred subsets.
This motivates a more precise question for MARS:
Given that a fixed subset can already transfer effectively, can lightweight recipient-model feedback further improve selection while preserving reusable candidate representations?
To answer this, we extend the original Llama–OLMo experiment (Appendix C.2) to all three recipient models. We hold the Candidate Ability Atlas, target reference sets, selection algorithm, data budget, and recipient training configurations fixed, changing only which model provides the reference NLLs.
Table R2: Controlled comparison of calibration sources (5% budget; single-run results).
| Recipient Model | Calibration Source | BBH | TyDiQA | Five-Task Avg. |
|---|---|---|---|---|
| Qwen2.5-7B | Llama | 63.89 | 71.08 | 69.14 |
| Recipient | 66.11 | 72.22 | 70.64 | |
| OLMo-2-7B | Llama | 48.61 | 63.47 | 52.85 |
| Recipient | 49.44 | 65.92 | 54.50 | |
| Gemma-3-4B-PT | Llama | 55.09 | 71.59 | 54.94 |
| Recipient | 56.30 | 74.44 | 57.49 |
Recipient-specific calibration improves the five-task average by +1.50, +1.65, and +2.55 points on Qwen, OLMo, and Gemma, respectively, with improvements in 14 of the 15 individual task comparisons.
The comparison also clarifies the relationship with strong transfer baselines. With Llama-based calibration, MARS scores 69.14, 52.85, and 54.94 on the three recipients, below the strongest LESS/NAIT transfer results of 69.50, 53.52, and 55.51. After recipient-specific calibration, MARS achieves 70.64, 54.50, and 57.49, exceeding the strongest transferred baseline by +1.14, +0.98, and +1.98 points, respectively.
This establishes the incremental value of recipient feedback under a controlled selection procedure, even when direct subset transfer is already effective.
2. Why Are the Allocation Shifts Similar in Figure 4?
We agree that Figure 4 shows similar ability-allocation patterns across models on BBH and TyDiQA. Such patterns are consistent with shared task-level ability demands.
Importantly, Figure 4 visualizes allocation proportions, while the complete MARS calibration also adjusts difficulty-weighted reference profiles within each ability bucket. Similar aggregate allocations can therefore lead to different within-bucket rankings and selected subsets.
The controlled results in Table R2 directly demonstrate the value of recipient-specific calibration on these two tasks: all six BBH and TyDiQA comparisons improve, with gains ranging from +0.83 to +2.85 points, despite their similar allocation patterns.
These findings show that shared task-level ability demands and recipient-specific selection gains can coexist. MARS combines global ability allocation with local reference-profile matching, allowing reusable candidate representations to respond effectively to each recipient model's reference difficulty information.
We will clarify this distinction in the discussion of Figure 4 and include the full calibration-source comparison in the revised manuscript.
Novelty and Relation to Prior Work (W1, Part 1/2)
We thank the reviewer for recognizing the practical relevance, transparency, and reproducibility of MARS. Your comments have helped us sharpen our methodological contributions and motivated additional controlled experiments.
1. Research Motivation and Key Contributions
We acknowledge the important foundations established by prior work on multidimensional data evaluation and capability-guided selection. We will expand Section 2.2 to discuss these studies and clarify the methodological distinctions.
MARS investigates a distinct question: how can a reusable characterization of instruction data support effective selection for different recipient models using only lightweight model-specific feedback?
Our central insight is that model-specific data selection does not necessarily require model-specific candidate representations. Model dependence can instead be introduced when deciding how to prioritize reusable candidate information.
MARS realizes this insight through two complementary mechanisms:
These mechanisms introduce recipient feedback at both the allocation and instance-matching levels, allowing different models to select different subsets from the same candidate representation.
The key methodological contribution is decoupling reusable candidate characterization from recipient-specific selection decisions. The Candidate Ability Atlas is constructed once and reused across target tasks and models. Adaptation to a new recipient requires reference-set NLL evaluation, without repeating model-dependent feature extraction over the entire candidate pool.
This provides a concrete way to combine cross-model reusability, model-specific adaptation, and low selection overhead within a unified framework.
2. Distinction from Existing Capability-Guided Selection
Capability-based descriptions alone do not determine how to allocate a fixed data budget for a particular recipient model, nor which candidates within an ability group are most appropriate for that model.
MARS addresses both decisions through reference-calibrated allocation and continuous profile matching. Its contribution therefore extends beyond assigning capability labels or matching task categories: it provides a model-conditioned decision mechanism operating on reusable functional representations.
We further test this distinction against direct scoring, ability distribution matching, and CDT using controlled comparisons, reported in the following comment.
References: [1] DataMan (ICLR 2025); [2] Meta-rater (ACL 2025); [3] QuRating (ICML 2024); [4] DEITA (ICLR 2024); [5] DS² (ICLR 2025); [6] mmSSR (2025); [7] CDT (Findings of EMNLP 2025).