- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes MoReC, a data selection method for targeted instruction tuning that combines gradient-based relevance with diversity. It uses K-means to identify target modes and a facility-location objective to select a representative subset. Experiments on Llama-3.1-8B and Mistral-7B across five benchmarks evaluate its effectiveness against existing data selection methods, with additional ablation studies.
Critical strengths
Combining target-aware retrieval with facility-location selection is intuitive and addresses a real limitation of relevance-only ranking: selecting many similar training examples. Table 2 shows that MoReC preserves most of LESS's target relevance while reducing intra-subset similarity by 25.3%. This supports the practical value of the method, although it does not directly establish better capability coverage.
Critical weaknesses
-
The paper calls K-means clusters of validation gradients "latent capability modes," but provides little evidence that these clusters correspond to meaningful capabilities rather than similar topics, answer formats, or optimization signals. The examples in Figure 3 are not particularly convincing, and there is no quantitative validation. More importantly, the final facility-location objective optimizes global relevance and coverage of the candidate pool, without requiring each target mode to be represented in the selected subset. Mode-wise retrieval only ensures that different modes contribute candidates, not that they are actually covered after selection. The reported reduction in cosine similarity and the t-SNE visualizations do not resolve this issue. As a result, the experiments support reduced redundancy, but not the paper's stronger claim that MoReC discovers and covers diverse target capabilities. This weakens the main motivation and interpretation of the method.
-
The method mainly combines existing gradient representations, K-means clustering, and facility location. The submodularity proof and greedy approximation guarantee in Appendix A are standard results rather than new theoretical contributions. The positioning against LESS is also concerning. The paper repeatedly suggests that prior methods represent the target set using a single averaged gradient, but Section 4.2 of LESS already computes separate average gradients for different validation subtasks and uses the maximum influence across them. Although MoReC automatically discovers clusters and adds set-level selection, its distinction from LESS is narrower than the paper suggests. It is also unclear whether the LESS baseline faithfully implements this subtask-wise strategy. Since the contribution is already fairly incremental, accurate positioning and a fair comparison with LESS are important to my assessment.
Questions for authors
-
Did you implement the subtask-wise influence aggregation described in Section 4.2 of the original LESS paper, including taking the maximum influence across validation subtasks? If not, why was this omitted? Would MoReC still outperform LESS when using its original aggregation strategy, especially on benchmarks with natural subtask divisions such as MMLU and BBH?
-
What evidence shows that the discovered gradient clusters represent distinct target capabilities rather than topic or answer-format similarities? Can you quantitatively measure how well the final selected subset covers each discovered mode, rather than only reporting global pairwise similarity? Since the final objective has no explicit mode-coverage constraint, how do you establish that better capability coverage is responsible for the downstream improvements?
-
I attempted to reproduce the data selection and LoRA fine-tuning pipeline based on the paper's description, but obtained results substantially different from Table 1. Could you provide the complete training and evaluation configurations, selected data indices, and reference implementation?
In particular, how were the MBPP pass@1 scores of 53.64 and 44.01 obtained? With the standard 500-problem test set and one deterministic generation per problem, pass@1 scores should be multiples of 0.2 percentage points. Please clarify the exact evaluation set size, number of generations, decoding settings, and whether scores were averaged across multiple runs.
- How sensitive are the results to the number of clusters K, the candidate-pool multiplier , and clustering random seeds? Why is K=8 used for eve.
4 comments
Response to Reviewer 42b8 (Part 2/3)
W1 / Q2. Interpretation of target modes and representation in the final subset
We thank the reviewer for raising the important question of how target modes should be interpreted and represented in the final subset. MoReC adopts a training-oriented perspective on target capabilities: different validation examples may provide heterogeneous optimization signals, even within the same benchmark. By clustering optimizer-aware gradient representations, MoReC identifies groups of examples with similar adaptation directions, which we refer to as latent target capability modes. These modes capture model-dependent patterns of target requirements rather than relying on predefined subtask labels. We further examine how these discovered modes are represented in the final selected subsets through the quantitative analysis below.
Importantly, mode-wise retrieval ensures that each discovered mode contributes candidates, but the final facility-location objective has no explicit per-mode quota or guarantee. To examine the final selected subsets, we perform the same post-hoc assignment for LESS and MoReC. For each selected training example , we compute its mean non-negative gradient similarity to queries in mode and assign it to the mode with the highest affinity: We reconstruct the eight target modes using the saved validation gradients and original clustering seed. The table reports mode-affinity counts in the final 5% subsets (12,142 examples each), not original candidate-retrieval labels:
| Benchmark | Method | M0 | M1 | M2 | M3 | M4 | M5 | M6 | M7 |
|---|---|---|---|---|---|---|---|---|---|
| ARC-C | LESS | 3761 | 958 | 89 | 1653 | 1148 | 342 | 1705 | 2486 |
| ARC-C | MoReC | 3435 | 1015 | 139 | 1781 | 1243 | 403 | 1651 | 2475 |
| BBH | LESS | 1638 | 2598 | 1169 | 228 | 1981 | 1103 | 3313 | 112 |
| BBH | MoReC | 1698 | 2387 | 1498 | 255 | 1802 | 1087 | 3201 | 214 |
| GSM8K | LESS | 933 | 6107 | 977 | 1684 | 592 | 982 | 481 | 386 |
| GSM8K | MoReC | 996 | 5677 | 808 | 1364 | 489 | 870 | 1245 | 693 |
| MMLU | LESS | 910 | 1856 | 1990 | 883 | 1558 | 1402 | 2772 | 771 |
| MMLU | MoReC | 717 | 1352 | 1712 | 850 | 1554 | 1648 | 3248 | 1061 |
| MBPP | LESS | 1655 | 1031 | 5323 | 567 | 1884 | 730 | 78 | 874 |
| MBPP | MoReC | 1854 | 1200 | 3984 | 704 | 2020 | 1191 | 128 | 1061 |
We summarize these distributions using normalized Shannon entropy (larger means less concentrated) and total variation distance (TVD):
| Benchmark | Entropy (LESS / MoReC) | TVD |
|---|---|---|
| ARC-C | 0.863 / 0.884 | 3.22% |
| BBH | 0.871 / 0.893 | 4.27% |
| GSM8K | 0.773 / 0.818 | 9.34% |
| MMLU | 0.959 / 0.949 | 8.33% |
| MBPP | 0.801 / 0.882 | 11.03% |
Both methods have examples assigned to all eight modes, but their distributions differ. MoReC exhibits higher normalized entropy on four of the five benchmarks; for MBPP, the largest mode's share falls from 43.84% to 32.81%. MMLU is an exception. The ablation results (55.79 overall vs. 53.27 without mode discovery and 54.05 without set-level coverage) support the usefulness of both components.
Response to Reviewer 42b8 (Part 3/3)
Q3. Reproducibility and MBPP evaluation
We thank the reviewer for pointing out the unusual numerical precision in the reported MBPP results. We re-evaluated MBPP using the same evaluation protocol (3-shot, pass@1, 500 test examples). The corrected MoReC scores are 53.60% for Llama-3.1-8B and 44.00% for Mistral-7B. We have updated the corresponding results in the revised manuscript to ensure accurate reporting.
Regarding reproducibility, we provide the dataset construction procedures, LoRA training hyperparameters, gradient computation details, selection configurations, and evaluation protocols in the manuscript and appendix.
Q4. Sensitivity to the number of target modes and other parameters
We thank the reviewer for this suggestion. We conducted an additional sensitivity experiment on GSM8K using Llama-3.1-8B, varying the number of target modes while keeping other settings fixed.
| Number of modes | 4 | 8 | 16 | 32 | 64 |
|---|---|---|---|---|---|
| GSM8K accuracy (%) | 56.10 | 58.45 | 56.63 | 56.33 | 59.82 |
We use as a unified default across all benchmarks, rather than tuning it separately for each task. This choice is motivated by the relatively small target validation sets, where excessively large values may fragment the target examples into very small clusters.
The results show that performance varies non-monotonically with . Although outperforms several alternative settings, achieves the highest accuracy of 59.82%, exceeding the default setting by 1.37 points. This suggests that adapting the number of target modes to individual benchmarks may further improve performance, while our main experiments deliberately use a fixed configuration across tasks.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, particularly the additional comparison with the original LESS aggregation strategy and the analysis of selected mode distributions. However, my main concerns remain. The method still largely combines existing gradient representations, K-means clustering, and facility-location optimization, with limited technical novelty. More importantly, the new mode-distribution analysis does not establish that the discovered gradient clusters correspond to meaningful target capabilities. Higher entropy within the same gradient space used for selection does not necessarily demonstrate improved capability coverage. The connection between the proposed mechanism and downstream improvements therefore remains insufficiently validated.
I also have some remaining concerns about experimental reliability and reproducibility. I appreciate the correction of the MBPP scores, but noticed a few additional inconsistencies worth clarifying. For example, the MBPP mean relevance values change from 0.0862/0.0859 (LESS/MoReC) in the original paper to 0.0228/0.0220 in the rebuttal, without an explanation for this substantial difference. Also, the newly reported 1B results include an ARC-C accuracy of 40.59%, which does not correspond to an integer number of correct predictions on the stated 1,172-example test set under a single-run evaluation. Clarifying whether these scores involve averaging across runs or different evaluation procedures would be helpful. In addition, the new sensitivity results show noticeable performance variation across different values of K, while multi-seed statistics and reference code remain unavailable. Overall, the rebuttal addresses several specific issues, but I believe the central concerns about novelty, capability coverage, and experimental reliability remain.
Response to Reviewer 42b8 (Part 1/3)
W2 / Q1. LESS subtask-wise aggregation and technical novelty
We thank the reviewer for pointing out Section 4.2 of LESS. We agree that LESS supports subtask-wise influence aggregation, whereas our original experiments used global aggregation for all five benchmarks. To address this concern, we additionally implemented the subtask-wise strategy: we average validation gradients within each predefined subtask and rank training examples by their maximum influence across subtasks. We evaluate this variant on MMLU and BBH, which have natural subtask divisions, for both backbones under the same 5% selection budget.
Subtask-wise aggregation improves LESS on Llama-3.1-8B BBH (39.46 to 40.62), but does not consistently improve performance. MoReC still outperforms subtask-wise LESS on all four comparisons, by 0.22/2.14 points on Llama-3.1-8B and 0.80/2.47 points on Mistral-7B (MMLU/BBH).
While LESS already accounts for predefined validation subtasks, MoReC automatically discovers target-gradient modes without subtask labels and combines mode-aware candidate retrieval with set-level coverage optimization, rather than relying solely on per-example influence ranking. We will clarify this distinction and include the additional results in the revised manuscript.