acceptodds
Under review as a conference paper at ICLR 2027

MoReC: Mode-aware Relevance-Gated Coresets for Targeted Instruction Tuning

Abstract

Instruction tuning is widely used to adapt LLMs to specialized tasks, yet its effectiveness is often constrained by the limited availability of high-quality target-specific data. To alleviate this limitation, existing data selection methods identify, from general instruction corpora, examples whose learning effects align with the target task. However, they often treat the target set as homogeneous and evaluate training examples independently, which may obscure fine-grained target capability requirements and produce redundant subsets. We propose MoReC, a Mode-aware Relevance-gated Coreset selection framework. Built upon optimizer-aware gradient representations, MoReC first discovers latent capability modes in the target validation set and retrieves relevant training examples for each mode, constructing a candidate pool that preserves diverse target requirements. It then performs set-level selection through a facility-location objective that jointly optimizes target relevance and coverage. The overall objective is monotone submodular and admits an efficient greedy algorithm with a approximation guarantee. Experiments on benchmarks spanning knowledge, reasoning, and code generation show that, using only 5% of the instruction data, MoReC achieves competitive or state-of-the-art performance, while yielding subsets with lower intra-subset similarity. These results indicate that covering the target task's diverse capability modes is more effective than concentrating the budget on top-ranked examples.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
large language models, instruction tuning, data selection, coreset selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_42b8OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MoReC, a data selection method for targeted instruction tuning that combines gradient-based relevance with diversity. It uses K-means to identify target modes and a facility-location objective to select a representative subset. Experiments on Llama-3.1-8B and Mistral-7B across five benchmarks evaluate its effectiveness against existing data selection methods, with additional ablation studies.

Critical strengths

Combining target-aware retrieval with facility-location selection is intuitive and addresses a real limitation of relevance-only ranking: selecting many similar training examples. Table 2 shows that MoReC preserves most of LESS's target relevance while reducing intra-subset similarity by 25.3%. This supports the practical value of the method, although it does not directly establish better capability coverage.

Critical weaknesses
  1. The paper calls K-means clusters of validation gradients "latent capability modes," but provides little evidence that these clusters correspond to meaningful capabilities rather than similar topics, answer formats, or optimization signals. The examples in Figure 3 are not particularly convincing, and there is no quantitative validation. More importantly, the final facility-location objective optimizes global relevance and coverage of the candidate pool, without requiring each target mode to be represented in the selected subset. Mode-wise retrieval only ensures that different modes contribute candidates, not that they are actually covered after selection. The reported reduction in cosine similarity and the t-SNE visualizations do not resolve this issue. As a result, the experiments support reduced redundancy, but not the paper's stronger claim that MoReC discovers and covers diverse target capabilities. This weakens the main motivation and interpretation of the method.

  2. The method mainly combines existing gradient representations, K-means clustering, and facility location. The submodularity proof and greedy approximation guarantee in Appendix A are standard results rather than new theoretical contributions. The positioning against LESS is also concerning. The paper repeatedly suggests that prior methods represent the target set using a single averaged gradient, but Section 4.2 of LESS already computes separate average gradients for different validation subtasks and uses the maximum influence across them. Although MoReC automatically discovers clusters and adds set-level selection, its distinction from LESS is narrower than the paper suggests. It is also unclear whether the LESS baseline faithfully implements this subtask-wise strategy. Since the contribution is already fairly incremental, accurate positioning and a fair comparison with LESS are important to my assessment.

Questions for authors
  1. Did you implement the subtask-wise influence aggregation described in Section 4.2 of the original LESS paper, including taking the maximum influence across validation subtasks? If not, why was this omitted? Would MoReC still outperform LESS when using its original aggregation strategy, especially on benchmarks with natural subtask divisions such as MMLU and BBH?

  2. What evidence shows that the discovered gradient clusters represent distinct target capabilities rather than topic or answer-format similarities? Can you quantitatively measure how well the final selected subset covers each discovered mode, rather than only reporting global pairwise similarity? Since the final objective has no explicit mode-coverage constraint, how do you establish that better capability coverage is responsible for the downstream improvements?

  3. I attempted to reproduce the data selection and LoRA fine-tuning pipeline based on the paper's description, but obtained results substantially different from Table 1. Could you provide the complete training and evaluation configurations, selected data indices, and reference implementation?

In particular, how were the MBPP pass@1 scores of 53.64 and 44.01 obtained? With the standard 500-problem test set and one deterministic generation per problem, pass@1 scores should be multiples of 0.2 percentage points. Please clarify the exact evaluation set size, number of generations, decoding settings, and whether scores were averaged across multiple runs.

  1. How sensitive are the results to the number of clusters K, the candidate-pool multiplier β\beta, and clustering random seeds? Why is K=8 used for eve.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 42b8 (Part 1/3)

W2 / Q1. LESS subtask-wise aggregation and technical novelty

We thank the reviewer for pointing out Section 4.2 of LESS. We agree that LESS supports subtask-wise influence aggregation, whereas our original experiments used global aggregation for all five benchmarks. To address this concern, we additionally implemented the subtask-wise strategy: we average validation gradients within each predefined subtask and rank training examples by their maximum influence across subtasks. We evaluate this variant on MMLU and BBH, which have natural subtask divisions, for both backbones under the same 5% selection budget.

ModelBenchmarkLESS (Global)LESS (Subtask)MoReC
Llama-3.1-8BMMLU64.4964.4564.67
Llama-3.1-8BBBH39.4640.6242.76
Mistral-7BMMLU61.3460.4861.28
Mistral-7BBBH35.0234.4236.89

Subtask-wise aggregation improves LESS on Llama-3.1-8B BBH (39.46 to 40.62), but does not consistently improve performance. MoReC still outperforms subtask-wise LESS on all four comparisons, by 0.22/2.14 points on Llama-3.1-8B and 0.80/2.47 points on Mistral-7B (MMLU/BBH).

While LESS already accounts for predefined validation subtasks, MoReC automatically discovers target-gradient modes without subtask labels and combines mode-aware candidate retrieval with set-level coverage optimization, rather than relying solely on per-example influence ranking. We will clarify this distinction and include the additional results in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 42b8 (Part 2/3)

W1 / Q2. Interpretation of target modes and representation in the final subset

We thank the reviewer for raising the important question of how target modes should be interpreted and represented in the final subset. MoReC adopts a training-oriented perspective on target capabilities: different validation examples may provide heterogeneous optimization signals, even within the same benchmark. By clustering optimizer-aware gradient representations, MoReC identifies groups of examples with similar adaptation directions, which we refer to as latent target capability modes. These modes capture model-dependent patterns of target requirements rather than relying on predefined subtask labels. We further examine how these discovered modes are represented in the final selected subsets through the quantitative analysis below.

Importantly, mode-wise retrieval ensures that each discovered mode contributes candidates, but the final facility-location objective has no explicit per-mode quota or guarantee. To examine the final selected subsets, we perform the same post-hoc assignment for LESS and MoReC. For each selected training example ii, we compute its mean non-negative gradient similarity to queries in mode mm and assign it to the mode with the highest affinity: ri,m=1∣Qm∣∑q∈Qmmax⁡(0,cos⁡(gi,hq)),m∗(i)=arg⁡max⁡mri,m.r_{i,m}=\frac{1}{|Q_m|}\sum_{q\in Q_m}\max(0,\cos(g_i,h_q)),\qquad m^*(i)=\arg\max_m r_{i,m}. We reconstruct the eight target modes using the saved validation gradients and original clustering seed. The table reports mode-affinity counts in the final 5% subsets (12,142 examples each), not original candidate-retrieval labels:

BenchmarkMethodM0M1M2M3M4M5M6M7
ARC-CLESS3761958891653114834217052486
ARC-CMoReC343510151391781124340316512475
BBHLESS163825981169228198111033313112
BBHMoReC169823871498255180210873201214
GSM8KLESS93361079771684592982481386
GSM8KMoReC996567780813644898701245693
MMLULESS91018561990883155814022772771
MMLUMoReC717135217128501554164832481061
MBPPLESS165510315323567188473078874
MBPPMoReC185412003984704202011911281061

We summarize these distributions using normalized Shannon entropy (larger means less concentrated) and total variation distance (TVD):

BenchmarkEntropy (LESS / MoReC)TVD
ARC-C0.863 / 0.8843.22%
BBH0.871 / 0.8934.27%
GSM8K0.773 / 0.8189.34%
MMLU0.959 / 0.9498.33%
MBPP0.801 / 0.88211.03%

Both methods have examples assigned to all eight modes, but their distributions differ. MoReC exhibits higher normalized entropy on four of the five benchmarks; for MBPP, the largest mode's share falls from 43.84% to 32.81%. MMLU is an exception. The ablation results (55.79 overall vs. 53.27 without mode discovery and 54.05 without set-level coverage) support the usefulness of both components.

AuthorsOpenReview ↗

Response to Reviewer 42b8 (Part 3/3)

Q3. Reproducibility and MBPP evaluation

We thank the reviewer for pointing out the unusual numerical precision in the reported MBPP results. We re-evaluated MBPP using the same evaluation protocol (3-shot, pass@1, 500 test examples). The corrected MoReC scores are 53.60% for Llama-3.1-8B and 44.00% for Mistral-7B. We have updated the corresponding results in the revised manuscript to ensure accurate reporting.

Regarding reproducibility, we provide the dataset construction procedures, LoRA training hyperparameters, gradient computation details, selection configurations, and evaluation protocols in the manuscript and appendix.

Q4. Sensitivity to the number of target modes and other parameters

We thank the reviewer for this suggestion. We conducted an additional sensitivity experiment on GSM8K using Llama-3.1-8B, varying the number of target modes KK while keeping other settings fixed.

Number of modes KK48163264
GSM8K accuracy (%)56.1058.4556.6356.3359.82

We use K=8K=8 as a unified default across all benchmarks, rather than tuning it separately for each task. This choice is motivated by the relatively small target validation sets, where excessively large KK values may fragment the target examples into very small clusters.

The results show that performance varies non-monotonically with KK. Although K=8K=8 outperforms several alternative settings, K=64K=64 achieves the highest accuracy of 59.82%, exceeding the default setting by 1.37 points. This suggests that adapting the number of target modes to individual benchmarks may further improve performance, while our main experiments deliberately use a fixed configuration across tasks.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, particularly the additional comparison with the original LESS aggregation strategy and the analysis of selected mode distributions. However, my main concerns remain. The method still largely combines existing gradient representations, K-means clustering, and facility-location optimization, with limited technical novelty. More importantly, the new mode-distribution analysis does not establish that the discovered gradient clusters correspond to meaningful target capabilities. Higher entropy within the same gradient space used for selection does not necessarily demonstrate improved capability coverage. The connection between the proposed mechanism and downstream improvements therefore remains insufficiently validated.

I also have some remaining concerns about experimental reliability and reproducibility. I appreciate the correction of the MBPP scores, but noticed a few additional inconsistencies worth clarifying. For example, the MBPP mean relevance values change from 0.0862/0.0859 (LESS/MoReC) in the original paper to 0.0228/0.0220 in the rebuttal, without an explanation for this substantial difference. Also, the newly reported 1B results include an ARC-C accuracy of 40.59%, which does not correspond to an integer number of correct predictions on the stated 1,172-example test set under a single-run evaluation. Clarifying whether these scores involve averaging across runs or different evaluation procedures would be helpful. In addition, the new sensitivity results show noticeable performance variation across different values of K, while multi-seed statistics and reference code remain unavailable. Overall, the rebuttal addresses several specific issues, but I believe the central concerns about novelty, capability coverage, and experimental reliability remain.

Reviewer_14QTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the targeted instruction data selection problem and proposes the MoReC method. Specifically, it first constructs gradient features for the entire source pool and the target sets. It then performs k-means clustering on the targets to identify capability modes. For each mode, it retrieves a set of examples to build a candidate pool. Finally, it performs coreset selection to jointly maximize the quality and diversity of the selected subset. Experiments on two base models and five benchmarks show the effectiveness of the MoReC method.

Critical strengths
  1. The paper is well motivated. Most targeted instruction data selection methods treat the target set as a single unit, which can cause less common capabilities in the target set to be dominated by more frequent ones. Standard top-k selection also introduces redundancy in the selected subset. MoReC addresses this issue by clustering the target set into different capability components, ensuring that each capability receives a certain number of examples, and then using a quality–diversity scoring function to rank samples for selection.

  2. The experiments show consistent gains over two models and five benchmarks. Most baseline methods and MoReC share the same gradient store and fine-tuning setup, so the performance differences can be more confidently attributed to the selection method itself.

Critical weaknesses
  1. The overall technical novelty appears limited. The objective follows a very similar pattern to QDIT [R1]. The main differences seem to be that “quality” is replaced by target relevance and the ground set is filtered by the targets. MoReC is essentially LESS gradients plus k-means clustering on the targets plus the QDIT objective. Without comparisons against standard QDIT (e.g., reward-model quality and sentence-embedding similarity) or LESS with a max-over-modes selection, it is unclear what MoReC contributes beyond combining existing components.

  2. My biggest concern is about the reported experimental results. In Table 1, the “FULL” performance is lower than RANDOM and many other methods, which is very unusual. In almost all target-agnostic or targeted instruction data selection papers, “FULL” is a very strong baseline and at least outperforms random selection on average. The MoReC results are also puzzling. For example, in the MBPP benchmark, the numbers of the other baseline methods end in “0” at the last decimal place, which may be due to the test size of MBPP; however, the MoReC numbers do not follow the same pattern. The statistics are also suspicious. Section 4.4 reports mean relevance 0.0449 and intra-subset similarity 0.1496 for the α=1 subset, but these are exactly the same numbers reported for LESS in Table 2.

[R1] Data Diversity Matters for Robust Instruction Tuning, EMNLP 2024

Questions for authors
  1. Can the authors report the test size of each benchmark and explicitly explain how the baseline and MoReC numbers are computed? Can authors also report mean and standard deviation for each method?

  2. For LESS and MoReC, how many of the finally selected examples fall into each mode?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 14QT (Part 1/3)

We thank the reviewer for the constructive feedback and for recognizing the motivation and controlled experimental setup of MoReC. We appreciate the concerns regarding technical novelty and experimental reliability, and address them below.

W1. Technical novelty and differences from QDIT and LESS

We agree that quality–diversity objectives, gradient-based relevance estimation, and facility-location optimization are established techniques. We do not claim novelty in these individual components. Instead, the contribution of MoReC lies in coupling target-mode-aware candidate construction with set-level optimization to address heterogeneous capability requirements in targeted instruction tuning.

Specifically, MoReC differs from QDIT and LESS in three aspects:

  1. Target-conditioned relevance. Unlike QDIT, which optimizes general data quality and diversity without conditioning on a specific target capability, MoReC employs optimizer-aware gradient representations to estimate the relevance of training examples to a given target task. In this respect, MoReC builds upon the gradient-based relevance formulation introduced by LESS.

  2. Automatic target-mode discovery. While LESS primarily ranks training examples by their aggregated relevance to the target set, MoReC explicitly models heterogeneity within that set. It discovers latent modes through clustering target gradient representations and independently retrieves relevant examples for each mode. This constructs a candidate pool that accounts for distinct target requirements without relying on predefined task categories.

  3. Target-aware set-level selection. Unlike relevance-only top-k selection, MoReC subsequently optimizes a facility-location objective over the mode-aware candidate pool, jointly considering target relevance and representative coverage. This enables the selection process to account for redundancy among relevant examples rather than evaluating each example independently.

Thus, the novelty of MoReC is not a new submodular objective or gradient similarity metric, but the integration of target-mode discovery, mode-conditioned retrieval, and set-level selection. Unlike QDIT's general quality–diversity optimization or LESS's per-example relevance ranking, MoReC explicitly incorporates heterogeneous target requirements into the data selection pipeline.

AuthorsOpenReview ↗

Response to Reviewer 14QT (Part 2/3)

W2. Reliability of experimental results

(a) FULL vs. RANDOM. We confirm that these experiments were conducted using identical training hyperparameters, as stated in Appendix B.2.

First, identical training epochs do not imply identical optimization budgets. Since RANDOM uses only 5% of the training corpus, FULL involves approximately 20 times more optimizer updates under the same batch-size and epoch settings. Therefore, this comparison controls the training configuration but is not compute-matched.

Moreover, our evaluation focuses on specific downstream capabilities, whereas the full instruction corpus covers a broad range of tasks. Additional training examples may not necessarily improve target-task performance, as some may be irrelevant or provide less useful training signals. Prior studies have also demonstrated that filtering instruction data can improve fine-tuning effectiveness (Chen et al., 2024).

Second, this phenomenon is consistent with previous findings in targeted instruction tuning. LESS (Xia et al., 2024, Table 2) demonstrates that fine-tuning with only 5% of selected data can match or outperform full-data fine-tuning. Notably, it also reports that RANDOM (5%) outperforms FULL on Mistral-7B for BBH (54.5 vs. 53.0).

We therefore emphasize that FULL is not necessarily an upper bound on downstream performance under fixed-epoch training. However, we do not interpret this result as evidence that smaller datasets are universally preferable.

(b) MBPP evaluation. We thank the reviewer for pointing out the unusual numerical precision in the reported MBPP results. We re-evaluated MBPP using the same evaluation protocol (3-shot, pass@1, 500 test examples). The corrected MoReC scores are 53.60% for Llama-3.1-8B and 44.00% for Mistral-7B. We have updated the corresponding results in the revised manuscript to ensure accurate reporting.

(c) Identical statistics for α=1\alpha=1 and LESS. We thank the reviewer for pointing this out. In the earlier draft, the intra-subset similarity was computed on a partial subset, whereas in the revised analysis, we recomputed it over the full selected subset.

When α=1\alpha=1, the facility-location term vanishes, and MoReC reduces to a relevance-only objective. However, this does not make the complete selection procedure identical to LESS, since MoReC still performs mode-aware candidate construction and uses a different similarity transformation.

Specifically, LESS achieves a mean relevance of 0.044921 and an intra-subset similarity of 0.145689, while MoReC (α=1\alpha=1) achieves 0.044881 and 0.145217, respectively. The mean relevance values become identical when rounded to four decimal places, whereas the recomputed intra-subset similarity values remain slightly different.

MethodMean relevanceIntra-subset similarity
LESS0.0449210.145689
MoReC (α=1\alpha=1)0.0448810.145217
MoReC (α=0.6\alpha=0.6)0.0437390.116149

Thus, α=1\alpha=1 recovers relevance-only optimization, but not necessarily the identical subset produced by LESS. Moreover, the default setting (α=0.6\alpha=0.6) achieves lower intra-subset similarity while maintaining comparable mean relevance.

Q1. Evaluation protocols

We thank the reviewer for this suggestion. All evaluations were conducted using lm-evaluation-harness. We verified the evaluation configurations and test set sizes from the evaluation records, as summarized below.

BenchmarkTest examplesEvaluation metricSetting
MMLU14,042Accuracy5-shot
ARC-C1,172Accuracy0-shot
GSM8K1,319Exact Match5-shot
BBH6,511Exact Match0-shot
MBPP500pass@13-shot

MMLU comprises 57 subtasks, while our BBH evaluation covers 27 task configurations.

We will clarify these evaluation settings in the revised manuscript to make the experimental protocol more transparent and reproducible.

AuthorsOpenReview ↗

Response to Reviewer 14QT (Part 3/3)

Q2. Distribution of selected examples across target modes

Thank you for this suggestion. We agree that examining the final selected subsets, rather than only the candidate pools, is important for understanding the effect of mode-aware selection.

Analysis protocol. We analyze the final LESS and MoReC subsets across all five benchmarks, with 12,142 training examples selected for each benchmark (5% of OpenHermes).

We reconstruct the eight target modes by rerunning the original K-means procedure on the saved target gradient representations with the same random seed. For each selected training example ii, we compute its average non-negative gradient similarity to the target examples in each mode:

ri,m=1∣Qm∣∑q∈Qmmax⁡(0,cos⁡(gi,hq)),m∗(i)=arg⁡max⁡mri,m.r_{i,m}=\frac{1}{|Q_m|}\sum_{q\in Q_m}\max(0,\cos(g_i,h_q)),\qquad m^*(i)=\arg\max_m r_{i,m}.

We apply the same post-hoc assignment rule to LESS and MoReC to ensure a consistent comparison. These assignments represent each training example's dominant target-mode affinity rather than its original retrieval provenance.

Final-subset distributions. The following table reports the number of selected training examples assigned to each target mode.

BenchmarkMethodM0M1M2M3M4M5M6M7
ARC-CLESS3761958891653114834217052486
ARC-CMoReC343510151391781124340316512475
BBHLESS163825981169228198111033313112
BBHMoReC169823871498255180210873201214
GSM8KLESS93361079771684592982481386
GSM8KMoReC996567780813644898701245693
MMLULESS91018561990883155814022772771
MMLUMoReC717135217128501554164832481061
MBPPLESS165510315323567188473078874
MBPPMoReC185412003984704202011911281061

To quantify the differences, we compute the normalized Shannon entropy of each final-subset mode distribution (higher indicates less concentration) and the total variation distance (TVD) between LESS and MoReC.

BenchmarkEntropy (LESS / MoReC)TVD
ARC-C0.863 / 0.8843.22%
BBH0.871 / 0.8934.27%
GSM8K0.773 / 0.8189.34%
MMLU0.959 / 0.9498.33%
MBPP0.801 / 0.88211.03%

Observations. Both methods have nonzero assignments to all eight discovered modes across all five benchmarks. Nevertheless, their final mode distributions differ. MoReC achieves higher normalized entropy on four of the five benchmarks, indicating a generally less concentrated distribution.

For example, on MBPP, the proportion assigned to the largest mode decreases from 43.84% (LESS) to 32.81% (MoReC), while the proportions assigned to all seven other modes increase. Similarly, on GSM8K, the largest mode decreases from 50.30% to 46.76%, accompanied by increased representation of several other modes. MMLU is an exception, where MoReC exhibits slightly lower entropy.

Importantly, MoReC does not impose explicit mode-balancing constraints on the final subset, nor does it guarantee greater coverage for every individual mode. The observed differences reflect the combined effects of mode-aware candidate construction and subsequent set-level optimization.

Overall, this analysis shows that MoReC and LESS yield different final-subset compositions across target modes, with MoReC exhibiting less concentrated post-hoc mode assignments on four of the five benchmarks. These findings characterize the behavior of MoReC's combined mode-aware retrieval and set-level selection pipeline without implying uniformly improved per-mode coverage. We will incorporate this analysis and clarify these distinctions in the revised manuscript.

References

[1] Chen et al. AlpaGasus: Training a Better Alpaca with Fewer Data. ICLR 2024.

[2] Xia et al. LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 2024.

[3] Bukharin et al. Data Diversity Matters for Robust Instruction Tuning. Findings of EMNLP 2024.

Thanks for running the additional experiments. The subtask-wise LESS results help, but they only partly address my concern about novelty. Comparisons with standard QDIT and with LESS using the highest influence score across MoReC’s own learned modes are still missing. Those comparisons matter because they would give a clearer picture of what MoReC adds beyond combining the two approaches. On MBPP, the corrected scores make sense, but I still don’t know where the original score came from. For these reasons, I’m keeping my rating at 2.

Reviewer_yHRcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Instruction tuning is a widely adopted technique in large language model research to enhance model capabilities for specialized tasks. Data selection is an effective approach for efficient instruction tuning. Existing data selection methods treat datasets as homogeneous and evaluate each data point independently, which can lead to redundant samples being selected into the coreset. To address this issue, this paper proposes MoReC, a Mode-aware Relevance-gated Coreset selection framework. Experimental results across knowledge, reasoning, and code generation tasks show that MoReC achieves competitive or state-of-the-art performance using only 5% of the instruction data.

Critical strengths
  1. This work addresses a highly popular research topic. Instruction tuning is critical for adapting LLMs to downstream applications, and high-quality data is central to it. Accordingly, data selection and filtering remain an active research frontier.

  2. The paper is well structured and clearly written, making it easy to understand.

  3. Figures and tables are carefully arranged to highlight key findings.

Critical weaknesses
  1. The experiments are conducted only on two models, Llama-3.1-8B and Mistral-7B-v0.1. This limited scope is insufficient to demonstrate the generalizability of the proposed method. To strengthen the evaluation, it is recommended to include smaller (e.g., 1B) or larger models (e.g., 14B, 30B).

  2. The experimental analysis lacks complementary perspectives. The core motivation of MoReC is that redundant data degrades performance. Two follow-up questions deserve discussion: (1) Do the adopted baseline methods suffer from substantial redundancy compared with MoReC? (2) Does redundancy inherently hurt performance? For instance, if samples in the full original dataset are highly diverse, is redundancy still a major concern? These questions are not explored in the paper. Overall, the experimental perspective is not sufficiently comprehensive.

  3. Some prior works have already taken data diversity into account, such as "Difficulty–Diversity Collaborative Filtering for Data-Efficient LLM Fine-Tuning" at ICLR2026, which could serve as a baseline for comparison.

  4. No public repository for datasets and code is provided in the current submission, which hinders code inspection and result reproducibility.

Questions for authors
  1. Does the proposed method MoReC still work for larger models (e.g., 14B and 30B) or smaller ones (e.g., 1B)?

  2. Do the existing methods (e.g., selected baselines) suffer from substantial redundancy compared with MoReC?

  3. If samples in the full original dataset are highly diverse, is redundancy still a major concern?

  4. How does MoReC perform compared with recent data diversity work, e.g., "Difficulty–Diversity Collaborative Filtering for Data-Efficient LLM Fine-Tuning"?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yHRc (Part 1/3)

W1 / Q1. Generalization across model scales

We agree that evaluating MoReC on models of different scales would strengthen the generalizability claim. To address this concern, we conducted additional experiments on Llama-3.2-1B.

We compare MoReC with Random and LESS under the same 5% data selection budget across all five benchmarks. The results are summarized below.

MethodMMLUARC-CGSM8KBBHMBPPAverage
Random32.2537.885.6926.1725.0025.40
LESS33.6338.658.1126.6925.4026.50
MoReC32.2240.598.9427.3427.0027.22

MoReC achieves the highest average score of 27.22, outperforming LESS by 0.72 points and Random by 1.82 points. It achieves the best results on four of the five benchmarks, with MMLU being the exception. Together with our original results on Llama-3.1-8B and Mistral-7B, these findings provide further evidence of MoReC's effectiveness across different model scales.

Nevertheless, we acknowledge that our evaluation does not yet cover larger models such as 14B or 30B. We will incorporate the 1B results into the revised manuscript and clarify that generalization to larger model scales remains to be validated.

AuthorsOpenReview ↗

Response to Reviewer yHRc (Part 2/3)

W2 / Q2. Do existing baselines suffer from greater redundancy?

We thank the reviewer for raising this question. To examine whether existing methods select redundant training examples, we compare the final subsets selected by LESS, G2IS, and MoReC on Llama-3.1-8B across all five benchmarks using three complementary metrics. Mean relevance measures average gradient cosine similarity to the target validation set, reflecting target alignment. Intra-subset similarity measures average pairwise gradient similarity within the selected subset, capturing global redundancy. Nearest-neighbor (NN) mean measures the average similarity between each selected example and its most similar counterpart, capturing local redundancy. All metrics are computed using the same gradient representations over the full selected subsets, with self-similarity excluded from redundancy measurements. All methods use approximately the same 5% selection budget.

BenchmarkRel. LESSRel. G2ISRel. MoReCIntra LESSIntra G2ISIntra MoReCNN LESSNN G2ISNN MoReC
MMLU0.03160.00190.02960.04420.17780.03450.29560.58620.2220
ARC-C0.02270.00720.02180.06000.16170.04750.25290.56790.1983
GSM8K0.04490.01410.04370.14570.18050.11610.30340.58360.2591
BBH0.04370.01880.04280.14210.21040.11190.29330.56540.2537
MBPP0.02280.00820.02200.11780.17580.08630.28390.57250.2328
Average0.03310.01000.03200.10200.18130.07930.28580.57510.2332

MoReC exhibits consistently lower global and local redundancy than both baselines across all five benchmarks. Compared with LESS, it retains 96.6% of mean relevance while reducing intra-subset similarity by 22.3% and NN mean by 18.4%. Compared with G2IS, the corresponding reductions are 56.3% and 59.5%. These findings indicate that MoReC distributes its selection budget across a broader range of gradient directions while largely preserving target relevance, rather than repeatedly selecting highly similar examples.

W2 / Q3. Does redundancy inherently harm performance, even if the original pool is diverse?

We agree that redundancy is not inherently harmful, and that greater diversity alone does not guarantee better downstream performance. Our motivation is more specific: under a limited selection budget, independently ranking examples by target relevance may repeatedly select examples with similar gradient representations, leaving other target-relevant regions underrepresented. Notably, OpenHermes integrates data from diverse sources, including GPTeacher, WizardLM, Airoboros GPT-4, Camel-AI, CodeAlpaca, and GPT4-LLM, covering general instruction following, role-playing, domain-specific tasks, and code generation. However, diversity in the original corpus does not necessarily translate into diversity in the selected subset.

We provide two complementary pieces of evidence. First, in the component ablation on Llama-3.1-8B, removing set-level coverage (α=1\alpha=1) while retaining mode discovery and candidate retrieval decreases the five-benchmark average from 55.79 to 54.05 (a 1.74-point drop), with other settings fixed.

Second, the GSM8K sensitivity analysis reveals a relevance–redundancy trade-off. Compared with relevance-only selection (α=1\alpha=1), the default setting (α=0.6\alpha=0.6) retains 97.5% of mean relevance (0.043739 vs. 0.044881), while reducing intra-subset similarity by 20.0% (0.116149 vs. 0.145217). The corresponding downstream performance improves from 58.15 to 58.45. Importantly, α=0\alpha=0 yields even lower redundancy but does not achieve the best performance, indicating that minimizing redundancy alone is insufficient.

These findings highlight the importance of balancing target relevance and subset coverage rather than simply minimizing redundancy.

AuthorsOpenReview ↗

Response to Reviewer yHRc (Part 3/3)

W3 / Q4. Comparison with Difficulty–Diversity Collaborative Filtering (DDCF)

We thank the reviewer for highlighting DDCF (Hoang et al., ICLR 2026). We agree that it is an important related diversity-aware method. However, DDCF and MoReC address different data selection settings.

DDCF combines model-specific question difficulty, estimated through collaborative filtering, with embedding-based diversity. In contrast, MoReC focuses on targeted instruction tuning, discovering latent target modes through gradient clustering and combining target relevance with set-level coverage optimization. Thus, DDCF emphasizes model-specific difficulty and semantic diversity, whereas MoReC emphasizes target-task relevance and coverage of heterogeneous capability requirements.

Moreover, DDCF primarily evaluates mathematical reasoning using OpenR1-Math-220K, whereas MoReC selects instruction data from OpenHermes for five heterogeneous downstream tasks. These differences in datasets and evaluation protocols prevent a direct comparison of the reported results. Our experiments already include the diversity-aware baseline TAGCOS under a unified evaluation setting. We will discuss DDCF in the revised manuscript and acknowledge a direct comparison under matched conditions as future work.

W4. Code and data availability

We acknowledge that the absence of a public code repository limits independent verification. The training corpus (OpenHermes) and evaluation benchmarks are publicly available, and we provide detailed descriptions of the algorithms, datasets, hyperparameters, and training and evaluation protocols in the manuscript and appendix.

Thank you for the authors' detailed response. Some of my questions have been addressed, but I still have concerns about the method's performance on larger models and how it compares with recent approaches.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.