MoReC: Mode-aware Relevance-Gated Coresets for Targeted Instruction Tuning

Instruction tuning is widely used to adapt LLMs to specialized tasks, yet its effectiveness is often constrained by the limited availability of high-quality target-specific data. To alleviate this limitation, existing data selection methods identify, from general instruction corpora, examples whose learning effects align with the target task. However, they often treat the target set as homogeneous and evaluate training examples independently, which may obscure fine-grained target capability requirements and produce redundant subsets. We propose MoReC, a Mode-aware Relevance-gated Coreset selection framework. Built upon optimizer-aware gradient representations, MoReC first discovers latent capability modes in the target validation set and retrieves relevant training examples for each mode, constructing a candidate pool that preserves diverse target requirements. It then performs set-level selection through a facility-location objective that jointly optimizes target relevance and coverage. The overall objective is monotone submodular and admits an efficient greedy algorithm with a $(1-1/e)$ approximation guarantee. Experiments on benchmarks spanning knowledge, reasoning, and code generation show that, using only 5% of the instruction data, MoReC achieves competitive or state-of-the-art performance, while yielding subsets with lower intra-subset similarity. These results indicate that covering the target task's diverse capability modes is more effective than concentrating the budget on top-ranked examples.