On the Robustness of Targeted Instruction Data Selection and Beyond

Targeted instruction data selection and target-agnostic instruction selection are treated as two separate lines of work, yet they meet at a natural limit: when the target set is entirely absent or carries no task signal, a targeted selector reduces to a target-agnostic one. In practice, this limit is approached more often than it appears. Target sets collected from real deployments are typically small or may contain redundant or mislabeled examples, causing the estimated target profile to be noisy and unstable. A matching signal built on such a profile can steer selection away from high-quality source data and toward noise, producing worse results than simply ignoring the target set. We call a targeted selector robust if it consistently beats target-agnostic selection across varying target sizes and noise levels. Measured by this criterion, most existing targeted selectors are not robust: four of the five we evaluate score below a target-agnostic baseline, meaning the target set hurts rather than helps. This failure is largely invisible in existing benchmarks, because targeted methods are evaluated against random selection rather than against target-agnostic selection. We identify two structural causes and address them with our method, CLASS (Closed-set LAbel Space Selection), which combines a quality term to supply the target-agnostic floor with closed-set capability matching to keep profile estimation stable under both target scarcity and target noise across different benchmarks. CLASS reaches an average of 54.1 across five tasks, above the target-agnostic baseline at 52.0 and above every targeted baseline, with highest selection stability.