BiTFlow: Efficient Coreset Selection for Instruction Tuning via Balanced Token Flow Matching

Coreset selection for instruction tuning aims to reduce fine-tuning computation while preserving downstream performance. Forward-only selection methods avoid costly model updates and backward passes by comparing hidden-state fingerprints, most often from the last layer. However, samples with highly similar last-layer states can induce different intermediate token updates, a phenomenon we call *last-layer similarity collapse*, indicating that intermediate layers carry information beyond the last-layer state. We capture this information with residual flows, the residual updates of each token between selected layers. Building on these flows, we introduce BiTFlow, a forward-only coreset selection algorithm that compares candidate samples with a small set of target-task samples at the token level. BiTFlow summarizes target residual flows into token-level prototypes and aligns candidate tokens to them via one-to-one bipartite matching. A balanced aggregation then scores each sample by both how well its tokens match and how many distinct target token IDs are covered. At a 5% selection budget on the LESS (0.27M samples) and SmolTalk (1.04M samples) pools, BiTFlow achieves the highest average performance and exceeds full-data fine-tuning on multiple benchmarks, and its selections transfer across model scales and families. Scoring requires no backward passes or warmup training and is $\sim$18$\times$ faster than gradient-based methods.