acceptodds
Under review as a conference paper at ICLR 2027

ATCS: Accelerating Target-model-aware Coreset Selection

Abstract

Large-scale supervised fine-tuning (SFT) of foundation models increasingly relies on instruction data pools aggregated from heterogeneous sources such as the web, open communities, and synthetic generation pipelines. To support reliable model adaptation, such data must be transformed into a compact, high-value training set by removing redundant, noisy, and low-utility examples. This transformation naturally calls for data selection methods that can reduce the training pool while preserving the information needed for downstream adaptations. Target-model-aware coreset selection is particularly effective for this purpose because it evaluates candidates according to their utility for the model to be fine-tuned. However, existing target-model-aware methods are difficult to deploy at scale. Specifically, they typically require the target model to score every candidate instruction-response pair, turning data selection itself into a costly inference workload. To address this bottleneck, we propose ATCS, a general acceleration framework that treats selection as a two-stage data processing pipeline. ATCS exploits the agreement between the proxy and target models in identifying low-value samples and salient instructional content, reducing both the number and length of records evaluated by the target model. Besides, ATCS is compatible with multiple utility functions and supports cross-utility selection across coarse filtering and target-model reranking. Experiments on large-scale instruction-tuning datasets show that ATCS preserves downstream performance comparable to exhaustive target-model-aware selection while reducing end-to-end selection time by more than 77%, making model-aware coreset selection practical for scalable LLM data preparation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coreset selection, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_V1CKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ATCS, a two-stage framework for efficient and scalable SFT data selection. In the first stage, ATCS uses a smaller proxy LLM to score and filter training samples based on a utility metric, such as IFD. It also computes attention scores to identify the most relevant sentences in each prompt. In the second stage, the target LLM evaluates the utility of the retained samples using prompts shortened according to the attention scores from the first stage. Experiments on six benchmarks show that ATCS achieves performance comparable to directly using the target model for data selection, while reducing selection time by 77.2%–85.8%.

Additionally, this work explores the potential benefits of using different utility metrics across the two stages (cross-utility).

Critical strengths
  1. The paper focuses on a practical problem of the scalability of SFT data selection.
  2. The idea of shortening prompts based on attention scores to reduce inference costs during data selection is interesting.
  3. The results in Table 1 and Figure 3 show that ATCS achieves performance comparable to directly using the target model for data selection, while substantially reducing computational costs.
Critical weaknesses

Major Weaknesses

  1. The paper provides two main motivations for prompt shortening: (1) the limited context window of the proxy LLM and (2) reducing inference costs in Stage 2, which uses the target model. However, both motivations need further justification.
  • Limited context window: This limitation appears to be largely due to the choice of GPT-2, which supports only 1,024 tokens. Why not consider a more recent small LLM, such as Qwen3-0.6B, which supports a much longer context window (32K tokens)? Would prompt shortening still be necessary with such a model?
  • Inference efficiency: The paper does not provide concrete measurements of how much inference time is saved by shortening prompts in Stage 2. It also lacks a comparison of the downstream performance when using full versus shortened prompts. These results are important for understanding the actual benefits and potential trade-offs of prompt shortening.
  1. Prompt and response lengths can be particularly large in SFT datasets involving long CoT reasoning or agentic tasks. These are also scenarios where efficient data selection could be especially valuable. How well does ATCS generalize to such settings, where long prompts and responses may contain information essential to assessing sample utility?
  2. The motivation for using cross-utility is also weak. The paper argues that relying on a single utility metric may reduce the diversity of selected samples. However, the importance of diversity in SFT data selection has already been extensively studied, so this observation is not particularly novel. More importantly, it is unclear why cross-utility selection is preferable to explicitly incorporating diversity as an additional selection criterion, as explored in prior work.
  3. The presentation of the prompt-shortening component is difficult to follow. The paper would benefit from a more formal description of the procedure, including mathematical formulations (beyond equations 3-5 that solely focus on attention scores, not sentence splitting) and concrete examples using actual prompts.

Other Weaknesses

  1. The experimental evaluation primarily considers baselines published in or before 2024 and relatively older model families, such as Qwen2.5 and Llama 3.1. Including more recent data-selection baselines and model families would strengthen the empirical evaluation and help establish the relevance of ATCS to current SFT settings.
Questions for authors

Please refer to the (major) weaknesses section for my questions.

Overall, I feel that the research problem studied is real. However, the motivations and justifications for the key design choices are not sufficiently convincing.

Reviewer_Gzm8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Target-model-aware coreset selection identifies useful instruction-tuning data but incurs substantial overhead by scoring every candidate with the target model. ATCS accelerates this process through a two-stage pipeline that uses a lightweight proxy to filter candidates and identify salient instructional content before target-model reranking. The framework reduces both the number and length of samples evaluated by the target model and supports different utility functions across stages. Experiments show comparable downstream performance to exhaustive target-model selection while reducing total selection time

Critical strengths
  • The paper is well-written and easy to follow, with a clear presentation of the motivation and proposed framework.

  • The motivation is practically relevant. The overhead of existing data-selection methods can offset the computational savings from training on a smaller subset. ATCS addresses this bottleneck by using a lightweight proxy to reduce both the number and length of samples evaluated by the target model, substantially reducing selection time while maintaining competitive downstream performance.

Critical weaknesses
  • Limited analysis of proxy-model choice and proxy–target relationships. Although the paper evaluates GPT-2 and Pythia-70M, it does not systematically explore how the relationship between proxy and target models, such as shared model family, pretraining data, or domain expertise, affects their agreement and downstream selection quality. Moreover, the effects of proxy choice on candidate filtering and attention-based sentence selection are not separately analyzed. A broader, controlled evaluations are required to clarify when proxy–target agreement holds and provide practical guidance for selecting a suitable proxy.

  • Limited comparison with established and recent baselines. Since ATCS is primarily supported by empirical evidence rather than theoretical guarantees, broader comparisons are particularly important to establish its effectiveness and efficiency. However, comparisons with existing methods are limited to Superfiltering and FastCore. The evaluation omits relevant approaches discussed in the related work, including DEITA, InsTag, SelectIT, LESS, SCAR, and TAGCOS, as well as recent data-selection methods [1, 2] and the established gradient-norm baseline [3]. Comparing downstream performance and selection costs against these methods would provide a more convincing assessment of ATCS’s performance–efficiency trade-off.

  • Mismatch between agreement analysis and filtering aggressiveness. Figure 2(a) examines proxy–target agreement on the bottom 20–50% of samples, whereas Stage 1 discards 90% in practice. Reporting the fraction of the target model’s top-ranked samples retained after filtering would more directly validate this aggressive pruning.

  • Insufficient component-wise ablation. ATCS combines candidate filtering and attention-based input shortening, but their individual contributions to performance and runtime are not isolated. Comparing filtering-only, shortening-only, and combined variants are required

  • The computational cost comparison is limited to exhaustive target-model scoring (Fine). Including random selection and recent data-selection baselines, and reporting both downstream performance and selection costs, would provide a clearer assessment of ATCS’s performance-efficiency trade-off.

References
[1] Lee et al., OASIS: Online Sample Selection for Continual Instruction Tuning, ACL 2026
[2] Qin et al., Infobatch: Lossless training speed up by unbiased dynamic data pruning, ICLR 2024
[3] Katharopoulos et al., Not all samples are created equal: Deep learning with importance sampling, ICML 2018

Questions for authors

Please refer to the weakness section

Reviewer_oqiQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Large-scale supervised fine-tuning (SFT) of base large language models enables better adaptation to downstream tasks. For effective adaptation, data quality is a core requirement, necessitating the removal of redundant, noisy, and low-quality samples. Efficient data selection algorithms are therefore needed to achieve this goal. This paper proposes ATCS, a general acceleration framework that formulates selection as a two-stage data processing pipeline. Experiments show that the method reduces end-to-end selection time by more than 77%.

Critical strengths
  1. This work addresses a highly timely research hotspot. SFT has attracted significant attention from both academia and industry, and the paper introduces a novel perspective centered on selection efficiency.

  2. The figures, tables, and writing are clear and easy for readers to follow.

Critical weaknesses
  1. My primary concern is whether improved selection efficiency translates into gains across the overall SFT pipeline. To my understanding, the dominant computational cost of SFT lies in fine-tuning the large language model itself; data selection is not the main bottleneck. It remains unclear how much overall SFT efficiency can be improved by accelerating the selection stage. For example, if the proposed method selects a 5% data subset for fine-tuning, can it outperform a baseline method that selects 4% of the data in both performance and efficiency?

  2. The set of baselines is limited. Superfiltering (2024) and FastCore (2023) are included. Because data selection for SFT is an active research area, numerous relevant works have emerged. Even approaches not originally designed for acceleration should be included for comparison, including recent publications from 2025 and 2026.

  3. Three models are evaluated: Llama3.1-8B and Qwen2.5-7B/14B, spanning several model sizes. It is still recommended to include smaller models (e.g., 1B) or larger ones (e.g., 30B) in the evaluation suite.

  4. The authors describe ATCS as a “general acceleration framework”. It is unclear whether this generality implies that existing data selection algorithms can be adapted and accelerated within this framework. The scope of ATCS’s applicability needs elaboration: can mainstream data selection algorithms be integrated into this framework?

  5. The introduction is poorly structured and fails to justify why accelerating data selection is important within the full SFT workflow.

Questions for authors
  1. Does improved selection efficiency translate into gains across the overall SFT pipeline?

  2. How does ATCS perform compared with the recent work from 2025 and 2026?

  3. How does ATCS still work for smaller models (e.g., 1B) and larger ones (e.g., 30B)?

  4. Can mainstream data selection algorithms be integrated into the ATCS framework?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.