- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents ATCS, a two-stage framework for efficient and scalable SFT data selection. In the first stage, ATCS uses a smaller proxy LLM to score and filter training samples based on a utility metric, such as IFD. It also computes attention scores to identify the most relevant sentences in each prompt. In the second stage, the target LLM evaluates the utility of the retained samples using prompts shortened according to the attention scores from the first stage. Experiments on six benchmarks show that ATCS achieves performance comparable to directly using the target model for data selection, while reducing selection time by 77.2%–85.8%.
Additionally, this work explores the potential benefits of using different utility metrics across the two stages (cross-utility).
Critical strengths
- The paper focuses on a practical problem of the scalability of SFT data selection.
- The idea of shortening prompts based on attention scores to reduce inference costs during data selection is interesting.
- The results in Table 1 and Figure 3 show that ATCS achieves performance comparable to directly using the target model for data selection, while substantially reducing computational costs.
Critical weaknesses
Major Weaknesses
- The paper provides two main motivations for prompt shortening: (1) the limited context window of the proxy LLM and (2) reducing inference costs in Stage 2, which uses the target model. However, both motivations need further justification.
- Limited context window: This limitation appears to be largely due to the choice of GPT-2, which supports only 1,024 tokens. Why not consider a more recent small LLM, such as Qwen3-0.6B, which supports a much longer context window (32K tokens)? Would prompt shortening still be necessary with such a model?
- Inference efficiency: The paper does not provide concrete measurements of how much inference time is saved by shortening prompts in Stage 2. It also lacks a comparison of the downstream performance when using full versus shortened prompts. These results are important for understanding the actual benefits and potential trade-offs of prompt shortening.
- Prompt and response lengths can be particularly large in SFT datasets involving long CoT reasoning or agentic tasks. These are also scenarios where efficient data selection could be especially valuable. How well does ATCS generalize to such settings, where long prompts and responses may contain information essential to assessing sample utility?
- The motivation for using cross-utility is also weak. The paper argues that relying on a single utility metric may reduce the diversity of selected samples. However, the importance of diversity in SFT data selection has already been extensively studied, so this observation is not particularly novel. More importantly, it is unclear why cross-utility selection is preferable to explicitly incorporating diversity as an additional selection criterion, as explored in prior work.
- The presentation of the prompt-shortening component is difficult to follow. The paper would benefit from a more formal description of the procedure, including mathematical formulations (beyond equations 3-5 that solely focus on attention scores, not sentence splitting) and concrete examples using actual prompts.
Other Weaknesses
- The experimental evaluation primarily considers baselines published in or before 2024 and relatively older model families, such as Qwen2.5 and Llama 3.1. Including more recent data-selection baselines and model families would strengthen the empirical evaluation and help establish the relevance of ATCS to current SFT settings.
Questions for authors
Please refer to the (major) weaknesses section for my questions.
Overall, I feel that the research problem studied is real. However, the motivations and justifications for the key design choices are not sufficiently convincing.