- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Many forward-only data selection methods utilize the final layer of a model. The paper is built around the observation that different data points can create models with different intermediate layers and the same final layer. To address this and select the right data points, the paper proposes BiTFlow, which tracks how data updates across many layers in a token-wise fashion. The paper demonstrates that BiTFlow selects highly effective training subsets and outperforms other selection methods while being efficient.
Critical strengths
The paper identifies and addresses the problem of last-layer similarity collapse by tracking how data updates across middle layers. The algorithm proposed, BitFlow, is also very computationally efficient and shows strong performance in tests. It outperformed other top selection methods using only a strict 5% data budget. Finally, the training data chosen by BiTFlow transfers well to other models.
Critical weaknesses
None.
Questions for authors
- The results show that performance drops when you select 15% of the data. Besides general noise, why does this happen? How can a user find the best data size for a new dataset?
- The system's success depends heavily on how it weights the prompt versus the response. How should users choose the right weight for a new task? Is there a simple rule to avoid testing every option?
- BiTFlow only slightly beat simpler methods on the very clean SmolTalk dataset. Is BiTFlow less useful for data that is already filtered? When is the extra effort worth it?
Dear reviewers,
Please be aware that little time remains for the author-reviewer discussion. As the deadline is approaching, we kindly request that you refrain from asking for additional experiments. Please summarize your final position concisely.
Sincerely, AC