- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper uses the CLIVO method to combine semantic allocation, automatically learned validation objectives, and differentiable mixture search, enabling useful feedback to be retained with a moderate budget and thereby improving model performance.
Critical strengths
- The paper has a comprehensive framework, is written transparently, and provides detailed experimental records
- The paper divides data selection into Unit, Signal, and Learning, clearly breaks down the problem, and can be combined with existing methods
Critical weaknesses
- It is unreasonable for the paper to use the final evaluation benchmark as the basis for data selection, as this may cause training to overfit the benchmark scores. The method’s effectiveness in real-world applications therefore remains to be confirmed.
- The overall data-selection process is complex and involves extensive probe training. The choice of parameters, such as the value of K in K-means, also lacks detailed analysis.
- The total training compute is unclear. Final training consists of 100 optimization steps, and the reported s0 experiment has an effective global batch size of 32. If this applies to all final models, each model sees only about 3,200 samples, far fewer than 100K; the recipe for Full(8M) is not specified.
- The writing is unclear, and some passages and explanations of the formulas are lengthy and difficult to understand.
Questions for authors
See Weakness above.