- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In this work, the authors explore the measure of quality in the pretraning data selecton. The quality should contain different signals, and combining these signals can help to balance the quality from different perspectives. The experiment results demonstrate performance gain over random and Dolma3-based baselines.
Critical strengths
- The data selection problem is important in large language model training, and quality-based selection is widely adopted in both industrail and academic areas.
- Extensive experiments are conducted to evaluate the data quality from different perspectives.
Critical weaknesses
- The performance gain seems modest compared to baseline for MoE models. There is no experiments on varying data budgets, and other data domains.
- The data scoring process requires further investigation and justification. The Qwen3-32B model seems small, compared with the 7B-MoE model and existing LLM-as-a-Judge works. Typically, a 70B (e.g. Llama-3-70B-Instruct) or state-of-the-art open source model (e.g., GPT) is preferred for the precision of the result.
- The score distribution is not aligned with each other. For example, the mean for orthography is 4.35, which makes the signals unbalanced. The rubrics should be adjusted to make the mean aligned.
- The mechanism behind the improvement is obscure. The discussion in page 7 says the performance gain originates from the mix on the simplex, but the limitation also indicates that the gate of Dolma 3 and score normalization are changed at the same time. A more rigorous experiment should be conducted to locate the reason for performance improvement.
- The dimension definition does not align with the experiment result. Several dimensions are construct from the benchmark usefulness, but there is result to show that these dimensions are effective.
- Although 192 scorers are trained, only 5 signals are used in the experiments and many of them received a 0 weight. The effectiveness of the dimensions should be checked.
- The baseline comparison is insufficient. Standard data selection methods, such as PPL, DSIR, QuRating, Meta-Rater, and other scorer-based methods are not incorporated.
- Although the performance are given in BPB, the accuracy of the benchmark should be provided.
- The presentation of the work should be improved. There are several typos, e.g., there is no Section 2.2 but it appears in the appendix.
Questions for authors
- How would the method perform on other data domains, such as code, multimodal?
- Is this method sensitive to the selection of LLM-judges?
- Why only 5 are selected among 192 signals, and why not continue to explore other signals?
We thank Reviewer Lxjm for their frank review. We are glad they find that the paper "proposes reasonable methods for combining these scores for an overall data selection strategy". The central concern, that the submission read "more as a loose experiment log than a paper written for conference publication". We have considered this feedback when completing our revision, wherein we redesigned and reran the mixture experiments more systematically to provide clearer insights into the techniques we propose and their potential benefits. We also revised the text of the paper.
In response to the critical weaknesses listed, we have revised the paper and provide the following response (in order of the paragraphs of the review):
We are very grateful for the directness of the feedback, which led us to more systematically organise the experiments and improve the paper.