- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GRIP, which combines model-conditioned allocation with length-aware sampling to improve data efficiency for language model pre-training.
Critical strengths
- This paper is well-motivated, it addresses token-budget allocation in language-model pre-training, where informative clusters receive insufficient capacity and cluster budgets are filled with redundant examples.
- This paper proposes GRIP, an effective method that combines model-conditioned allocation with length-aware geometric selection.
- This paper trains 8B and 16B MoE models from scratch and provides sufficient experimental validation.
Critical weaknesses
- I am concerned that the cluster quality scores rely entirely on Qwen3-235B-A22B. Although several outputs from each cluster are manually inspected, the paper does not report the scale of this manual inspection and does not evaluate robustness using alternative judge models. It remains unclear whether a single model’s preference for particular programming languages could bias the final token budget allocation.
- I notice that GRIP underperforms UniGeM on several tasks in Table 1. Could the authors explain these degradations and clarify whether they reflect systematic trade-offs introduced by GRIP’s data allocation strategy?
- The x-axis label in the left panel of Figure 4 is obscured. Please correct the figure in the main paper.
Questions for authors
See the Weakness.
Final Follow-up on Rebuttal Response
Dear Reviewer fFsr,
Thank you again for your valuable time and constructive feedback.
As today is the final day of the rebuttal discussion period, we would like to kindly follow up on our previous responses. We have carefully addressed your concerns but have not yet received your further feedback.
We would sincerely appreciate it if you could take a moment to review our responses and let us know whether your concerns have been resolved. If you find our clarifications satisfactory, we would be grateful if you could consider updating your evaluation score accordingly.
Thank you very much for your time and consideration.