- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work addresses the scaling of data selection in large scale pre-training. It answers the important questions of (1) construction of high quality datasets and, (2) whether the advantages of that data persist at scale.
A target distribution aligned dataset is constructed using a pair of small language models that estimate a document level score, that provides the direction of redistribution. This is then converted into a family of sampling policies from which the dataset is sampled using Gumbel Top-k, after joint calibration (minimization of JS Divergence of the selected and sampled distributions) of two parameters representing the strength of emphasis on target-preferred documents, and candidate support retention amount .
Advantage of this dataset at scale is estimated using the data scaling ladder framework wherein an effective frontier is constructed by searching for a training recipe that minimizes validation loss and then frozen across distributions. Benchmark aligned proxy-losses then map to capability via the compute. Experiments show that an 80X smaller scorer performs comparably and achieves a 3.9X data-efficiency speedup when continuing pretraining of a 30B-A3B MOE model.
Critical strengths
-
PAIR-19M and PAIR-1.5B perform similarly across various benchmarks and compute budgets despite the difference in size. This provides strong evidence of the scale gap possible in data selection. This addresses one of the core ideas in the paper of the sufficiency of smaller compute in being an effective data selection signal.
-
The analysis of the capability proxies are useful and quite good. Additionally, the experimental setup is solid across the tasks.
Critical weaknesses
-
The works lacks comparison with a few closely related methods such as DSIR by Xie et al (which is not at all cited anywhere) and CoLor-Filter. This would help understand how PAIR performs against closely related methods and allow a better evaluation of the contributions.
-
The inputs to the experiments are under specified or omitted. The reference data is just described as 200B tokens with no info about mixtures, sources, or possible contamination with the benchmarks. Similarly the pre-training on 10T tokens from the 3T candidate corpus is under specified and overlaps with the HQ reference and selected subsets are not available.
Questions for authors
- Could you clarify the data sources ?
- What if any overlaps exist between the candidate, the HQ reference and the selected subsets?
- How was the 10T token pre-trained checkpoint trained? What was the specific data mixture from the 3T candidate corpus ?
- What was the decontamination procedure for the training data to protect evals against the benchmark?
4 comments
Rebuttal by Authors (2/2)
Q2: What overlaps exist between the Candidate, HQ-Reference, and the selected subsets?
Re: By construction, the selected subsets come from the Candidate Corpus, while HQ-Reference is disjoint from both.
To test the case where HQ-Reference data are mixed into the Candidate, we mix them in at 50% to 0.1%, and each method selects a fraction equal to the mixing ratio (new Figure 21, Appendix A.7). The table reports the percentage of HQ-Reference data in each selection (averaged over 10 runs): PAIR-19M recovers the most at every ratio, and its lead grows as the ratio drops.
| Method | 50% | 20% | 10% | 1% | 0.1% |
|---|---|---|---|---|---|
| Random | 50.01 | 20.02 | 10.03 | 1.04 | 0.12 |
| PPL | 35.21 | 7.18 | 3.54 | 0.58 | 0.06 |
| KenLM | 78.41 | 48.76 | 29.39 | 2.53 | 0.20 |
| QuRating | 58.01 | 27.98 | 15.33 | 1.78 | 0.19 |
| DCLM-FastText | 67.20 | 38.20 | 22.91 | 2.40 | 0.24 |
| DSIR | 65.64 | 53.45 | 40.23 | 6.78 | 0.16 |
| CoLoR-Filter | 76.52 | 50.50 | 32.37 | 4.08 | 0.25 |
| PAIR-19M | 78.80 | 58.67 | 44.88 | 12.05 | 2.26 |
Q3: How was the 10T-token checkpoint trained, and on what data mixture?
Re: The 30B-A3B MoE model shares the architecture design of our ladder models and was trained on 10T tokens with the Candidate's mixture (W2 & Q1) under a complete WSD schedule. Its hyperparameters, such as learning rate and batch size, were predicted from scaling fits on our ladder search runs, following [1, 2].
[1] Zhang et al. How Does Critical Batch Size Scale in Pre-training? ICLR 2025.
[2] Li et al. Predictable Scale: Part I, Step Law — Optimal Hyperparameter Scaling Law in Large Language Model Pretraining. 2025.
Q4: What decontamination procedure was used?
Re: Before selection and training, we decontaminated the Candidate Corpus against our evaluation benchmarks following [3, 4]. PAIR and all baselines select only from this decontaminated corpus, so no selected subset contains removed documents.
- 13-gram matching [3]. After removing punctuation, we split English into words and Chinese into characters (each counting as half a gram, to match English granularity), and remove any Candidate document sharing a 13-gram with a test example.
- Source-level filtering [4]. We also exclude data from machine-learning repositories such as huggingface.co.
[3] Brown et al. Language Models are Few-Shot Learners. NeurIPS 2020.
[4] The Microsoft AI Team. MAI-Thinking-1: Building a Hill-Climbing Machine. 2026.
We thank the reviewer again for the thoughtful and constructive review. We are genuinely passionate about this work and have invested resources rarely available in academia to validate it. We welcome further questions, and if our responses have addressed your concerns, we would be grateful if you would consider raising your score.
Thanks to the authors for adding the DSIR experiments and the clarifications provided regarding the data mixture. My main concern about the baselines have now been addressed with the DSIR result at 30B and the new 15B MOE results for CoLOR-Filter. Additionally the clarifications about the data source are informative and help with interpretation (broad re-weighting of categories etc).
I am raising my score.
Response to Reviewer 9upA by Authors
We sincerely thank the reviewer for the careful re-reading of our responses, and we are glad that the added baseline comparisons and the clarified data sources have resolved the concerns raised. We are genuinely passionate about this work and have invested computational resources rarely available in academia to validate it, so the reviewer's recognition means a great deal to us. We thank the reviewer again for raising the score and for the constructive engagement that helped us improve the paper.
Rebuttal by Authors (1/2)
We sincerely thank the reviewer for the constructive suggestions. To respect your time, we keep our responses brief and state the conclusion first.
Re: We apologize for omitting DSIR and the CoLoR-Filter comparison from the initial submission, because CoLoR-Filter had been run only in an early exploration under a different setting, and DSIR's full-corpus selection and continued pre-training had not finished by the deadline. DSIR is now cited in Section 2, both comparisons are added under matched settings, and PAIR clearly outperforms both.
CoLoR-Filter (Figure 20, Appendix A.6).
DSIR (Figure 6, Section 2).
Re: The table below gives each mixture. The setting is controlled: all selected subsets come from the same Candidate Corpus, HQ-Reference is disjoint from it, and compared runs differ only in training data.