acceptodds
Under review as a conference paper at ICLR 2027

Small Models Know Which Data Scales

Abstract

Large-scale pre-training requires scalable data selection that can both construct a high-quality training distribution and assess whether its advantage persists at scale. Existing methods typically score documents independently, which gives little control over the distribution of the resulting corpus. They also validate selected data at a few isolated scales, using metrics that are neither comparable across data distributions nor extrapolable across compute. To address these challenges, we propose PAIR (Pairwise Autoregressive Importance Resampling), which uses paired lightweight language models to characterize the Candidate and high-quality Reference corpora and jointly calibrates resampling strength and retention rate to reconstruct the reference distribution under a finite data budget. To assess scaling value, we introduce the Data Scaling Ladder, which evaluates training distributions with benchmark-aligned proxy losses that follow predictable trends across compute and remain comparable across data distributions, enabling lower-compute experiments to forecast target-scale capability. With only a 19M dense scorer, PAIR achieves a 3.9 data-efficiency speedup in continued pre-training of a 30B-A3B MoE model, closely matching the gain obtained with a 1.5B scorer. Across 1e18–1e21 FLOPs, the two selected distributions follow nearly identical scaling trajectories and preserve their advantages on target-aligned capabilities, while lower-compute fits accurately predict the held-out 1e21-FLOP proxy loss. Together, these results show that training data for large-scale models can be effectively modeled, selected, and evaluated at a small fraction of the scale at which they are ultimately consumed.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
data efficiency, llm pre-training, data scaling ladder, scaling laws

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9upAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses the scaling of data selection in large scale pre-training. It answers the important questions of (1) construction of high quality datasets and, (2) whether the advantages of that data persist at scale.

A target distribution aligned dataset is constructed using a pair of small language models that estimate a document level score, that provides the direction of redistribution. This is then converted into a family of sampling policies from which the dataset is sampled using Gumbel Top-k, after joint calibration (minimization of JS Divergence of the selected and sampled distributions) of two parameters representing the strength of emphasis on target-preferred documents, β\beta and candidate support retention amount ρ\rho.

Advantage of this dataset at scale is estimated using the data scaling ladder framework wherein an effective frontier is constructed by searching for a training recipe that minimizes validation loss and then frozen across distributions. Benchmark aligned proxy-losses then map to capability via the compute. Experiments show that an 80X smaller scorer performs comparably and achieves a 3.9X data-efficiency speedup when continuing pretraining of a 30B-A3B MOE model.

Critical strengths
  1. PAIR-19M and PAIR-1.5B perform similarly across various benchmarks and compute budgets despite the difference in size. This provides strong evidence of the scale gap possible in data selection. This addresses one of the core ideas in the paper of the sufficiency of smaller compute in being an effective data selection signal.

  2. The analysis of the capability proxies are useful and quite good. Additionally, the experimental setup is solid across the tasks.

Critical weaknesses
  1. The works lacks comparison with a few closely related methods such as DSIR by Xie et al (which is not at all cited anywhere) and CoLor-Filter. This would help understand how PAIR performs against closely related methods and allow a better evaluation of the contributions.

  2. The inputs to the experiments are under specified or omitted. The reference data is just described as 200B tokens with no info about mixtures, sources, or possible contamination with the benchmarks. Similarly the pre-training on 10T tokens from the 3T candidate corpus is under specified and overlaps with the HQ reference and selected subsets are not available.

Questions for authors
  1. Could you clarify the data sources ?
  2. What if any overlaps exist between the candidate, the HQ reference and the selected subsets?
  3. How was the 10T token pre-trained checkpoint trained? What was the specific data mixture from the 3T candidate corpus ?
  4. What was the decontamination procedure for the training data to protect evals against the benchmark?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank the reviewer for the constructive suggestions. To respect your time, we keep our responses brief and state the conclusion first.


W1: How does PAIR compare with DSIR and CoLoR-Filter?

Re: We apologize for omitting DSIR and the CoLoR-Filter comparison from the initial submission, because CoLoR-Filter had been run only in an early exploration under a different setting, and DSIR's full-corpus selection and continued pre-training had not finished by the deadline. DSIR is now cited in Section 2, both comparisons are added under matched settings, and PAIR clearly outperforms both.

CoLoR-Filter (Figure 20, Appendix A.6).

  • Relation to PAIR. It is essentially our top-k (β = ∞) ablation (Figure 7), except that its target-side model is fine-tuned from a Candidate-trained model.
  • Results. In 50B-token continued pre-training of a 15B MoE model, it falls below even the Candidate Corpus at nearly every checkpoint, consistent with Figure 7, while PAIR-1.5B outperforms both.
  • Why. It selects documents resembling HQ-Reference's dominant components, which suits only simple targets. For a complex target, its selection is biased: QA rises from 5% to 40%, above even HQ-Reference's 31%.
WebCodeMathKnowledgeQABookReasoningOthers
Candidate Corpus34%20%4%5%5%17%4%0%
HQ-Reference1%15%12%31%31%2%5%3%
PAIR-1.5B21%19%10%14%16%15%5%0%
CoLoR-Filter9%11%20%10%40%2%7%0%

DSIR (Figure 6, Section 2).

  • Continued pre-training. As Figure 6 shows, DSIR clearly underperforms PAIR, with a speedup of only 1.9× versus 3.9× for PAIR-19M. This is because its hashed n-gram features capture only surface lexical statistics, i.e., which words a document uses but not how they are organized, whereas a language model can capture complex, deeper semantics.
  • Favors long documents. Because DSIR's weight accumulates frequency ratios over every n-gram in a document, its selection is biased toward long documents, whereas PAIR averages per-token log-likelihood ratios and avoids this length bias. As the table below shows, Books make up 28% of DSIR's selection, even more than in the Candidate itself (17%).
WebCodeMathKnowledgeQABookReasoningOthers
Candidate Corpus34%20%4%5%5%17%4%0%
HQ-Reference1%15%12%31%31%2%5%3%
PAIR-19M19%16%13%12%24%11%6%0%
DSIR24%5%13%11%12%28%6%0%
  • From-scratch pre-training. For a 1B MoE model, PAIR-19M also reaches lower proxy loss than DSIR. As a full Data Scaling Ladder takes about 220 GPU-days, we will add it in the final version.

W2 & Q1: What are the sources and mixtures of the Candidate, HQ-Reference, and 10T pre-training data?

Re: The table below gives each mixture. The setting is controlled: all selected subsets come from the same Candidate Corpus, HQ-Reference is disjoint from it, and compared runs differ only in training data.

WebCodeMathKnowledgeQABookReasoningOthers
3T Candidate / 10T pre-training34%20%4%5%5%17%4%0%
HQ-Reference1%15%12%31%31%2%5%3%
PAIR-19M19%16%13%12%24%11%6%0%
PAIR-1.5B21%19%10%14%16%15%5%0%
  • 10T pre-training. It shares the Candidate's mixture.
  • HQ-Reference. It has far less web data (1% vs. 34%) and far more knowledge and QA (31% each vs. 5%), so PAIR selects less web data and more knowledge and QA.
  • Imperfect match. PAIR cannot match HQ-Reference exactly, because these categories are a coarse, human-defined partition, and selection is limited by the Candidate's mixture (e.g., only 5% Knowledge vs. 31% in HQ-Reference).
AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q2: What overlaps exist between the Candidate, HQ-Reference, and the selected subsets?

Re: By construction, the selected subsets come from the Candidate Corpus, while HQ-Reference is disjoint from both.

To test the case where HQ-Reference data are mixed into the Candidate, we mix them in at 50% to 0.1%, and each method selects a fraction equal to the mixing ratio (new Figure 21, Appendix A.7). The table reports the percentage of HQ-Reference data in each selection (averaged over 10 runs): PAIR-19M recovers the most at every ratio, and its lead grows as the ratio drops.

Method50%20%10%1%0.1%
Random50.0120.0210.031.040.12
PPL35.217.183.540.580.06
KenLM78.4148.7629.392.530.20
QuRating58.0127.9815.331.780.19
DCLM-FastText67.2038.2022.912.400.24
DSIR65.6453.4540.236.780.16
CoLoR-Filter76.5250.5032.374.080.25
PAIR-19M78.8058.6744.8812.052.26

Q3: How was the 10T-token checkpoint trained, and on what data mixture?

Re: The 30B-A3B MoE model shares the architecture design of our ladder models and was trained on 10T tokens with the Candidate's mixture (W2 & Q1) under a complete WSD schedule. Its hyperparameters, such as learning rate and batch size, were predicted from scaling fits on our ladder search runs, following [1, 2].

[1] Zhang et al. How Does Critical Batch Size Scale in Pre-training? ICLR 2025.

[2] Li et al. Predictable Scale: Part I, Step Law — Optimal Hyperparameter Scaling Law in Large Language Model Pretraining. 2025.


Q4: What decontamination procedure was used?

Re: Before selection and training, we decontaminated the Candidate Corpus against our evaluation benchmarks following [3, 4]. PAIR and all baselines select only from this decontaminated corpus, so no selected subset contains removed documents.

  • 13-gram matching [3]. After removing punctuation, we split English into words and Chinese into characters (each counting as half a gram, to match English granularity), and remove any Candidate document sharing a 13-gram with a test example.
  • Source-level filtering [4]. We also exclude data from machine-learning repositories such as huggingface.co.

[3] Brown et al. Language Models are Few-Shot Learners. NeurIPS 2020.

[4] The Microsoft AI Team. MAI-Thinking-1: Building a Hill-Climbing Machine. 2026.


We thank the reviewer again for the thoughtful and constructive review. We are genuinely passionate about this work and have invested resources rarely available in academia to validate it. We welcome further questions, and if our responses have addressed your concerns, we would be grateful if you would consider raising your score.

Thanks to the authors for adding the DSIR experiments and the clarifications provided regarding the data mixture. My main concern about the baselines have now been addressed with the DSIR result at 30B and the new 15B MOE results for CoLOR-Filter. Additionally the clarifications about the data source are informative and help with interpretation (broad re-weighting of categories etc).

I am raising my score.

AuthorsOpenReview ↗

Response to Reviewer 9upA by Authors

We sincerely thank the reviewer for the careful re-reading of our responses, and we are glad that the added baseline comparisons and the clarified data sources have resolved the concerns raised. We are genuinely passionate about this work and have invested computational resources rarely available in academia to validate it, so the reviewer's recognition means a great deal to us. We thank the reviewer again for raising the score and for the constructive engagement that helped us improve the paper.

Reviewer_Y4PKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PAIR, a targeted data-selection method that uses paired lightweight language models to estimate relative preferences between candidate and reference corpora. It jointly calibrates resampling strength and retention rate to match the reference distribution in scorer-feature space. The paper also introduces a Data Scaling Ladder that evaluates training distributions using benchmark-aligned proxy losses. Experiments include continued pretraining of a 30B-A3B MoE model and from-scratch training across multiple compute budgets. The authors report that data selected by 19M scorers achieves performance broadly comparable to selection using 1.5B scorers.

Critical strengths

The paper presents a relatively comprehensive practical study covering data selection, downstream training, and evaluation across training scales. The combination of large-model continued pretraining, multi-budget from-scratch experiments, and evaluation across multiple capability categories provides useful empirical coverage.

The finding that 19M scorers can select useful training data for a substantially larger model is practically interesting. Their broadly comparable performance to 1.5B scorers suggests that inexpensive models may suffice for data selection in the evaluated setting.

The method provides an explicit criterion for calibrating resampling strength and retention rate. The accompanying ablations offer evidence that balancing target preference and corpus coverage is beneficial compared with overly weak or aggressive selection.

Critical weaknesses

1、Insufficient comparison with the closest related methods.

The incremental benefit over closely related data-selection approaches is not sufficiently established. CoLoR-Filter [1] also selects data using relative losses from auxiliary language models and studies small-to-large selection transfer. DSIR [2] explicitly formulates data selection as target-distribution matching through importance resampling. However, CoLoR-Filter is cited without a direct experimental comparison, while DSIR is not discussed or evaluated. The existing KenLM baseline and the resampling-strength ablation provide partial evidence, but do not replace direct comparisons with these methods. Matched comparisons using the same candidate corpus, reference information, and downstream training budget are needed to establish PAIR's additional value.

2、Comparisons against existing selection methods report proxy losses rather than downstream benchmark results.

Figure 5 compares PAIR with Random, PPL, KenLM, QuRating, and DCLM-FastText using proxy-loss trajectories. It does not report the corresponding downstream benchmark scores for these comparisons. Although the paper includes benchmark results elsewhere, those experiments compare a different set of training distributions. Consequently, the current evidence does not directly establish whether PAIR's proxy-loss improvements over existing selection methods translate into better downstream capabilities. This is particularly relevant because the shared loss–score relationship is primarily examined on the Candidate, HQ-Reference, and PAIR distributions, rather than on the distributions produced by these competing methods.

3、The loss–score mapping lacks independent held-out validation.

Appendix A.4.2 states that all seven compute points, including the highest-compute observations, are used to fit the shared mapping between proxy loss and benchmark score. Therefore, the reported fit does not independently validate the mapping's predictive accuracy on unseen observations. I acknowledge that the highest-compute point is excluded from the compute–loss fit, and that the authors explicitly describe the loss–score mapping as pre-calibrated. Nevertheless, this setup supports loss extrapolation conditional on an already calibrated mapping, rather than a fully held-out demonstration of target-scale capability prediction. An independent evaluation of the mapping is needed to substantiate the stronger forecasting claims.

[1] Brandfonbrener et al. CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training. NeurIPS 2024. https://arxiv.org/abs/2406.10670

[2] Xie et al. Data Selection for Language Models via Importance Resampling. NeurIPS 2023. https://arxiv.org/abs/2302.03169

Questions for authors

Can the authors compare PAIR directly with CoLoR-Filter and DSIR under matched data access and downstream training budgets, reporting both benchmark performance and selection cost? A representative medium-scale experiment would already be informative.

For the models already trained in Figure 5, can the authors report the corresponding downstream benchmark scores? Do the relative rankings under proxy loss agree with the rankings under actual task performance?

Can the authors freeze the proxy design and loss–score mapping before evaluating a held-out compute point or training distribution? How accurate are the resulting benchmark-score predictions, and what calibration data are required in the intended practical setting?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely thank the reviewer for the constructive suggestions and for recognizing our comprehensive study, practical 19M scorers, and explicit calibration criterion. To respect your time, we keep our responses brief and state the conclusion first.


W1 & Q1: How does PAIR compare with CoLoR-Filter and DSIR under matched settings?

Re: We apologize for not including these comparisons in the initial submission, because CoLoR-Filter had been run only in an early exploration with a different setting, and DSIR's full-corpus selection and continued pre-training had not finished by the deadline. Both are now added under the same Candidate Corpus, HQ-Reference, and training budget, and PAIR clearly outperforms both.

CoLoR-Filter (Figure 20, Appendix A.6).

  • Relation to PAIR. CoLoR-Filter is essentially the top-k (β = ∞) setting of our ablation (Figure 7), except that its target-side model is fine-tuned from a Candidate-trained model rather than trained on HQ-Reference alone.
  • Results. In 50B-token continued pre-training of a 15B MoE model, CoLoR-Filter falls below even the Candidate Corpus at nearly every checkpoint, while PAIR-1.5B outperforms both, consistent with top-k underperforming uniform sampling in Figure 7.
  • Why. It selects documents resembling the dominant components of HQ-Reference, which suits only simple targets. For a complex target, its selection becomes biased: QA rises from 5% to 40%, above even HQ-Reference's 31% (table below).
WebCodeMathKnowledgeQABookReasoningOthers
Candidate Corpus34%20%4%5%5%17%4%0%
HQ-Reference1%15%12%31%31%2%5%3%
PAIR-1.5B21%19%10%14%16%15%5%0%
CoLoR-Filter9%11%20%10%40%2%7%0%

DSIR (Figure 6, Section 2).

  • Continued pre-training. DSIR's speedup is only 1.9×, versus 3.9× for PAIR-19M, because its hashed n-gram features capture only surface lexical statistics, not the deeper semantics a language model captures.
  • Favors long documents. Its log-weights accumulate over every n-gram, so Books make up 28% of its selection, more than in the Candidate itself (17%).

PAIR-19M's composition does not exactly match HQ-Reference, because these categories are a coarse, human-defined partition, and the achievable composition is constrained by the Candidate's (e.g., Knowledge is 5% of the Candidate vs. 31% of HQ-Reference).

WebCodeMathKnowledgeQABookReasoningOthers
Candidate Corpus34%20%4%5%5%17%4%0%
HQ-Reference1%15%12%31%31%2%5%3%
PAIR-19M19%16%13%12%24%11%6%0%
DSIR24%5%13%11%12%28%6%0%
  • From-scratch pre-training. For a 1B MoE model, PAIR-19M likewise reaches lower proxy loss than DSIR. As a full Data Scaling Ladder takes about 220 GPU-days, we report only this model within the rebuttal period and will add the full ladder in the final version.

W2 & Q2: Can Figure 5 report downstream scores, and do proxy-loss rankings agree with benchmark rankings?

Re: Figure 5 uses proxy loss by design: its 300M, 500M, and 1B MoE models correspond to 1e18–1e19 FLOPs on the Data Scaling Ladder (Figure 4), where benchmark scores mostly stay near their floor and barely separate distributions, while proxy losses separate them clearly. Where both are measured, the rankings agree:

  • DSIR (Figure 6). PAIR-19M ranks first and the Candidate Corpus last in both the continued pre-training score and the 1B MoE proxy loss.
  • CoLoR-Filter. It falls below PAIR both in proxy loss (top-k in Figure 7) and in benchmark score (Figure 20).

W3 & Q3: Can the frozen loss–score mapping be validated on held-out data, and what calibration data does it need?

Re: Yes. Fitted only on Candidate and HQ-Reference, with all PAIR-19M and PAIR-1.5B points held out, the frozen mapping predicts PAIR's benchmark scores with R² of 0.89–0.99 on all 8 tasks, even beyond the fitted loss range, as on LAMBADA (new Figure 9, Appendix A.4.2).

Calibration data. The mapping is calibrated once on existing distributions spanning the relevant loss range, here 14 runs (Candidate and HQ-Reference at seven budgets each). In production, we likewise fit it in advance and then quantify each new dataset's benchmark gain from its proxy loss, without refitting.


We thank the reviewer again for the constructive review. We are genuinely passionate about this work and have invested resources rarely available in academia to validate it. We welcome further questions, and if our responses have addressed your concerns, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up by Authors

We sincerely thank you again for your detailed review and for the time you have already devoted to our paper. As the discussion period is drawing to a close, we would like to gently follow up and ask whether our responses have addressed your concerns, in particular the CoLoR-Filter and DSIR comparisons under matched settings and the held-out validation of the frozen loss–score mapping.

We are genuinely passionate about this work and invested computational resources rarely available in academia to validate it, which is why we were able to run both baselines at full scale within the rebuttal period. We would be very happy to clarify anything that remains unclear, and if you find your concerns resolved, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up by Authors

Dear Reviewer Y4PK,

We understand this is a busy period and truly appreciate your time. With the discussion period closing soon, we would be very grateful for any brief feedback on whether our responses, including the matched-setting CoLoR-Filter and DSIR comparisons, have addressed your concerns.

This work means a great deal to us, and if your concerns have been resolved, we would be grateful if you would consider raising your score.

Sincerely,

Paper #18442 Authors

Reviewer_dYZNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAIR selects pretraining data with two small LMs, one trained on a 3T-token candidate corpus and one on a 200B-token high-quality reference. Each document is scored by the per-token loss difference s(x), which they state as the Moore-Lewis loglikelihood ratio. Documents are sampled with a function of Beta and keeping a fraction p of corpus. (Beta, p) is chosen to minimize the token Jensen Shannondivergence to the reference.

With 19M scorers, CPT of a 30B MoE for 100B tokens reaches the candidate corpus's final 17 task score 3.9x sooner. The paper also proposes a Data Scaling Ladder, which is a compute-optimal recipes tuned on the candidate corpus at various FLOPs and fit a reasonable power law.

Critical strengths
  • The scale of the main experiment is rare, since it is a 30B MoE CPT'd for 100B tokens, with selection costing about 1/380 of the model pretraining compute. The fact that 19M scorers do as well as 1.5B scorers is practically useful.
  • The proxy study in A.4 provides some good lessons. Smooth scaling is not enough, and a proxy has to keep the tokens that decide the task. For example, answer-only works for code and LAMBADA, but worked solutions are needed for Math.
Critical weaknesses
  • The experiments do not test the setting that motivates PAIR, and the comparison to the reference itself is missing. The reason to rebuild the reference distribution from the candidate pool is to get more reference-like tokens than the 200B reference has. But the CPT run uses 100B tokens, less than the reference, so one could simply train on the reference. HQ-Reference is in Figure 3, but the text never says how PAIR compares to it, and the 3.9× is only against the Candidate Corpus. In case possible, I'd love to see PAIR vs. HQ-Reference stated directly, and one run where the budget exceeds the reference (or a reference shrunk to 10B tokens).
  • Novelty against prior importance resampling is unclear. DSIR (Data Selection for Language Models via Importance Resampling) from Xie et al. is not cited but could be a baseline.
Questions for authors
  • How much do the 19M and 1.5B subsets overlap? And how does their source composition compare with Human-Curated mixture?
  • Potential minor error: The PPL baseline "selects those with the highest perplexity scores". Is this intended?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank the reviewer for the positive assessment and for recognizing the rare scale of our main experiment and the practical value of this work. To respect your time, we keep our responses brief and state the conclusion first.


W1: How does PAIR compare with HQ-Reference, and does PAIR pay off when the budget exceeds the reference?

Re: HQ-Reference is about 3.5× more token-efficient than PAIR-19M, so PAIR pays off once its selected corpus exceeds 3.5× the size of HQ-Reference. The larger the candidate pool and the smaller the HQ-Reference, the more PAIR gains, which is exactly the regime of large-scale pre-training.

  • PAIR vs. HQ-Reference. In Figure 3, HQ-Reference reaches PAIR-19M's final 100B-token score after about 28.8B tokens, so PAIR-19M needs about 3.5× as many tokens to match it; we will state this explicitly in Section 5.2. Which option is better thus depends on three quantities:

    • the candidate pool size,
    • the retention rate, which together determine the selected corpus size, and
    • the HQ-Reference size.
  • Larger pools and smaller references favor PAIR. The table applies PAIR-19M's retention rate to 3T (ours), 10T, and 30T candidate pools, and compares the selected corpus with our 200B HQ-Reference and with the 3B HQ tokens used to train PAIR-19M's target-side scorer, fewer than the 10B suggested by the reviewer. In our 3T/200B setting, the ratio is only 1.88×, so training directly on HQ-Reference is preferable; as the pool grows or the reference shrinks, the ratio rises well above 3.5×, reaching 18.75× for a 30T pool and 125–1,250× for a 3B reference.

Candidate poolSelected corpusvs. 200B HQ-Referencevs. 3B HQ-Reference
3T (ours)375B1.88×125×
10T1.25T6.25×417×
30T3.75T18.75×1,250×
  • This is the regime of frontier pre-training. Frontier models are pre-trained on corpora far larger than our 3T pool: Llama 3 on 15.6T tokens [1], MAI-Thinking-1 on 30T [2], and Qwen3 on 36T [3]. Yet high-quality data remain scarce: MAI-Thinking-1 has only about 300B high-quality math tokens, repeated 5.28× on average [2]. PAIR is built for exactly this setting.

[1] Grattafiori et al. The Llama 3 Herd of Models. 2024.

[2] The Microsoft AI Team. MAI-Thinking-1: Building a Hill-Climbing Machine. 2026.

[3] Yang et al. Qwen3 Technical Report. 2025.


W2: What is PAIR's novelty relative to importance resampling? DSIR is not cited and could serve as a baseline.

Re: We apologize for not including DSIR in the initial submission, because scoring our full 3T-token Candidate Corpus with DSIR and the subsequent continued pre-training had not finished by the deadline. Both are now complete: the revision cites DSIR in Section 2 and compares against it in Figure 6, where PAIR clearly outperforms DSIR in both continued and from-scratch pre-training.

  • Continued pre-training (Figure 6, left). DSIR's data-efficiency speedup is only 1.9×, versus 3.9× for PAIR-19M, because its hashed n-gram features capture only surface lexical statistics, not the deeper semantics captured by a language model.
  • DSIR favors long documents. Its log-importance weights accumulate over every n-gram, so Books make up 28% of its selection (table below), even more than in the Candidate Corpus itself (17%).

PAIR-19M's composition does not exactly match HQ-Reference, both because these source categories are only a coarse, human-defined partition, and because the achievable composition is constrained by the Candidate's: Knowledge, for example, is only 5% of the Candidate, so reaching HQ-Reference's 31% is hard.

WebCodeMathKnowledgeQABookReasoningOthers
Candidate Corpus34%20%4%5%5%17%4%0%
HQ-Reference1%15%12%31%31%2%5%3%
PAIR-19M19%16%13%12%24%11%6%0%
DSIR24%5%13%11%12%28%6%0%
  • From-scratch pre-training (Figure 6, right). For a 1B MoE model pre-trained from scratch, PAIR-19M likewise reaches lower proxy loss than DSIR. One full Data Scaling Ladder takes about 220 GPU-days, so given the limited rebuttal period, we report only this 1B model here and will add the full ladder in the final version.
AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q1: How much do the 19M and 1.5B subsets overlap, and how does their composition compare with the Human-Curated Mixture?

Re: The two subsets largely coincide: 84.9% of PAIR-19M's documents (92.2% of its tokens) are also selected by PAIR-1.5B. Both shift the Candidate mixture in the same direction as expert curation, yet both outperform the Human-Curated Mixture (Figure 3).

WebCodeMathKnowledgeQABookReasoningOverall
Subset Overlap79.8%86.1%91.0%75.3%88.5%94.6%88.0%84.9%

Source composition of each distribution:

WebCodeMathKnowledgeQABookReasoningOthers
Candidate Corpus34%20%4%5%5%17%4%0%
HQ-Reference1%15%12%31%31%2%5%3%
PAIR-19M19%16%13%12%24%11%6%0%
PAIR-1.5B21%19%10%14%16%15%5%0%
Human-Curated6%25%8%27%15%10%3%6%
  • Same direction. Like the experts, both PAIR variants cut web data (34% → 19–21%) and raise knowledge, QA, and math toward HQ-Reference.
  • Different emphasis. Human curation favors code (25%) and knowledge (27%), whereas PAIR-19M favors QA (24%) and math (13%), tracking HQ-Reference (31% and 12%).
  • Category ratios alone do not determine data value. The Human-Curated Mixture is closer to HQ-Reference in web and knowledge and reflects long-term expert iteration, yet both PAIR variants outperform it. PAIR matches the target at the document level, choosing which documents to keep within each category, which source-level ratios cannot express, and needs no hand-specified mixing ratios.

Q2: The PPL baseline selects the highest-perplexity documents. Is this intended?

Re: Yes, this is intended. The PPL baseline keeps the documents with the highest per-token loss under PAIR's 19M Candidate-side scorer alone; we will make this explicit in Appendix A.3.

  • Why high perplexity. Under a model trained on the Candidate distribution, perplexity reflects how much new information a document carries; low-perplexity text is highly predictable, mostly templates or duplicates. Prior work likewise finds that retaining easy, low-loss examples brings little benefit when data are abundant [4–6], and [6] finds that keeping high-perplexity documents works best on the multi-domain Pile.
  • Why it still underperforms. The likely cause is its scorer, a 19M dense model far smaller than the 290M–1B MoE models we train, so perplexity alone cannot reliably measure a document's value to them. PAIR uses scorers of the same size but contrasts Candidate-side and target-side losses, which largely cancels the difficulty caused by the scorer's limited capacity.

[4] Sorscher et al. Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning. NeurIPS 2022.

[5] Mindermann et al. Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. ICML 2022.

[6] Ankner et al. Perplexed by Perplexity: Perplexity-Based Data Pruning with Small Reference Models. ICLR 2025.


We thank the reviewer again for the encouraging and constructive review. We are genuinely passionate about this work and have invested resources rarely available in academic research to validate it. We welcome any further questions, and if our responses have addressed your concerns, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up by Authors

We sincerely thank you again for your careful review and for the time you have already devoted to our paper. As the discussion period is drawing to a close, we would like to gently follow up and ask whether our responses have addressed your concerns, in particular the comparison against HQ-Reference and the newly added DSIR baseline.

As you kindly noted, the scale of our main experiment is rare: we are genuinely passionate about this work and invested computational resources rarely available in academia to validate it. We would be very happy to clarify anything that remains unclear, and if you find your concerns resolved, we would be grateful if you would consider raising your score.

Comments on the paper

3 comments

Dear reviewers,

Please check your assigned papers for author responses. If the authors have addressed any of your questions or concerns, please leave a brief comment for them indicating which points have been resolved and which, if any, remain open. Even if your assessment is unchanged, a short follow-up is helpful.

Best, AC

Dear Reviewers,

The Author-Reviewer Discussion period is coming to an end soon. Please actively participate in the rebuttal discussion with the authors after carefully reading the authors' responses.

Best, AC

AuthorsOpenReview ↗

Author Final Remarks

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for the thoughtful evaluation and constructive discussion. Reviewers dYZN and 9upA confirmed that their concerns were resolved, with Reviewer 9upA raising their score; both now rate the paper 3. For Reviewer Y4PK, we addressed all three points in detail, but did not hear back during the discussion period.

We believe this work reframes data selection as a distribution-level problem and validates it at a scale rarely available in academic research. We hope this provides the community with a practical and measurable way to build pre-training data at frontier scale.

  1. Rather than scoring documents one by one, our work optimizes the distribution of the selected corpus.
  2. Rather than evaluating at a single small scale, our work validates the selected data through 30B-A3B MoE continued pre-training and a Data Scaling Ladder spanning 1e18 to 1e21 FLOPs.

During the discussion, we addressed the main concerns raised by the reviewers by adding comparisons with DSIR and CoLoR-Filter under matched corpora and training budgets, validating the frozen loss–score mapping on fully held-out distributions, and clarifying the data sources, mixtures, overlaps, and decontamination procedure. All revisions are highlighted in the updated PDF.

We are genuinely passionate about this work and have invested substantial compute and effort in it, because we believe data construction deserves the same rigor as model scaling. We sincerely thank the Area Chair and all reviewers again for their time and for helping us strengthen the paper.

Sincerely,

Paper #18442 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.