acceptodds
Under review as a conference paper at ICLR 2027

Quality Is Not One Thing: Auditing, Distilling, and Blending LLM-Judge Annotations for Pre-Training Data Selection

17 views

Abstract

Curation of data in language model training commits the corpus to particular notions of 'quality'. We study that commitment in three parts: judging different notions of quality, optimising them in a mix on a probability simplex, and training and evaluating on the filtered corpora. An LLM-judge scores a fixed one-million-document English Common Crawl sample using 22 unique prompts, and we distil each set of scores into lightweight classifiers. We train three classifier types (fastText, a frozen embedding with an MLP head, and fine-tuned BERT) for a total of 192 models. The scores share a dominant content factor (first principal component of variance) but can also be very distinct, e.g., conversationality is largely uncorrelated with the other tested axes. Disagreement between scores is largest among the documents different classifiers would label `high quality'. Within-document spread across standardised scores rises from in the bottom decile of mean score to in the top, and classifiers with similar aggregate metrics keep different top- sets (Jaccard –). We optimise a weighted mix of five of these scores on a probability simplex. We show a weighted sum of the raw scores performs worse than a corpus selected only with the Dolma 3 quality filter. However, putting each score on a shared rank scale and keeping documents above a Dolma 3 classifier cutoff lets those dimensions combine more effectively. Training on this latter mix, the Dolma 3 classifier filtered data, and a random crawl sample in a B dense model and a B-AB mixture-of-experts model shows the the raw weighted sum does not beat Dolma 3 classified, while the rank-scaled mix does and also has some positive transference to performance outside of tested English language capabilities (including coding, mathematics, and German language).

Primary area:
foundation or frontier models, including LLMs
Keywords:
pre-training, text classification, LLM-as-a-judge, data curation, classifier distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Am42OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this work, the authors explore the measure of quality in the pretraning data selecton. The quality should contain different signals, and combining these signals can help to balance the quality from different perspectives. The experiment results demonstrate performance gain over random and Dolma3-based baselines.

Critical strengths
  1. The data selection problem is important in large language model training, and quality-based selection is widely adopted in both industrail and academic areas.
  2. Extensive experiments are conducted to evaluate the data quality from different perspectives.
Critical weaknesses
  1. The performance gain seems modest compared to baseline for MoE models. There is no experiments on varying data budgets, and other data domains.
  2. The data scoring process requires further investigation and justification. The Qwen3-32B model seems small, compared with the 7B-MoE model and existing LLM-as-a-Judge works. Typically, a 70B (e.g. Llama-3-70B-Instruct) or state-of-the-art open source model (e.g., GPT) is preferred for the precision of the result.
  3. The score distribution is not aligned with each other. For example, the mean for orthography is 4.35, which makes the signals unbalanced. The rubrics should be adjusted to make the mean aligned.
  4. The mechanism behind the improvement is obscure. The discussion in page 7 says the performance gain originates from the mix on the simplex, but the limitation also indicates that the gate of Dolma 3 and score normalization are changed at the same time. A more rigorous experiment should be conducted to locate the reason for performance improvement.
  5. The dimension definition does not align with the experiment result. Several dimensions are construct from the benchmark usefulness, but there is result to show that these dimensions are effective.
  6. Although 192 scorers are trained, only 5 signals are used in the experiments and many of them received a 0 weight. The effectiveness of the dimensions should be checked.
  7. The baseline comparison is insufficient. Standard data selection methods, such as PPL, DSIR, QuRating, Meta-Rater, and other scorer-based methods are not incorporated.
  8. Although the performance are given in BPB, the accuracy of the benchmark should be provided.
  9. The presentation of the work should be improved. There are several typos, e.g., there is no Section 2.2 but it appears in the appendix.
Questions for authors
  1. How would the method perform on other data domains, such as code, multimodal?
  2. Is this method sensitive to the selection of LLM-judges?
  3. Why only 5 are selected among 192 signals, and why not continue to explore other signals?
Reviewer_TZTNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper audits LLM-based quality annotations for pretraining data, distills them into lightweight classifiers, and studies how to combine their scores. It compares quality definitions and classifier backends on a fixed English Common Crawl sample, then searches for mixtures of quality signals. A blend using rank normalization and Dolma-based gating is evaluated through language-model training at two scales.

Critical strengths

There is substantial experimental groundwork here. The fixed-corpus comparison of quality definitions, prompts, and classifier backends provides a useful basis for studying data selection. I appreciate the effort to connect the annotation audit to actual pretraining experiments, including confirmation runs and a German transfer check. Reporting the failure of the raw-score mixture is also useful.

Critical weaknesses
  1. I am not convinced that the paper establishes enough new knowledge beyond existing data-curation practice. Multiple quality criteria, judge-to-classifier distillation, and combining quality signals are already well established. QuRating studies several quality criteria and their selection effects. Nemotron-CC, Section 3.3 and Appendix A explicitly compares the high-quality sets selected by FineWeb-Edu and DCLM: their intersection is only 10.1% of their union. It also uses rank-based score buckets and classifier ensembling. The present audit expands the coverage and quantification, but observing disagreement among quality definitions or selected sets does not yet explain which signals should be trusted or combined. I would like a clearer account of what the audit changes about how we select data. The related work should also discuss CritiQ, which learns quality criteria from human preferences and distills a scorer; this is directly relevant to the choice and learning of quality definitions.

  2. The mixture experiments do not isolate the explanation offered for their success. The raw and successful recipes differ in gating, normalization, and the token-budget rule. The limitations acknowledge this, but Section 5 still describes the successful blend as differing only in making the axes commensurable. The current comparison cannot identify normalization as the reason for the improvement. The strongest single-score comparison also needs confirmation: Figure 3 shows the chat-like vertex close to the optimum, with overlapping error bars, whereas the final confirmation focuses on the optimum, Dolma, and random selection. A controlled comparison of raw scores, rank normalization, gating, and their combination, together with the strongest single score under that same protocol, would address the central claim much more directly.

  3. The validation leaves uncertainty about how well the chosen recipe transfers. The English benchmark suite is used to select the mixture and to report its main English performance. New training seeds do not provide an evaluation independent of recipe selection. The 400M improvement is encouraging, but the larger mixture-of-experts (MoE) comparison is less decisive: the reported posterior probability of improvement over Dolma is 0.62, with 0.36 probability of practical equivalence. The German check is useful, but does not settle whether the English-selected recipe improves on unseen English evaluations.

I recommend rejection in the current form. This work has a solid empirical foundation, and I encourage the authors to build on it. A stronger version would use the audit to establish a clear, tested conclusion about choosing or combining quality signals, with controlled evidence for the source of the gains. That would make the contribution much easier to assess in a future submission.

Questions for authors
  1. What new data-selection conclusion follows from this audit beyond the different quality standards and complementary selections already documented in prior work?
  2. Under a fixed corpus, tokenizer, and training-token budget, how much do rank normalization and gating each contribute? How does the final blend compare with the strongest single-score arm from the same protocol, particularly chat-like?
  3. Which evaluations were excluded from recipe selection, and how does the selected recipe perform on them?
Reviewer_LxjmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to optimize a mix of quality scores towards good data quality.

Critical strengths

The paper investigates many different notion of pre-training document quality and trains small models to investigate data quality. The paper analyzes the relationship between these quality notions and proposes reasonable methods for combining these scores for an overall data selection strategy.

Critical weaknesses

The paper reads more as a loose experiment log than a paper written for conference publication. In my opinion, this warrants clear rejection.

The experimental setup makes esoteric claims like "The single one to beat is Dolma 3: in the background sweep of Appendix A its top quartile is tied for the best English general aggregate and has the lowest code bits-per-byte of the sixteen classifiers, so another off-the-shelf dimension is not a route past it." In fact, I think there are other reasonable baselines here and this argument is circular and makes no sense.

The experimental optimization detail of the score mixing in Section 4 are unclear. While I can follow that ten models are pre-trained on different slices of the data, it is unclear what the "gate" is and how an optimal "blend" is extrapolated. No reference to prior literature like RegMix is made in this section. I particularly question the authors' judgement in including the following paragraph discussing the results:

The German aggregate is the unweighted mean of those four benchmarks, over seeds that scored all four. At 400M it is 4.505 for the mix (three seeds; one seed’s final eval is missing Global- MMLU), 4.584 for Dolma 3, and 5.220 for random (four seeds each). Both filters lean ahead of random, P (better) = 0.75 for the mix and 0.76 for Dolma 3, and the mix does not separate from Dolma 3: P (better) = 0.54, P (ROPE) = 0.10, P (worse) = 0.37. The lean towards the mix is in the two QA tasks (P (better than Dolma 3) = 0.70). On German code the mix sits behind Dolma 3 (P (worse) = 0.70), while Dolma 3 leans ahead of random (P (better) = 0.77). At the MoE the four-benchmark mean is 0.827 for the mix, 0.852 for Dolma 3, and 0.847 for the single random seed. The mix leans ahead of Dolma 3 (P (better) = 0.77) and of random (P (better) = 0.76). Dolma 3 against that random seed is P (better) = 0.26, P (ROPE) = 0.30, P (worse) = 0.45. The MoE edge is again in QA (P (mix better than Dolma 3) = 0.88); German code is spread across the trichotomy (0.41, 0.31, 0.27). Transfer is partial. The English-selected mix keeps German bits-per-byte at or below the unfiltered crawl, and at the MoE it is the lowest of the three. At 400M the four-benchmark mean does not improve on Dolma 3, because the code tasks go the other way.

Most of the attempted and documented methods do not yield any performance gain despite substantial methodological overhead.

Questions for authors

n/a

1 comment

We thank Reviewer Lxjm for their frank review. We are glad they find that the paper "proposes reasonable methods for combining these scores for an overall data selection strategy". The central concern, that the submission read "more as a loose experiment log than a paper written for conference publication". We have considered this feedback when completing our revision, wherein we redesigned and reran the mixture experiments more systematically to provide clearer insights into the techniques we propose and their potential benefits. We also revised the text of the paper.

In response to the critical weaknesses listed, we have revised the paper and provide the following response (in order of the paragraphs of the review):

  1. The revision is organised around the three stages of Fig. 1, with one section and one contribution for each. Sections 3 and 4 now have a more systematic experimental design. We train five fleets of 52 blends which change one factor at a time (the document-quality classifier, whether it gates the other scores, and how strict the gate is). Each subsection of §4 answers one question (§4.1–4.5), the per-fleet detail has moved to Appendix I, and all subsections are now numbered.
  2. We agree that the sentence you quote was poorly argued. We have removed it. The introduction now states the main scope and focus of our submission. We work within the framework of small proxy classifiers and ask whether a mixture of annotations distilled into such classifiers can perform better than any of its singular consituents. Appendix A is now presented as background for the choice of reference, and is what motivated us to pursue this work. We added the DCLM fastText classifier as a second document-quality score, with two fleets of its own. The introduction now cites perplexity-based pruning, DSIR, QuRating, CritiQ and Meta-rater as other approaches to selection. We did not run these baselines for comparison, since we wish to focus on the comparison between a mixture and the single classifiers it is built from, which is also what has been commonly used in public pre-training data recipes.
  3. §3.1 now defines the combined score in one equation (Eq. 1). Each of five per-document scores is put on a shared rank scale. The gate is a hard threshold on the document-quality classifier which multiplies a score before ranking, so that documents below the threshold are tied at the bottom of that column, and a blend is a weighted mean of the columns with weights on the probability simplex. §3.2 describes how the optimum is found. For each fleet we fit a degree-2 ridge regression and a Gaussian process from the blend weights to a benchmark mean of relative bits-per-byte (BPB), on the 52 trained blends and the one-hot corners, keep the fit with the higher leave-one-out R², minimise it over the simplex, and train the predicted optimum with three seeds. This is regression-based mixture search, and we now cite RegMix there. We also report where the procedure is weak. The surrogates have a leave-one-out R² of 0.06–0.85 (Table 3), and under the strict gate the selected optimum is an extrapolation which measures behind its fleet's own corner (§4.1). We also agree that the German paragraph you quote should not have been in the main text in that form. §4.4 is now a short paragraph which states the finding, and the per-benchmark values and posteriors are in Appendix I.6.
  4. In the revision all five trained optima are below the Dolma 3 top-25% filter, by 2.6% to 9.7% in BPB over five English benchmarks, each with a posterior probability of being better of at least 0.98 over three seeds (Table 3). The choices which appear to improve downstream performance are which construct one commits to and how strictly one selects on it, with mixing these rather than relying on any singular construct shown to make it possible to improve this even further.

We are very grateful for the directness of the feedback, which led us to more systematically organise the experiments and improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.