- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This article explores to what extent the data-selection results support claims about trained models. Its eight-stage audit treats selection scores as hypotheses, then checks for training parity, uncertainty, reference loss, generated outputs, and possible mechanisms. In pre-training and instruction tuning, conclusions sometimes change depending on the resampling unit or the way NLL is averaged; reference likelihood and decoded reference overlap can also disagree.
Critical strengths
The authors focus on an important problem and the results are interpreted cautiously.
Critical weaknesses
(1) It is unclear what the eight-stage procedure adds beyond rigorous standard evaluation. Random baselines, matched budgets, multiple seeds, clustered uncertainty, and behavioral testing seem to be familiar methods. It is useful to organize them into a claim-evidence ledger, and Table 5 shows how the audit limits claims in these cases. The case studies illustrate the rules, but do not show whether the framework yields more reliable or actionable decisions than careful conventional evaluation. No prospective case reaches the later stages, and Section 6 notes that the impact of the audit on research practice has not yet been measured. Therefore, I believe the added methodological value of this framework has not yet been fully established.
(2) These cases demonstrate the audit process but only support narrow empirical claims. The three Pythia-70M contrasts and the Qwen2.5-0.5B instruction-tuning comparisons each represent one training seed (Section 3; Tables 2-3). The 998 SFT comparison rows cover only 498 distinct prompts, with some baselines shared across comparisons. Therefore, the bootstrap intervals quantify uncertainty over evaluation examples, rather than across new selection or training runs. Training exposure also does not fully match. P3 uses about 1.32% fewer non-padding tokens than its parent and changes filtering and frequency capping; I would consider its worse loss a result of the combined recipe, not just frequency capping. More importantly, S1 reports 24,452 supervised completion tokens for random selection and 39,699 for MMR (Section 3.1; Appendix B). These are not total training-token counts, but matching the number of selected examples clearly does not match the supervised completion-token budget. This complicates attributing the S1 difference solely to the selector.
S1 is an important example, but its interpretation remains limited. Token-weighted reference NLL improves, while ROUGE-L declines; the prompt-macro NLL result remains unresolved (see Table 3). NLL uses Qwen's chat template, while generation uses a literal Instruction/Input/Response format (Section 4.2), so matching example IDs does not give the model matching contexts. The target-test text and references are also used for duplicate-risk control during selection (Section 3.1). This is not necessarily problematic and is not evidence of outcome-based tuning. However, the paper should clarify whether this information was used exclusively for decontamination or also influenced other selection, model, or hyperparameter decisions. ROUGE-L measures reference overlap rather than overall answer quality; human checks cover only 20 deliberately selected pairs judged by one annotator (Section 4.4). Under the saved decoding protocol, MMR also increases the empty outputs from 12/200 to 22/200. This is a concrete behavioral diagnostic, and its robustness under matched templates has not been tested. Earlier scale, placebo, and cross-family numerical claims are associated with a synthetic data generator, and are explicitly excluded from empirical support. The earlier reported training and evaluation runs remain unverified.
The authors acknowledge these limitations and do not claim a general quality reversal. I believe these cases are useful diagnostics, but they are not yet independent, matched-protocol behavioral validation or mechanism tests for the broader framework.
(3) The manuscript contains statements that read as unresolved notes to the authors rather than finished prose. Line 717 states, “Before publication the authors must confirm the annotator’s age category, relationship to the authors, recruitment and consent procedure, and applicable institutional review or exemption determination.” Line 583 states, “The author reports that the annotator was an unpaid linguistics-Olympiad candidate; age, authorship relation, informed consent, and ethics review or exemption remain author-confirmation items. No claim of clearance is made.”
Questions for authors
Please see the weaknesses.