- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper claims that the work of Heineman et al., Zhao et al., and Magnusson et al. contains errors or lack transferability outside of the regime in which they present headline claims, and these errors in particular concern 'noise instrument', which seems like an umbrella term that the authors use to refer to the estimates for random training outcome of LLMs.
Critical strengths
None
Critical weaknesses
- The paper appears dense and difficult to parse, a writing style that's typical to a fully llm-generated paper.
- The paper's characterization of the citations seems inaccurate. The work of Heineman et al. does not claim that the spread of last n run performances serves as a substitute for spread of performance from full LLM training under different training seeds, but merely use both metrics as one of the options to quantify the idea of noise to evaluate benchmarks. This appears a severe mischaracterization of a main motivating prior work of the paper, and raises question on the worth of the study.
- The paper appears to do only data analysis, which does not seem up to standard for ICLR publication.
Questions for authors
The main motivation of paper is study the effects of seed-level noise in training and how it affects training hyperparameter selection in larger-scale training runs, and it seems easily achievable by running new LLM experiments and study transfer. Why not collect these data?
4 comments
Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.
I hold the view that the paper’s peculiar approach to what is otherwise a clear scientific question substantially limits its scientific value.
Presentation aside (though itself central to scientific value) I find the paper's approach and framing puzzling. It selects three isolated works and claims to show that each fail to estimate effects from random noise. In the case of Heineman et al., I mean to highlight that this estimate is not the main claim of the paper. To then refer to each of the paper's methodology "practices" seems to suggest a wider adoption of these approaches, but it's not illustrated clearly in writing. As a result, this paper reads more like helpful reports that should be directed to the cited paper for corresponding authors to re-examine their methodology. But the overall value of the work remains unclear.
Thank you for clarifying this framing concern, and we agree on two scope boundaries. Three case studies do not establish community-wide adoption. Likewise, the checkpoint proxy is not the main claim of Heineman et al., and our audit examines only its use as a noise instrument rather than the broader contribution of that paper. In fact, the purpose of the three cases is not to produce isolated errata, but to instantiate three general inferential designs: inferring across-seed noise from a within-run proxy, inferring an all-source noise floor from single-source arms, and making target-scale recipe decisions from proxy-scale rankings.
The paper's contributions therefore extend beyond errata to methodological conditions on when these designs support decisions. First, Proposition 1 gives the sample complexity for conventional certification of high proxy correlation, showing why small seed budgets can be insufficient even for a perfect proxy. Second, under our variance decomposition, Remark 1 shows that single-source arms cannot identify an all-source noise floor regardless of how many single-source seeds are run; recovering that floor requires bundled or crossed measurements. Third, the target-free separability SNR and decidable share provide diagnostic and operational tools for assessing whether proxy-scale rankings resolve candidate recipes without target-scale runs. On the independent Datablations stack, the unchanged SNR diagnostic likewise predicts pairwise transfer accuracy, while DataDecide's specific thresholds and transition scales remain dataset-specific.
In the next revision, we will separate the paper-specific audits more clearly from the general methodological results.
Response to CHuX
We appreciate the reviewer's time and comments, and address each concern against the primary sources and our experimental record.
1. Characterization of Heineman et al. (Signal & Noise)
The review states that Heineman et al. (NeurIPS 2025) do not present a single run's final-n-checkpoint spread as a "substitute" for multi-seed spread, "but merely use both metrics as one of the options to quantify the idea of noise." In fact, after comparing init, data-order, whole-run checkpoint, and final-n-checkpoint variation, Heineman et al. adopt the final-n-checkpoint measure specifically because multi-seed training is too costly:
They then apply this single-run estimate to DataDecide 60M–750M (§4.1, p. 5), the exact setting our §3 audits. Paraphrasing "a cheap alternative to seed noise" as "a cheap substitute" (§2) is faithful to their text, and our submission already separated their correlational validation (reproduced first: log-space R2=0.82/0.84 vs. published 0.82/0.86; §3, App. B.1) from stricter downstream certification and error-bar uses (§3: "The certification target here is ours, not Signal & Noise's: they reported the 0.9 as a point estimate and never claimed a certificate"; "a use S&N never advertised"). The revision quotes their exact phrase ("a cheap alternative to seed noise"). Thus, there is no mischaracterization of prior work.
2. On "only data analysis"
Where public releases lack required measurements, we ran new pretraining, evaluated unscored checkpoints, and proved formal bounds (App. H, Reproducibility Statement):
3. Why auditing published ladders is required rather than a custom small-scale setup
4. Presentation and writing style
We state unequivocally that the research questions, study design, derivations, runs, and analyses are the authors' own, with bounded AI use disclosed in the paper. We acknowledge that the submitted prose was overly dense and stiff, and that pre-submission AI assistance contributed to this: when compressing three audits, an atlas, and five rules into 9 pages, and after using an LLM to stress-test the draft for objections, we folded too many inline caveats into sentences during polishing rather than consolidating boundaries in Limitations. Importantly, this is purely a presentation issue with no bearing on our analyses or conclusions; the revised PDF rewrites the abstract and §§1–10 for clarity while keeping all experiments, figures, tables, and conclusions unchanged.