acceptodds
Under review as a conference paper at ICLR 2027

Works Where Calibrated, Fails Where Misused: Auditing Small-Scale Pretraining's Seed-Noise Instruments

Abstract

Small-scale pretraining selects data recipes under a handful of seeds, read by three instruments. On public per-seed assets (plus two small self-run controls; Appendix H), we audit each where it is used: two of the three reproduce at home (the floor's per-seed data were never released), and each fails where misused — differently for each. The checkpoint-noise proxy of Signal & Noise reproduces at home, but its advertised correlation is beyond certification at realistic seed budgets under the conventional Fisher- certificate (a perfect proxy reads – against ), and it under-reads seed noise in magnitude on the accuracy readout ( median, up to , reversed on perplexity). An init-only noise floor cannot identify the all-source floor from single-source arms at any budget; only a bundled or crossed initorder design can. DataDecide's recipe ranking reproduces its exactly at its recommended scale, but on the accuracy readout its deconvolved separability SNR stays at or below up to 60M, and at 4M the macro ranking inverts against the 1B ranking (; robust to recipe resampling, borderline once seed noise is priced in) and seed averaging does not rescue it; the 11-domain perplexity readout does not invert. We release code, a per-cell noise atlas, and report label–config-mismatched files in PolyPythias's release.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pretraining data selection, evaluation methodology, seed noise, experimental audit, variance decomposition, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CHuXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims that the work of Heineman et al., Zhao et al., and Magnusson et al. contains errors or lack transferability outside of the regime in which they present headline claims, and these errors in particular concern 'noise instrument', which seems like an umbrella term that the authors use to refer to the estimates for random training outcome of LLMs.

Critical strengths

None

Critical weaknesses
  • The paper appears dense and difficult to parse, a writing style that's typical to a fully llm-generated paper.
  • The paper's characterization of the citations seems inaccurate. The work of Heineman et al. does not claim that the spread of last n run performances serves as a substitute for spread of performance from full LLM training under different training seeds, but merely use both metrics as one of the options to quantify the idea of noise to evaluate benchmarks. This appears a severe mischaracterization of a main motivating prior work of the paper, and raises question on the worth of the study.
  • The paper appears to do only data analysis, which does not seem up to standard for ICLR publication.
Questions for authors

The main motivation of paper is study the effects of seed-level noise in training and how it affects training hyperparameter selection in larger-scale training runs, and it seems easily achievable by running new LLM experiments and study transfer. Why not collect these data?

4 comments
AuthorsOpenReview ↗

Response to CHuX

We appreciate the reviewer's time and comments, and address each concern against the primary sources and our experimental record.

1. Characterization of Heineman et al. (Signal & Noise)

The review states that Heineman et al. (NeurIPS 2025) do not present a single run's final-n-checkpoint spread as a "substitute" for multi-seed spread, "but merely use both metrics as one of the options to quantify the idea of noise." In fact, after comparing init, data-order, whole-run checkpoint, and final-n-checkpoint variation, Heineman et al. adopt the final-n-checkpoint measure specifically because multi-seed training is too costly:

  • §3.1, p. 4: "These results lead us to define noise as the relative standard deviation of the final n checkpoints, as this requires no additional training cost and only uses the final n checkpoints rather than the full training curve."
  • App. A.3.1, p. 17: "seed noise and data order noise are too expensive to measure …, so only calculating step-to-step noise is a reasonable estimate for the modeling noise. Thus, we use step-to-step noise in as our estimate of the modeling noise."
  • §6, p. 10: "… and show the step-to-step noise is a cheap alternative to seed noise."

They then apply this single-run estimate to DataDecide 60M–750M (§4.1, p. 5), the exact setting our §3 audits. Paraphrasing "a cheap alternative to seed noise" as "a cheap substitute" (§2) is faithful to their text, and our submission already separated their correlational validation (reproduced first: log-space R2=0.82/0.84R^2=0.82/0.84 vs. published 0.82/0.860.82/0.86; §3, App. B.1) from stricter downstream certification and error-bar uses (§3: "The certification target here is ours, not Signal & Noise's: they reported the 0.9 as a point estimate and never claimed a certificate"; "a use S&N never advertised"). The revision quotes their exact phrase ("a cheap alternative to seed noise"). Thus, there is no mischaracterization of prior work.

2. On "only data analysis"

Where public releases lack required measurements, we ran new pretraining, evaluated unscored checkpoints, and proved formal bounds (App. H, Reproducibility Statement):

  • 54 from-scratch 20M runs (~103B tokens, ~200 A100 GPU-hours). On DataDecide's OLMo 20M setup (1.91B tokens/run), we split init and order RNG streams to train a 4×4×34\times 4\times 3 crossed grid (48 runs, 3 identical-seed replicates/cell) plus 6 bundled runs on 6 tasks, separating init, order, interaction, and hardware run noise (df=32\mathrm{df}=32) and testing the proxy at 10 seeds (Tables 22–23).
  • 16 PolyPythias 160M evaluations. We scored EleutherAI's 3 init-only, 3 order-only, and 10 bundled 160M models on 6 tasks to add a 160M decomposition point (Table 21).
  • Formal bounds. We prove finite-sample power limits for certifying noisy proxy correlations (Prop. 1, Cor. 1; App. B.4) and non-identifiability of all-source noise floors from single-source arms of any size (Remark 1; §4, App. C.3).

3. Why auditing published ladders is required rather than a custom small-scale setup

  • Audit target. Rather than generic hyperparameter tuning, we audit three specific instruments used in practice (S&N's proxy, single-source floors, and DataDecide's ranking); evaluating the released multi-seed records that generated them tests their empirical basis directly.
  • 1B multi-seed compute scale. Testing transfer requires multi-seed pretraining at both proxy scales and the 1B target: DataDecide's 1B tier alone is 25 recipes × 3 seeds = 75 runs at ∼7.1×1020\sim 7.1\times 10^{20} FLOPs/run (∼5×1022\sim 5\times 10^{22} FLOPs, tens of thousands of A100 GPU-hours before the 13 smaller scales). Auditing the 4M–1B ladder plus targeted 20M crossed runs (App. H) addresses both scale and identification.
  • Second-stack replication (rebuttal analysis). On Datablations (Muennighoff et al., NeurIPS 2023; Megatron-DeepSpeed / GPT-2, 14 recipes × 5 seeds, 2.8B/4.2B → 8.7B), our unchanged SNR predicts pairwise transfer accuracy at Spearman ρ=+0.839\rho = +0.839 (p<0.001p \lt 0.001, n=28n = 28) vs. +0.813 (n=120n = 120) on DataDecide, confirming cross-stack generalization.

4. Presentation and writing style

We state unequivocally that the research questions, study design, derivations, runs, and analyses are the authors' own, with bounded AI use disclosed in the paper. We acknowledge that the submitted prose was overly dense and stiff, and that pre-submission AI assistance contributed to this: when compressing three audits, an atlas, and five rules into 9 pages, and after using an LLM to stress-test the draft for objections, we folded too many inline caveats into sentences during polishing rather than consolidating boundaries in Limitations. Importantly, this is purely a presentation issue with no bearing on our analyses or conclusions; the revised PDF rewrites the abstract and §§1–10 for clarity while keeping all experiments, figures, tables, and conclusions unchanged.

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.

I hold the view that the paper’s peculiar approach to what is otherwise a clear scientific question substantially limits its scientific value.

Presentation aside (though itself central to scientific value) I find the paper's approach and framing puzzling. It selects three isolated works and claims to show that each fail to estimate effects from random noise. In the case of Heineman et al., I mean to highlight that this estimate is not the main claim of the paper. To then refer to each of the paper's methodology "practices" seems to suggest a wider adoption of these approaches, but it's not illustrated clearly in writing. As a result, this paper reads more like helpful reports that should be directed to the cited paper for corresponding authors to re-examine their methodology. But the overall value of the work remains unclear.

Thank you for clarifying this framing concern, and we agree on two scope boundaries. Three case studies do not establish community-wide adoption. Likewise, the checkpoint proxy is not the main claim of Heineman et al., and our audit examines only its use as a noise instrument rather than the broader contribution of that paper. In fact, the purpose of the three cases is not to produce isolated errata, but to instantiate three general inferential designs: inferring across-seed noise from a within-run proxy, inferring an all-source noise floor from single-source arms, and making target-scale recipe decisions from proxy-scale rankings.

The paper's contributions therefore extend beyond errata to methodological conditions on when these designs support decisions. First, Proposition 1 gives the sample complexity for conventional certification of high proxy correlation, showing why small seed budgets can be insufficient even for a perfect proxy. Second, under our variance decomposition, Remark 1 shows that single-source arms cannot identify an all-source noise floor regardless of how many single-source seeds are run; recovering that floor requires bundled or crossed measurements. Third, the target-free separability SNR and decidable share provide diagnostic and operational tools for assessing whether proxy-scale rankings resolve candidate recipes without target-scale runs. On the independent Datablations stack, the unchanged SNR diagnostic likewise predicts pairwise transfer accuracy, while DataDecide's specific thresholds and transition scales remain dataset-specific.

In the next revision, we will separate the paper-specific audits more clearly from the general methodological results.

Reviewer_ro19OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript audits three tools used in small-scale pretraining: checkpoint-based noise estimation, single-source seed-noise floors, and small-to-large recipe ranking. Through public-data reanalysis and small training controls, it identifies metric-dependent proxy miscalibration, the inability to identify bundled noise from single-source experiments, and a downstream macro-ranking inversion at 4M. It also provides a noise atlas and diagnostics for assessing whether proxy comparisons are statistically resolvable.

Critical strengths

A major strength is the careful distinction between reproducing a method in its original setting and identifying failures under extrapolation. The authors reproduce DataDecide’s ranking performance at 150M and the checkpoint proxy’s original results before examining magnitude errors, metric dependence, and aggregation-specific failures. This makes the audit balanced and credible. The robustness checks are also substantial, covering alternative metrics, seed and recipe resampling, dependence-aware comparisons, and a replicated 4 × 4 × 3 initialization-by-data-order design. Appendix H distinguishes run-level nondeterminism from interaction effects, while Section 5 separates pairwise ranking accuracy from the regret of selecting the top recipe. These controls address several plausible alternative explanations.

Critical weaknesses

My main concern is the precision of the statistical evidence. The noise atlas uses only three seeds per cell, making individual standard-deviation estimates uncertain; having many correlated cells does not compensate for limited replication within each recipe. The 4M inversion is also sensitive to how uncertainty is accounted for: its correlation interval becomes [−0.68, 0.08] after propagating seed uncertainty, while the estimated seed-noise attribution has a broad interval of [8.3%, 57.0%]. These results are informative, but support exploratory conclusions more strongly than precise numerical prescriptions.

The generality of the findings is another limitation. The recipe-ranking audit relies on DataDecide’s 25 recipes, training stack, and evaluation suite, without an independent replication on a second stack. Consequently, the proposed scale and SNR thresholds may not transfer beyond this setting. Moreover, separability does not imply ranking correctness: the target-free SNR can be high even when the proxy ordering disagrees with the target, as the task-level bits-per-byte analysis illustrates. Finally, some recommendations are stronger than the supporting evidence. The correlation obstruction applies to the specified Fisher-z certificate rather than all possible validation procedures, and the compute-matched advantage of increasing scale has an uncertainty interval that includes zero. Dense checkpoints, detrending, and seed-semantics conversion are examined separately, but their combination is not prospectively validated as a complete procedure. Similarly, the practical value of diagnostic-driven abstention remains unclear without an explicit cost for withholding a potentially useful choice.

Questions for authors

Which inversion and SNR conclusions survive joint uncertainty over seeds, recipe families, and tasks, rather than recipe resampling alone? Does the diagnostic predict selection regret on held-out recipe families or another training stack? How should practitioners choose between scaling, adding seeds, and abstaining when abstention has a cost and the compute-matched scale-first interval includes zero?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ro19

We appreciate the reviewer's constructive review, and thank the reviewer for noting that separating original-setting reproduction from extrapolation failure makes the audit balanced and credible, and that our controls rule out alternative explanations. We address each point below.

1. Precision at n=3n=3 and joint uncertainty across families, tasks, and seeds (W1 & Q1)

First, the σ\sigma atlas never uses one 3-seed cell SD as a standalone estimate: §6 reports cross-cell quantiles and tests heterogeneity after subtracting the n=3n=3 (χ22\chi^2_2) floor (p=0.005p=0.005), corroborated by 10-seed PolyPythias runs. Second, across B=2,000B=2,000 resamples over families, tasks, and seeds (keeping empirical 3-seed SD on SD rows to avoid n=3n=3 zero-variance draws; App. D.4):

ConclusionPointFamily 95% CIJoint 95% CI / prob.Survives?
SNR → pairwise acc. (120 cells)+0.813[+0.60, +0.82][+0.42, +0.88]Yes (100% > 0)
Recipe-varying σ\sigma (4M C4-en)2.29×[1.30, 2.98][1.30, 2.98] (p=0.015p=0.015)Yes
Abstain ≤60M (BY = 0)0%91%–100%89%–98%Yes (4M–60M)
60M→90M SNR jump (> 1)2.12×[1.70, 2.81]96.3% (> 1)Yes
4M inversion (r<0r<0)−0.56[−0.72, −0.11][−0.64, +0.56]Suite-dependent (96% < 0 fam×seed; widens on 10 tasks)

Takeaway. All four diagnostic and heterogeneity claims survive joint uncertainty; only the 4M inversion widens over the 10 OLMES tasks, confirming §5 (App. D.4) that it is a cross-task aggregation effect.

2. Generalization to held-out families and a second stack (W2 & Q2)

Without retuning, we tested the unchanged deconvolved SNR formula on (a) 252 held-out panels (126 5-vs-5 family splits of DataDecide, 132 cells/half) and (b) Datablations (Muennighoff et al., NeurIPS 2023; Megatron / GPT-2, 14 recipes × 5 seeds, 2.8B/4.2B → 8.7B, 14 tasks, 28 cells):

Out-of-sample testOutcomeMedian [95% range] / ρ\rhoConsistency
Held-out (252 halves)SNR vs. pairwise acc.+0.728 [+0.60, +0.82]> 0 in 252/252 (100%)
Held-out (252 halves)SNR vs. lower regret+0.361 [+0.18, +0.58]> 0 in 252/252 (100%)
Held-out (SNR ≤ 1.3)Abstained vs. retained+1.53 pts regret, −24 pp acc.Worse in 252/252 (100%)
Datablations (GPT-2)SNR vs. 8.7B acc.+0.839 (p<0.001,n=28p<0.001, n=28)Matches +0.813 (2.8B: +0.92, 4.2B: +0.78)

Takeaway. Target-free SNR generalizes zero-shot to held-out families (100% > 0 on accuracy and regret) and a second stack (ρ=+0.839\rho=+0.839). Scale cutoffs (60M vs. 90M) depend on stack/suite, while the SNR diagnostic transfers; per §5 (and Datablations' OSCAR sub-panel), separability is a necessary noise filter, not a guarantee of alignment.

3. Scaling, adding seeds, and abstaining under explicit costs (W2 & Q3)

Per Rule 3, the action depends on the goal (certified pairwise claims vs. forced single pick). Replaying strategies at matched FLOPs across 25 recipes (B=2,000B=2,000 joint resamples; random-fallback loss = 2.71 pts):

Goal & budgetActionMatched-FLOP outcome (1B top-1 loss & acc.)
1. Pairwise claimsAbstain (BY=0 / SNR≤1.3)Prevents 34% inflated discoveries at 60M (§6, Table 16); 50% BY needs 11–39 seeds at ≤60M vs. 5–6 at 90M+.
2. Pick, can escalateScale up (6M×1, ADAPT→90M)Saves +3.98 pts (+14.8 pp acc.) vs. 4M×3; ADAPT (1.86 pts) beats 150M×2 (4.66 pts) by +2.80 pts (87%–89% tie/win).
3. Pick, capped 10M–60MUncertified raw argmaxBeats fallback by +0.9 to +2.7 pts; 60M×3 and 90M×1 pick the same recipe (0.00 gap; +1.9 pp acc. at 90M×1).

Takeaway. Abstain from certified claims when SNR/BY fail; for a forced pick, scale above 60M if budget allows, and take the uncertified raw argmax only when capped at 10M–60M.

4. Correlation-bound generality and Rule 1 validation (W2)

  • Neyman–Pearson bound. Under Prop. 1, the exact Neyman–Pearson oracle bound (all nuisance parameters known) proves that at 3 seeds (k=10,m=5,n=3,σa=0.74k=10, m=5, n=3, \sigma_a=0.74), no level-0.05 test of λ≥0.9\lambda \ge 0.9 exceeds power 0.105 vs. λ=1\lambda=1 (Fisher-zz: 0.088), and 80% power needs k≈300k \approx 300 tasks; 3-seed non-certifiability is information-theoretic, not Fisher-zz-specific.
  • Rule 1 validation. Across 252 family splits (fit cc on half A with final-20% window + detrending; test on half B), the full pipeline passes 92.9% (234/252) on 4M–90M logits_per_char_corr (median ratio 1.000 [0.928, 1.078]; 2× share 52.4% vs. 56.2% perfect proxy). Dropping detrending (0% pass) or cc (2.83× bias) fails, and applying ≤90M cc to ≥150M passes 0/252 (median 0.632), confirming cc must be calibrated at target scale.

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.

Reviewer_ro19OpenReview ↗

Response to authors

Thank you for the detailed response and additional analyses. The joint resampling addresses my first question by showing which findings survive uncertainty across seeds, recipe families, and tasks. In particular, the positive association between SNR and pairwise ranking accuracy remains robust, while the joint interval for the 4M inversion crosses zero and supports a more qualified interpretation.

I have two remaining questions:

  1. Selection regret across training stacks. The held-out DataDecide splits provide evidence that SNR is associated with lower selection regret, and Datablations provides cross-stack evidence for pairwise ranking accuracy. Could you also report whether SNR predicts top-1 selection regret on Datablations? This would help establish whether the diagnostic transfers to the practical decision of choosing a single recipe.
  2. Scaling, adding seeds, and abstaining. The matched-FLOP comparisons and the distinction between certified claims and forced selection are helpful. The random fallback gives one concrete cost for abstention. When the uncertainty interval for the compute-matched benefit of scaling includes zero, how sensitive is the preferred action to that cost? A break-even analysis or a comparison across plausible abstention costs would make the recommendation easier to apply.
Reviewer_qsAwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits three tools used in small-scale pretraining: checkpoint-based noise proxies, initialization-only noise baselines, and data recipe ranking with small models. The authors reanalyze DataDecide, Signal & Noise, and PolyPythias, and conduct additional crossed experiments at the 20M scale. The results show that the main findings under the original experimental settings are reproducible. However, the magnitude calibration of noise proxies depends on the evaluation metric and training setup, single-source experiments cannot identify the total variance across all sources of randomness, and downstream task rankings obtained from extremely small models may be unreliable. The paper also provides tables of noise statistics and reports labeling errors in publicly released evaluation files.

Critical strengths
  • The evidence for magnitude miscalibration of noise proxies is convincing.

  • The audit is fair to both the original work and its own findings.

Critical weaknesses
  • The decision-making recommendations in Section 7 have not been directly validated. Being unable to distinguish a pair of recipes with statistical significance does not necessarily imply an inability to select a useful recipe. In Table 13, the top-1 loss at 60M is 1.9 points, actually lower than the 4.7 points observed at 150M. The interval for the "scale up first" comparison also includes zero. Furthermore, Table 16 shows that the fraction of decidable comparisons at 60M is notably sensitive to the variance model. Although the authors acknowledge these limitations, the current results are still insufficient to demonstrate that abstaining or scaling up improves actual selection outcomes.

  • The added value of the SNR analysis relative to prior work needs to be clarified. Section 3.4 of the original DataDecide paper already uses cross-seed standard deviation and the dispersion of recipe means to explain decision accuracy. Appendix D.5 of the present paper also shows that the improvement in predictive performance comes primarily from using empirically measured seed noise, while deconvolution does not further improve ranking predictions. Therefore, the paper's main contribution appears to lie in analyzing calibration and the scope of applicability. It remains unclear what additional insights or decisions the newly introduced metrics enable.

Questions for authors
  • Could the authors use the existing data to compare the expected top-1 loss of three strategies under the same total FLOPs budget: abstaining or scaling up according to the proposed rules, directly ranking recipes, and increasing the number of seeds?

  • Compared with the existing seed-noise/spread analysis in DataDecide §3.4, which specific conclusions or decisions are changed by the deconvolution and uncertainty estimation proposed in this paper? Please provide a direct comparison that distinguishes improvements in magnitude calibration from improvements in ranking prediction.

2 comments
AuthorsOpenReview ↗

Response to Reviewer qsAw

We appreciate the reviewer's thoughtful assessment and constructive questions, and thank the reviewer for noting that our proxy-calibration evidence is convincing and that the audit is fair to prior work. Below we address both questions directly.

1. Compute-matched top-1 selection loss across strategies (W1 & Q1)

Sections 5 and 7 (Rule 3) distinguish two goals: certifying pairwise winners (claiming Recipe A beats Recipe B in an ablation) and making a forced single pick (choosing one recipe to train at 1B). Up to 60M at 3 seeds, essentially zero pairs pass pair-specific multiplicity control, so Rule 3 recommends abstaining from certified pairwise claims. When compute is capped and one recipe must still be picked without escalating scale, Rule 3 already notes that raw rankings above the 4M inversion remain informative (as reflected in our Table 12/13 comparison of 60M's 1.86-point top-1 loss vs. 4.66 points at 150M).

To answer Q1 directly, we compared the three strategies on DataDecide's macro readout at matched total FLOPs across all 25 recipes (using DataDecide's exact per-run compute column), with B=2,000B = 2{,}000 joint resamples over recipe families, proxy seeds, and 1B target seeds:

Budget tierStrategyTotal FLOPs1B top-1 lossPairwise acc.Matched-budget outcome
Small (4M)Direct + add seeds (4M×3)6.3×10176.3 \times 10^{17}5.95 pts31.0%Baseline (inverted ranking)
Small (4M)Abstain at 4M (random fallback)6.3×10176.3 \times 10^{17}2.71 pts50.0%Saves +3.24 pts over 4M×3
Small (4M)Scale up (6M×1, 0.86×0.86\times cost)5.4×10175.4 \times 10^{17}1.97 pts45.8%Saves +3.98 pts (P(Δ>0)=0.85P(\Delta>0)=0.85) and +14.8 pp acc.
BoundaryAdd seeds at 60M (60M×3)1.47×10201.47 \times 10^{20}1.86 pts76.3%Baseline (0% BY decidable pairs)
BoundaryScale up (90M×1, 0.98×0.98\times cost)1.44×10201.44 \times 10^{20}1.86 pts78.2%Same pick (0.00 gap; 65% ties), +1.9 pp acc.
Full ladderFixed scale (150M×2, 1.08×1.08\times cost)6.7×10206.7 \times 10^{20}4.66 pts82.2%Matched-cost fixed baseline
Full ladderAbstain-and-escalate (ADAPT from 4M)6.2×10206.2 \times 10^{20}1.86 pts80.0%Skips ≤60M\le 60\text{M}, stops at 90M; beats 150M×2 by +2.80 pts (ties/beats matched fixed in 87%–89% of SNR resamples)

Takeaway. At matched FLOPs, our rules avoid the 4M inversion (+3.24 to +3.98 points saved), tie 60M×3 on top-1 loss at 90M×1 while improving global ranking (+1.9 pp), and beat matched fixed-scale selection (150M×2) by +2.80 points when escalating adaptively to 90M. Only when compute is hard-capped at 10M–60M and escalation is impossible should one take the uncertified raw argmax (1.09–1.86 loss vs. 2.71 fallback), matching Rule 3.

2. Direct comparison with DataDecide §3.4 (W2 & Q2)

DataDecide §3.4 (Figure 5) plots recipe spread against recipe-averaged 3-seed SD at 150M only; it does not form a cross-scale ratio, track 4M–1B scaling, or test recipe pairs. Comparing the un-deconvolved ratio implied by §3.4 (RDD=Sobs/σ^‾R_{\text{DD}} = S_{\text{obs}} / \overline{\hat\sigma}) against our deconvolved SNR and pair-specific Welch/BY layer across all 120 task$\times$scale cells yields:

DimensionDataDecide §3.4 (RDDR_{\text{DD}})Ours (deconvolved SNR + Welch/BY)Practical difference
Ranking prediction (n=3n{=}3)ρ=+0.800\rho = +0.800ρ=+0.813\rho = +0.813Unchanged at fixed nn (gain over S&N's +0.65+0.65 comes from measured seed noise; App. D.5).
Seed-budget shift (3→23 \to 2 seeds)+0.35 (+18% inflation)+0.03 (10×10\times smaller)Thresholds transfer across seed budgets instead of inflating at smaller nn.
Zero-signal false-alarm rate8.0% (n=3n{=}3), 46.9% (n=2n{=}2)2.3% (n=3n{=}3), 4.1% (n=2n{=}2); BY 0.6% / 0.3%Prevents false-signal claims: at 2 seeds, RDD>1R_{\text{DD}} > 1 mislabels nearly half of pure-noise panels.
Pairwise & seed-budget decisions150M only; pooled SD gives 34% at 60M (BH)0% resolved at ≤60M\le 60\text{M}; n=5,6,11,39n{=}5, 6, 11, 39 seeds for 50% BY at 150M/90M/60M/4MEliminates 34% inflated discoveries at 60M from variance pooling (§6, Table 16) and prices seed budgets.

Takeaway. Compared with DataDecide §3.4, our deconvolution and pair-specific Welch/BY layer contribute: (1) cross-budget calibration, reducing the 3-to-2 seed shift by 10x (+0.03 vs. +0.35); (2) zero-signal false-alarm control, cutting the 2-seed false-positive rate on pure-noise panels from 46.9% to 4.1% (0.3% under BY); and (3) heteroscedastic pairwise resolution, preventing the 34% inflated pairwise discoveries at 60M caused by pooling recipe variances (Table 16) and pricing exact seed budgets across scales.

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revision

Summary of Manuscript Revision

We have uploaded a revised PDF focusing on presentation clarity and bibliography formatting. No experiments, tables, figures, or empirical conclusions have been changed (additional analyses requested by the reviewers are reported directly in our individual responses below). Specifically:

  1. Main-text readability. As all three reviewers noted, the original main text was overly compressed and dense with inline caveats. We rewrote the prose across the abstract and main sections to state each audit's setup, estimand, and conclusion directly, expanded Section 8 (Related Work) for narrative clarity, and separated Limitations (Section 9) and Conclusion (Section 10). The main text fits within 9 pages.
  2. Bibliography. For cited works that exist as both an arXiv preprint and an archival venue publication, we standardized the bibliography entries and aligned their venue and publication metadata.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.