Works Where Calibrated, Fails Where Misused: Auditing Small-Scale Pretraining's Seed-Noise Instruments

Small-scale pretraining selects data recipes under a handful of seeds, read by three instruments. On public per-seed assets (plus two small self-run controls; Appendix H), we audit each where it is used: two of the three reproduce at home (the floor's per-seed data were never released), and each fails where misused — differently for each. The checkpoint-noise proxy of Signal & Noise reproduces at home, but its advertised correlation is beyond certification at realistic seed budgets under the conventional Fisher-$z$ certificate (a perfect proxy reads $0.66$–$0.82$ against $0.9$), and it under-reads seed noise in magnitude on the accuracy readout ($\sim 1.6\times$ median, up to $2.4\times$, reversed on perplexity). An init-only noise floor cannot identify the all-source floor from single-source arms at any budget; only a bundled or crossed init$\times$order design can. DataDecide's recipe ranking reproduces its $\sim 80\%$ exactly at its recommended scale, but on the accuracy readout its deconvolved separability SNR stays at or below $1.3$ up to 60M, and at 4M the macro ranking inverts against the 1B ranking ($r=-0.56$; robust to recipe resampling, borderline once seed noise is priced in) and seed averaging does not rescue it; the 11-domain perplexity readout does not invert. We release code, a per-cell noise atlas, and report $1,095$ label–config-mismatched files in PolyPythias's release.