What You Measure, Ablate, and Pool Decides the Conclusion: Auditing Sensorimotor Pretraining

Statistical rigour does not protect against instrument artefacts. We report a preregistered multi-body audit of a developmental sensorimotor curriculum — four MuJoCo MJX embodiments (DM humanoid, Unitree G1, Go2 quadruped, Franka Panda), a 318-run grid plus follow-ups, Holm families, bootstrap CIs, TOST — in which three routine choices each independently changed what we concluded: which *metric* you read, how you *ablate* a component, and whether you *pool* across bodies. The first indicts our own instrument. The curriculum's headline training-time advantage (Hedges $g$ up to $4.4$, raw ratios up to $195\times$) survived preregistration, four embodiments and Holm correction, yet a gain-matched diagnostic shows it is largely manufactured by the curriculum's own action-gain schedule. A fixed evaluation gain amplifies the curriculum arm's measured motion by $15$–$26\times$ and the control arm's not at all; removing that asymmetry removes $74$–$93\%$ of the raw ratio ($29$–$63\%$ of the log-ratio). Measured at each policy's own training gain on a corrected trainer (six instrument fixes), the curriculum improves the proxy on **0/4** bodies. Nor does it transfer, at final performance: on **none of four** downstream routes does the curriculum improve on its own ablation, and on the frozen routes the audited intrinsic-motivation objectives (curriculum, empowerment, prediction-error; ICM/RND) sit at or near *random init*, while a self-supervised dynamics control, itself reward-free, transfers near-ceiling and none overtakes random init at a $10\times$ pretraining budget (P27; ICM health caveat in Appendix K). Two further lessons reach beyond this system, on less direct evidence. First, ablating prediction error by *reward weight* understates it: removing the *component* shows it suppresses motion ($g=-3.0/-1.2$); under one matched evaluation the gating gap on H1_dm is larger by $21[13.5,28.0]$ $|q_{\text{vel}}|$ units, and on Go2 the raw difference is borderline; the representation effect reproduces on shared-encoder ICM and, dose-dependently, RND. Second, a cross-body diagnostic that looks predictive pooled ($\rho \approx 0.5$) is a leave-one-body-out Simpson's paradox (every diagnostic's mean $R^2<0$, best $-0.19$). That pooling failure reproduces on *released benchmark data*: naive pooling erases $85\%$ of a real within-task Atari advantage, and under URLB's own seed noise no "best" unsupervised-RL method can be crowned. Running that benchmark's standard pipeline ourselves ($144$ finetunings), no method tops all three of its domains at either seed budget we ran; the sharper claim — that two poolings of the same runs disagree about the winner — held at three seeds and *failed* at five, and we print that failure rather than the budget that suited us. We derive both failures as properties of the estimators, and release the protocol, per-seed data, and a checklist whose first row is the one we needed: check that a headline effect is not produced by the instrument that measures it.