acceptodds
Under review as a conference paper at ICLR 2027

What You Measure, Ablate, and Pool Decides the Conclusion: Auditing Sensorimotor Pretraining

Abstract

Statistical rigour does not protect against instrument artefacts. We report a preregistered multi-body audit of a developmental sensorimotor curriculum — four MuJoCo MJX embodiments (DM humanoid, Unitree G1, Go2 quadruped, Franka Panda), a 318-run grid plus follow-ups, Holm families, bootstrap CIs, TOST — in which three routine choices each independently changed what we concluded: which *metric* you read, how you *ablate* a component, and whether you *pool* across bodies. The first indicts our own instrument. The curriculum's headline training-time advantage (Hedges up to , raw ratios up to ) survived preregistration, four embodiments and Holm correction, yet a gain-matched diagnostic shows it is largely manufactured by the curriculum's own action-gain schedule. A fixed evaluation gain amplifies the curriculum arm's measured motion by – and the control arm's not at all; removing that asymmetry removes – of the raw ratio (– of the log-ratio). Measured at each policy's own training gain on a corrected trainer (six instrument fixes), the curriculum improves the proxy on **0/4** bodies. Nor does it transfer, at final performance: on **none of four** downstream routes does the curriculum improve on its own ablation, and on the frozen routes the audited intrinsic-motivation objectives (curriculum, empowerment, prediction-error; ICM/RND) sit at or near *random init*, while a self-supervised dynamics control, itself reward-free, transfers near-ceiling and none overtakes random init at a pretraining budget (P27; ICM health caveat in Appendix K). Two further lessons reach beyond this system, on less direct evidence. First, ablating prediction error by *reward weight* understates it: removing the *component* shows it suppresses motion (); under one matched evaluation the gating gap on H1_dm is larger by units, and on Go2 the raw difference is borderline; the representation effect reproduces on shared-encoder ICM and, dose-dependently, RND. Second, a cross-body diagnostic that looks predictive pooled () is a leave-one-body-out Simpson's paradox (every diagnostic's mean , best ). That pooling failure reproduces on *released benchmark data*: naive pooling erases of a real within-task Atari advantage, and under URLB's own seed noise no "best" unsupervised-RL method can be crowned. Running that benchmark's standard pipeline ourselves ( finetunings), no method tops all three of its domains at either seed budget we ran; the sharper claim — that two poolings of the same runs disagree about the winner — held at three seeds and *failed* at five, and we print that failure rather than the budget that suited us. We derive both failures as properties of the estimators, and release the protocol, per-seed data, and a checklist whose first row is the one we needed: check that a headline effect is not produced by the instrument that measures it.

Primary area:
datasets and benchmarks
Keywords:
measurement validity, evaluation methodology, sensorimotor pretraining, intrinsic motivation, curriculum learning, unsupervised reinforcement learning, Simpson's paradox, preregistration, negative results

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6cGuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper audits how measurement, ablation, and aggregation choices affect conclusions about sensorimotor pretraining. Using the authors’ Active Sensorimotor Curriculum implementation across four MuJoCo embodiments, it examines whether large motion-proxy improvements reflect downstream competence, whether setting an intrinsic-reward coefficient to zero adequately removes the corresponding component, and whether pooled diagnostic correlations predict performance on an unseen body. The validation includes checkpoint re-evaluation, downstream transfer, component-gating experiments, a corrected training pipeline, leave-one-body-out analysis, and external experiments involving shared-encoder ICM/RND variants and benchmark data. The paper also presents a restricted theoretical non-identifiability result and an audit checklist. It reports important qualifications, including the absence of a curriculum advantage in the examined final-performance comparisons and the disappearance of the pooling-induced winner disagreement when its own URLB experiments are extended from three to five seeds.

Critical strengths
  1. The investigation of the motion-proxy artifact provides useful intervention-based evidence. The strongest contribution is the examination of how evaluation conditions affect the apparent curriculum benefit. Re-evaluating fixed checkpoints, followed by comparisons using common states, matched gains, and common action noise, goes beyond observing a disagreement between proxy and downstream performance. These experiments directly investigate a mechanism through which the measurement procedure can produce a large apparent advantage. Together with the downstream evaluations and transfer-based positive controls, this provides persuasive local evidence that the original motion contrast should not be interpreted as acquired motor competence. The finding is consequential because the original contrast survived multiple embodiments and conventional statistical checks, illustrating a limitation that additional significance testing alone would not resolve. The small, post-hoc nature of the gain diagnostic appropriately limits its broader interpretation. See Section 3(d) and Appendix K, particularly pp. 29–30.
  2. The paper makes the limits of its conclusions unusually visible. The study combines several forms of evidence and explicitly reports results that weaken its initial interpretation. Examples include the matched-evaluation qualification of C2, the distinction between endpoint performance and adaptation efficiency, and the withdrawal of the winner-disagreement claim for the five-seed URLB extension. The treatment of tie handling and checkpoint dependence also helps distinguish conclusions that survive alternative analysis choices from those that do not. These analyses make the paper informative as an empirical audit: readers can learn both from the diagnosed failures and from the boundaries of the proposed lessons. The external experiments provide useful evidence beyond the original ASC implementation, while their stated restrictions prevent them from being read as universal results.
Critical weaknesses
  1. The attribution of the difference between ablation gaps remains less secure than the within-grid component-ablation result. Appendix K.1, item 12, states that the reward-weight and component-gating contrasts come from different training grids and trainer versions. The matched re-evaluation controls the evaluation regime, which is helpful, but the manuscript does not establish whether the training versions differ only in the intended gating behavior. Similar Full-arm means do not, by themselves, exclude a trainer-version interaction with the ablation. Consequently, the difference between the two ablation gaps cannot yet be attributed as confidently to the ablation method alone.

This concern does not invalidate the observation that zeroing a reward coefficient leaves auxiliary training paths active, nor does it require separating every encoder and actor mediation pathway. It concerns the empirical comparison supporting C2. Moreover, the matched analysis detects a reward-weight ablation effect on H1dm, while the raw-unit difference between ablation gaps is borderline on Go2. The main-text description of “opposite conclusions” should therefore consistently reflect the narrower evidence summarized in the appendix. Since C2 is one of the paper’s central methodological claims, this distinction materially affects my assessment. 2. The central audit protocol and its supporting results are not yet presented with sufficiently clear operational correspondence. The experiment-code map and extensive caveats are helpful, but reconstructing a particular conclusion still requires combining descriptions of training versions, evaluation conditions, analysis populations, and aggregation rules from several locations. For example, the precise Tier-1 composite and the correspondence between the H3 diagnostics and the separately described probe/residual procedures need a more explicit specification or reference to the relevant supplementary configuration. This is a request to connect existing definitions and implementations, rather than a claim that D3/D4 have not been described.

There are also concrete presentation inconsistencies: the Panda confidence intervals in Figures 3 and 4 do not match Table 1, and Figure 5 displays an ASC-versus-random probability near 0.31 while its caption describes the within-body value of 0.00. Appendix K explains that these latter values represent different aggregation procedures. These discrepancies do not establish that the underlying calculations are wrong, but they make it harder to identify the final result and estimand supporting each conclusion.

Because the paper positions the audit procedure as its principal contribution, the ability to reconstruct and apply that procedure is central to its significance. The convincing local C1 result is a substantial positive, but the unresolved C2 attribution and the operational correspondence of the broader protocol currently keep my assessment just below acceptance.

Questions for authors

Clarification of the C2 training comparison: Do the P2 and P10a trainer versions differ only in the intended component-gating behavior? Please identify the relevant code differences and explain whether any other change could affect the ablated arms. If the versions are otherwise equivalent, documenting that equivalence would substantially reduce my concern. If they are not, a comparison trained under a common implementation would be needed to isolate the difference between ablation methods.

Clarification of the operational protocol: Can you point to the exact definition of the Tier-1 composite, including task weighting, score direction, and scale handling, and specify how D3 and D4 correspond to the probe and prediction-error procedures described elsewhere? A compact mapping from each central claim to its trainer version, intervention, evaluation condition, analysis population, and final estimator would help establish what another researcher should reproduce. Existing supplementary definitions would address this question; I am not requesting a new diagnostic family.

Clarification of the final reported results: Which analysis populations and versions produced the Panda intervals in Figures 3 and 4, and should Figure 5 display the pooled or within-body probability of improvement? Please reconcile these figures with Table 1 and Appendix K and clarify whether updating them changes any stated inference or only the presentation.

Reviewer_aiF9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits Active Sensorimotor Curriculum (ASC), a system that combines a control range curriculum with intrinsic rewards. Experiments on four simulated bodies examine how measurement, ablation, and aggregation choices affect conclusions about pretraining. Much of the reported motion advantage is traced to evaluation gain and initial states. Complete component removal produces larger ablation effects in some settings. The tested representation diagnostics fail to predict performance on a body excluded from training despite correlations in pooled data. The study includes downstream transfer tests, a dynamics prediction control, and external benchmark analyses. The authors use these results to propose an evaluation checklist.

Critical strengths
  1. The gain intervention convincingly explains much of the apparent motion advantage. Controls for initial states, action gain, and sampling noise demonstrate why statistical significance and replication alone cannot establish measurement validity.
  2. The diagnostic analysis combines pooled correlations, comparisons within bodies, and prediction on unseen bodies. Accounting for differences in average performance strengthens the finding that the four diagnostics have little value for predicting competence across bodies.
Critical weaknesses
  1. C2 establishes a difference between the tested ablations, but its practical significance remains unclear. Complete removal also eliminates an auxiliary actor gradient that opposes the prediction error reward (p. 37). The paper does not establish whether the motion effect persists without this path or whether the ablation choice changes downstream task performance.
  2. The broader contribution of the proposed protocol remains only partly established. The external shared encoder studies mainly assess probes, while both aggregation rules select RND in the five seed URLB replication. Given that the general evaluation principles are established, stronger external evidence of changes in conclusions about downstream competence or method selection would help substantiate the contribution beyond the ASC findings.
Questions for authors

1.Does the difference between reward ablation and complete component removal persist within a single ASC trainer when reward and auxiliary updates are gated separately and direct auxiliary gradients to the actor are blocked? 2.Does the ablation choice change conclusions about downstream task performance in an independently validated shared encoder implementation beyond ASC? 3.Is the dynamics control’s transfer advantage preserved when pretraining data and compute are matched, or should the claim about learning objectives be limited to the tested pipelines? 4.How sensitive are the reported bootstrap winner frequencies and estimator disagreement in the published URLB table analysis to plausible correlations among tasks sharing a pretraining checkpoint?

Reviewer_jip9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims at pointing out how three methodological choices can produce arbitrary conclusions, whose significance is still supported by statistical tests. These factors are 1) the used metrics 2) the strategy used to ablate components 3) the strategy used to pool metrics across bodies. The paper's main contribution is a robust evaluation pipeline that identifies confounders. As an example, the authors bring a sensorimotor curriculum evaluated on multiple embodiments. They show how their pipeline reveals that: 1) much of their curriculum’s apparent benefit comes from how action gain is handled during evaluation; 2) the curriculum’s apparent benefit does not translate into better downstream performance; 3) two apparently similar ablations answer different questions; 4) a pooled correlation fails to predict performance on an unseen embodiment.

Critical strengths
  • The authors identify and highlight methodological choices that can lead to arbitrary conclusions, guiding researchers towards more robust evaluation protocols, and providing a showcase of how such confounding factors can corrupt or even reverse interpretations.
Critical weaknesses
  • The proposed guidelines are motivated by failure modes in the authors’ ASC implementation. Although the paper includes external benchmark analyses, the passages presented do not clearly establish which published conclusions change when the proposed audit is applied. The authors should distinguish implementation-specific findings from general methodological recommendations, and demonstrate more directly what the audit adds to existing evaluation practices.

  • This use of LLM in the writing is counter-effective and annoying. Already in the abstract and throughout the whole paper, the reader goes through a roller coaster of short sentences that pretend to be clear and a list of numerical data that are not contestualized in related experiments. This makes me suspect that LLMs where not only used for writing and designing experiments (which is welcome and accepted), but that the authors didn't make the effort to make their work understandable by a human reader, so potentially didn't even deeply understand it themselves. Also on this line, the paper has one only figure.

Questions for authors

Which existing research conclusions are affected by these methodological choices, and what does the proposed protocol reveal beyond what standard controls and separate analyses by task or embodiment would reveal?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.