- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper audits how measurement, ablation, and aggregation choices affect conclusions about sensorimotor pretraining. Using the authors’ Active Sensorimotor Curriculum implementation across four MuJoCo embodiments, it examines whether large motion-proxy improvements reflect downstream competence, whether setting an intrinsic-reward coefficient to zero adequately removes the corresponding component, and whether pooled diagnostic correlations predict performance on an unseen body. The validation includes checkpoint re-evaluation, downstream transfer, component-gating experiments, a corrected training pipeline, leave-one-body-out analysis, and external experiments involving shared-encoder ICM/RND variants and benchmark data. The paper also presents a restricted theoretical non-identifiability result and an audit checklist. It reports important qualifications, including the absence of a curriculum advantage in the examined final-performance comparisons and the disappearance of the pooling-induced winner disagreement when its own URLB experiments are extended from three to five seeds.
Critical strengths
- The investigation of the motion-proxy artifact provides useful intervention-based evidence. The strongest contribution is the examination of how evaluation conditions affect the apparent curriculum benefit. Re-evaluating fixed checkpoints, followed by comparisons using common states, matched gains, and common action noise, goes beyond observing a disagreement between proxy and downstream performance. These experiments directly investigate a mechanism through which the measurement procedure can produce a large apparent advantage. Together with the downstream evaluations and transfer-based positive controls, this provides persuasive local evidence that the original motion contrast should not be interpreted as acquired motor competence. The finding is consequential because the original contrast survived multiple embodiments and conventional statistical checks, illustrating a limitation that additional significance testing alone would not resolve. The small, post-hoc nature of the gain diagnostic appropriately limits its broader interpretation. See Section 3(d) and Appendix K, particularly pp. 29–30.
- The paper makes the limits of its conclusions unusually visible. The study combines several forms of evidence and explicitly reports results that weaken its initial interpretation. Examples include the matched-evaluation qualification of C2, the distinction between endpoint performance and adaptation efficiency, and the withdrawal of the winner-disagreement claim for the five-seed URLB extension. The treatment of tie handling and checkpoint dependence also helps distinguish conclusions that survive alternative analysis choices from those that do not. These analyses make the paper informative as an empirical audit: readers can learn both from the diagnosed failures and from the boundaries of the proposed lessons. The external experiments provide useful evidence beyond the original ASC implementation, while their stated restrictions prevent them from being read as universal results.
Critical weaknesses
- The attribution of the difference between ablation gaps remains less secure than the within-grid component-ablation result. Appendix K.1, item 12, states that the reward-weight and component-gating contrasts come from different training grids and trainer versions. The matched re-evaluation controls the evaluation regime, which is helpful, but the manuscript does not establish whether the training versions differ only in the intended gating behavior. Similar Full-arm means do not, by themselves, exclude a trainer-version interaction with the ablation. Consequently, the difference between the two ablation gaps cannot yet be attributed as confidently to the ablation method alone.
This concern does not invalidate the observation that zeroing a reward coefficient leaves auxiliary training paths active, nor does it require separating every encoder and actor mediation pathway. It concerns the empirical comparison supporting C2. Moreover, the matched analysis detects a reward-weight ablation effect on H1dm, while the raw-unit difference between ablation gaps is borderline on Go2. The main-text description of “opposite conclusions” should therefore consistently reflect the narrower evidence summarized in the appendix. Since C2 is one of the paper’s central methodological claims, this distinction materially affects my assessment. 2. The central audit protocol and its supporting results are not yet presented with sufficiently clear operational correspondence. The experiment-code map and extensive caveats are helpful, but reconstructing a particular conclusion still requires combining descriptions of training versions, evaluation conditions, analysis populations, and aggregation rules from several locations. For example, the precise Tier-1 composite and the correspondence between the H3 diagnostics and the separately described probe/residual procedures need a more explicit specification or reference to the relevant supplementary configuration. This is a request to connect existing definitions and implementations, rather than a claim that D3/D4 have not been described.
There are also concrete presentation inconsistencies: the Panda confidence intervals in Figures 3 and 4 do not match Table 1, and Figure 5 displays an ASC-versus-random probability near 0.31 while its caption describes the within-body value of 0.00. Appendix K explains that these latter values represent different aggregation procedures. These discrepancies do not establish that the underlying calculations are wrong, but they make it harder to identify the final result and estimand supporting each conclusion.
Because the paper positions the audit procedure as its principal contribution, the ability to reconstruct and apply that procedure is central to its significance. The convincing local C1 result is a substantial positive, but the unresolved C2 attribution and the operational correspondence of the broader protocol currently keep my assessment just below acceptance.
Questions for authors
Clarification of the C2 training comparison: Do the P2 and P10a trainer versions differ only in the intended component-gating behavior? Please identify the relevant code differences and explain whether any other change could affect the ablated arms. If the versions are otherwise equivalent, documenting that equivalence would substantially reduce my concern. If they are not, a comparison trained under a common implementation would be needed to isolate the difference between ablation methods.
Clarification of the operational protocol: Can you point to the exact definition of the Tier-1 composite, including task weighting, score direction, and scale handling, and specify how D3 and D4 correspond to the probe and prediction-error procedures described elsewhere? A compact mapping from each central claim to its trainer version, intervention, evaluation condition, analysis population, and final estimator would help establish what another researcher should reproduce. Existing supplementary definitions would address this question; I am not requesting a new diagnostic family.
Clarification of the final reported results: Which analysis populations and versions produced the Panda intervals in Figures 3 and 4, and should Figure 5 display the pooled or within-body probability of improvement? Please reconcile these figures with Table 1 and Appendix K and clarify whether updating them changes any stated inference or only the presentation.