- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors measure the expected final-performance gain of best-of-N morphology screening on fixed pools, using independent screening and final controller seeds. It studies seven trained pools across four locomotion design spaces and supplements them with an EvoGym archive. Short screening usually helps, but a forward-travel Walker2d variant exhibits replicated negative selection lift. The contribution is an empirical dataset and characterization of screening failure.
Critical strengths
-
The paper provides a credible negative-result analysis, which includes independent final seeds, new bodies, a second seed set, longer training, and a held-out termination-rule test make the Walker reversal substantially more convincing than a selected learning-curve anecdote. The authors correctly treat overlapping seed splits as dependent and report simultaneous bands.
-
Useful empirical extension of an existing problem. Arza et al. already study co-design training-budget allocation, and Mertan and Cheney show undervaluation of morphological potential. This paper's additional value is the measured selection lift, independent retraining protocol, and systematic budget/candidate sweeps, not discovery that early and late fitness can disagree.
Critical weaknesses
-
I think the released per-body learning curves and negative results could materially improve evaluation of low-fidelity morphology selection. The contribution is narrower than general brain-body co-design scaling. Moreover, generalization to other learners and deployable morphology quality remains unestablished.
-
The observed failure remains specific to a weak, fixed learning/evaluation regime (§3.2, Tables 2, 10–11).** All new pools use one PPO configuration with unnormalized observations; Walker evaluation is a single deterministic episode from an unperturbed start, final scores are still improving, and 98% of final episodes end in a fall. Repeating seeds and termination rules confirms this pipeline's failure, but does not show that the screened bodies have intrinsically inferior achievable control performance. The mass/torque analysis is observational, with action and exploration scales tied to actuator gear. The supported conclusion is “short training misranks designs under this learner and objective,” not a causal account of body potential or a general half-budget safety rule. This limits significance rather than invalidating the finite-budget result.
-
Deployment-facing compute claims are less complete than the selection analysis . The full-budget comparator retrains the winner even after screening at H; continuation reuses the selected training runs and therefore changes both cost and the target from reproducible body quality to a selected body-controller pair. The paper acknowledges this distinction, but the practical optimal allocation is not established by the retraining-only comparison or the post hoc envelope. Environment steps also omit morphology-dependent runtime. The retrained pools reach N=128; the million-body archive uses a different controller optimizer and continuation, so it cannot establish large-scale retraining or iterative-search behavior. Tighten the practical recommendation to these precisely defined protocols.
Questions for authors
-
Is the intended target reproducible performance after fresh training, or best retained body-controller performance? Explain why the full-budget comparison retrains rather than retains already-trained controllers.
-
Do any existing records include perturbed starts, observation normalization, or another PPO configuration? If not, explicitly delimit the Walker conclusion to the stated learner and fixed-start objective; no new large experiment is requested for rebuttal.
-
Supply accessible timestamped preregistrations/amendments and the bootstrap replicate settings. Which practical budget recommendations were fixed before results, rather than selected from the same grid?