- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes CROSSSIM-CURRICULUM, a method for selecting environment edits according to their predicted benefit for physical robot navigation. For each candidate edit, it performs short edited-versus-control training probes in two simulators from a shared policy checkpoint. The simulator gains, their disagreement, source difficulty, and edit descriptors are used to predict physical learning gain, with a Bayesian residual model calibrated using sparse physical trials. The curriculum selector accounts for predicted gain, uncertainty, training cost, and coverage across edit families. The paper presents conditional calibration and local-selection bounds and reports comparisons on structural and compositional navigation splits, a held-out simulator, and physical routes. However, the experimental reporting and supporting material are incomplete.
Critical strengths
The central objective is meaningful: an environment that is difficult in simulation is not necessarily useful for improving physical performance. Comparing edited and control training runs from the same checkpoint provides a sensible way to estimate the incremental value of an environment edit.
Critical weaknesses
The submission is incomplete to an extent that prevents a reliable assessment of its results. It refers to Appendices A–D for proofs, implementation details, cost accounting, and diagnostic experiments, but none are included, and the linked repository is empty. Section 6 still describes a proposed test plan rather than clearly documenting the experiments underlying Table 1. Essential information, including the robot and controller, simulator configurations, actual seed and site counts, and numerical training budgets, is missing. This goes beyond presentation quality: the reported physical-transfer results cannot be adequately checked or reproduced.
Questions for authors
Which engines correspond to simulators A and B, and what configurations differ between them? Please specify how the robot, control interface, observations, and rewards are matched, together with the probe horizon K, student-training horizon L, and total simulation budget.