- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Paper looks at robot training under progressively harder conditions (changes in env like motor strength and loading, but also external factors like disturbances). Similar to curriculum learning this happens slowly, but the paper adds an automated framework to check whether these changes are causing issues and fall-backj to previous checkpoints if so.
The contribution feels a little limited, but it primarily shows that the checkpoint restoring is a valuable addition. It is tested in both simulation and on a real robot, and finds that specifically selecting 4 types of vaiation can be more effective than bulk training over all types.
Critical strengths
The ablation is affective at testing whether the different components help. It makes sure that the different mechanisms (checking under new conditions vs checking retention of abilities) are complementary. In App D2 They also ablate their approach against retracting the training ranges, which is intresting. Some interesting qualitative results are also shown including a gradual failure case and one where the technique doesn't help, which is useful.
The approach seems practically useful, results look good and are repeated across 5 seeds, plus the inclusion of real-robot trials raises confidence in the technique.
Critical weaknesses
If I'm not mositaken, the main comparison protocol for HORIZON uses the last successful model, but for others approaches it uses the final without restoration, even if that model was worse. This hides how much of the improvement comes from the restoration process during training as opposed to selecting a better final checpoint.
The experiment regarding 4 selected variations versusis all 7 is a little surprising in it's conclusion. My concern is that these models receive different amounts of training, and in the broadest experiment use only one model of each. I worry the conclusion here is not very clean.
Questions for authors
What happens if you don't rollback the checkpoint and just report the most recently accepted checkpoint rather than the final? Does the conclusion still persist?
If you make Core and Full use the same amount of training, is their ranking still the same on the joint combination testing? Did ytou choose the combinations and severity levels before you chose the Core or after?
Details on the dificulty increases are a bit thin in terms of reproducibility, can you detail exactly how each parameter was expanded (i.e. the min-max range) how far they expanded before stopping, and what you did when you contracted them. Similarly, exactly how and when the conditions are randomised isn't clear, and same for the simulation steps/training time, checks for retained performance (which I couldn't quite follow) and mechanism for model restoration.
4 comments
Response to Reviewer (2/2): Settings and cost
Q3: Settings and cost. Thank you for suggesting that we specify the configuration and training cost. The settings are as follows.
(1) Expansion steps for each parameter. Table 10 gives each parameter's Base range and search limits. Upper and lower boundaries are maintained separately. The coarse proposal follows the paper's equation:
For continuous parameters, there is no additional fixed physical minimum step or independent maximum-step cap. Fresh proposal displacement is , with clipping to the search limits. Core uses an initial coarse factor of 1.5 and a lower bound of 1.1. Failure updates the coarse factor to ; outside refinement, success resets the challenged side to 1.5. The refinement ladder is 1.3, 1.4, 1.6, 1.8, 2.0, maintained separately for each side. The first failure marks that side pending; the next success activates the first rung. Subsequent successes advance one rung and failures descend one rung, bounded by the ladder endpoints.
(2) Reduction after failure and stopping conditions. After failure, we retain the accepted–failed bracket. When retrying that side, the smaller proposal follows the paper's equation:
For two-sided parameters, two consecutive failures on the same side trigger a switch to the other unfinished side instead of an immediate interpolated retry; the failed-side information is retained. A parameter side stops upon reaching its search limit or meeting the configured numerical-resolution/failure stopping condition. A group leaves the queue once all its relevant parameter sides have stopped. Global search ends when no expandable group remains, or when the budget is reached and the current phase is completed, returning the latest accepted policy.
For Core, a side stops after four failures that start in the pending state or at the lowest refinement rung; success resets the count. The failure that first creates pending is excluded. At these states, a next retry displacement of at most also stops the side. Epsilon provides numerical separation for interpolated retries; it is not a fixed physical proposal step. The search-limit tolerance and scalar failed-bracket width threshold are both . The lower actuation side additionally uses a near-limit stopping tolerance of . Each phase allows at most 1,500 updates, with a 500-update checking window.
(3) Randomization frequency.
| Condition | Randomization timing |
|---|---|
| Actuation/contact | 32/64 buckets, respectively; resampled at startup and on curriculum transitions, rollback, and retention-comparison resets |
| Mass, inertia, COM, external force/torque, initial joint state | Every episode reset |
| Push velocity | Every 10–15 s |
| Velocity command | Every 10 s |
| Procedural body morphology | Sampled at scene construction; fixed across resets |
(4) Total interactions and training time. For one Core trajectory accumulated to approximately 45k updates, costs are summed across continuation segments:
| Item | Value |
|---|---|
| Parallel environments / collected steps per environment per PPO update | 4,096 / 24 |
| PPO training interactions | |
| Retention checks / learner restorations | 86 paired comparisons / 14 restorations |
| Total training and in-run checking interactions | |
| Hardware / cumulative runtime | One RTX 5090 / approximately 38.1 h |
Interactions count environment control steps; checking interactions are accumulated using at most 1,000 steps per rollout. Runtime includes initialization, retention checks, model restoration and I/O, and in-run diagnostics. It excludes downtime between continuation segments and final post-training evaluation.
Thank you for helping us distinguish final checkpoint selection, restoration during training, and configuration comparisons under different training allocations.
Thanks a lot for the detailed response. My queries have been answered appropriately and I am satisfied to retain my recommendation of acceptance.
Thank you for carefully considering our response and for your positive assessment of the revised work. We appreciate your constructive questions and will use your feedback to guide and improve our future research.
Response to Reviewer (1/2): Return rule and Core/Full
Thank you for your clear summary. We address W1 with Q1 and W2 with Q2; the next comment answers Q3.
Q1 / W1: Return rule versus training-time restoration. Thank you for highlighting this distinction. The original Table 2 compared complete procedures at Core's budget, returning terminal for w/o rollback. Its terminal and latest accepted checkpoints were close late in training, with little subsequent expansion, so accepted was not initially reported separately.
With latest accepted, w/o rollback obtains 27.8 ± 1.5% OOD-all, 0.190 ± 0.004 m/s Base error, and 60.9 ± 0.7% CS-AUC (General, G1). HORIZON retains a 9.0 pp OOD-all advantage, approximately 86% of the original 10.5 pp gap, plus 1.6 pp higher CS-AUC and 0.084 m/s lower Base error. The ablation improves over terminal by 1.5 pp, 0.025 m/s, and 0.5 pp, respectively. Its checkpoint is selected from its own gate logs before OOD evaluation.
Appendix D.2 separately matches parent failure, retracted ranges, continuation budget, and endpoint. Restore improves 9/11 events (+2.47 ± 2.73 pp), retaining negative cases P5/P10 (G2). Its CS-AUC sweeps only the failed group, others at Base. Thus G1 quantifies final selection, while D.2/G2 tests learner restoration during training.
Q2 / W2: Core/Full allocation, joint combinations, and selection. Thank you for your detailed questions.