acceptodds
Under review as a conference paper at ICLR 2027

HORIZON: Recoverable Physical-Domain Expansion for Policy Generalization

Abstract

In on-policy reinforcement learning, expanding physical randomization changes both the control problem and the trajectories that drive learning. Retracting a failed expansion cannot undo learner updates that erode acquired behavior. We introduce HORIZON, a recoverable curriculum that treats physical-domain expansion as a coupled update of the training distribution and learning state. The method accepts an expansion only when the updated policy satisfies candidate-range feasibility and retains performance under previously accepted conditions. After failure, it restores the accepted ranges and learning state while preserving the failed boundary for refinement. Failed exploration thus informs subsequent proposals without its learner updates persisting. In quadruped locomotion, HORIZON achieves 36.8% success under the strongest joint physical shift, compared with 19.8% for OpenAI ADR and 22.7% for GRAM. We further find that broader physical diversity need not improve joint generalization. Under the evaluated training allocations, a four-group curriculum achieves a higher mean success on the tested joint shifts than full seven-group expansion, while the latter remains stronger on isolated shifts outside the four groups. The resulting fixed-weight policy transfers to hardware under payload changes, joint constraints, and leg extensions without online policy optimization. These findings identify recoverable expansion and domain composition as complementary considerations for learning generalizable motor control.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Physical-Domain Scaling, Curriculum Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KmkSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper looks at robot training under progressively harder conditions (changes in env like motor strength and loading, but also external factors like disturbances). Similar to curriculum learning this happens slowly, but the paper adds an automated framework to check whether these changes are causing issues and fall-backj to previous checkpoints if so.

The contribution feels a little limited, but it primarily shows that the checkpoint restoring is a valuable addition. It is tested in both simulation and on a real robot, and finds that specifically selecting 4 types of vaiation can be more effective than bulk training over all types.

Critical strengths

The ablation is affective at testing whether the different components help. It makes sure that the different mechanisms (checking under new conditions vs checking retention of abilities) are complementary. In App D2 They also ablate their approach against retracting the training ranges, which is intresting. Some interesting qualitative results are also shown including a gradual failure case and one where the technique doesn't help, which is useful.

The approach seems practically useful, results look good and are repeated across 5 seeds, plus the inclusion of real-robot trials raises confidence in the technique.

Critical weaknesses

If I'm not mositaken, the main comparison protocol for HORIZON uses the last successful model, but for others approaches it uses the final without restoration, even if that model was worse. This hides how much of the improvement comes from the restoration process during training as opposed to selecting a better final checpoint.

The experiment regarding 4 selected variations versusis all 7 is a little surprising in it's conclusion. My concern is that these models receive different amounts of training, and in the broadest experiment use only one model of each. I worry the conclusion here is not very clean.

Questions for authors

What happens if you don't rollback the checkpoint and just report the most recently accepted checkpoint rather than the final? Does the conclusion still persist?

If you make Core and Full use the same amount of training, is their ranking still the same on the joint combination testing? Did ytou choose the combinations and severity levels before you chose the Core or after?

Details on the dificulty increases are a bit thin in terms of reproducibility, can you detail exactly how each parameter was expanded (i.e. the min-max range) how far they expanded before stopping, and what you did when you contracted them. Similarly, exactly how and when the conditions are randomised isn't clear, and same for the simulation steps/training time, checks for retained performance (which I couldn't quite follow) and mechanism for model restoration.

4 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2): Return rule and Core/Full

Thank you for your clear summary. We address W1 with Q1 and W2 with Q2; the next comment answers Q3.

Q1 / W1: Return rule versus training-time restoration. Thank you for highlighting this distinction. The original Table 2 compared complete procedures at Core's budget, returning terminal for w/o rollback. Its terminal and latest accepted checkpoints were close late in training, with little subsequent expansion, so accepted was not initially reported separately.

With latest accepted, w/o rollback obtains 27.8 ± 1.5% OOD-all, 0.190 ± 0.004 m/s Base error, and 60.9 ± 0.7% CS-AUC (General, G1). HORIZON retains a 9.0 pp OOD-all advantage, approximately 86% of the original 10.5 pp gap, plus 1.6 pp higher CS-AUC and 0.084 m/s lower Base error. The ablation improves over terminal by 1.5 pp, 0.025 m/s, and 0.5 pp, respectively. Its checkpoint is selected from its own gate logs before OOD evaluation.

Appendix D.2 separately matches parent failure, retracted ranges, continuation budget, and endpoint. Restore improves 9/11 events (+2.47 ± 2.73 pp), retaining negative cases P5/P10 (G2). Its CS-AUC sweeps only the failed group, others at Base. Thus G1 quantifies final selection, while D.2/G2 tests learner restoration during training.

Q2 / W2: Core/Full allocation, joint combinations, and selection. Thank you for your detailed questions.

  • Budget and late-stage behavior. Core/Full receive approximately 45k/60k updates; Full gets 15k more yet scores lower. Core's late expansion had largely converged. At a common nominal budget of approximately 45k updates, each of five independent training runs per configuration returns its accepted checkpoint closest to 45k. Full obtains 22.9 ± 1.8% OOD-all success and 58.1 ± 0.4% CS-AUC, with mean Base tracking error of 0.122 m/s. Core obtains 36.8 ± 2.5% and 62.5 ± 1.2%, respectively; all ± values denote sample SD across the five trained models. The ordering therefore persists at a common budget. Both finish the current phase and return latest accepted. Core training records through 49k show only 3/14 phases accepted after 30k: the 32k actuation expansion and increments of 0.002/0.0004, with none between approximately 34.5k and 49k. Small retries can fail candidate mastery or retention, explaining the allocation.
  • Independent policies and combinations. Table 2 compares five independent training seeds under common seven-group conditions: Core/Full achieve 36.8%/28.1% OOD-all and 62.5%/59.1% CS-AUC. Table 25 uses one fixed checkpoint per configuration on the same 91 combinations; Core leads on 57, with means 48.94%/45.57%. These assess training variation and performance across joint conditions for fixed policies, respectively.
  • Group-level explanation. Base-DR RCS-AUC is 81–88% on non-Core groups versus 52–64% on Core groups (Figure 4a). Core keeps non-Core groups at Base yet reaches 91.0–94.5% isolated success; Full reaches 97.8–99.0% there but is 2.1–5.9 pp lower on Core groups (Table 13). Adding each non-Core shift to Core shifts still leaves Core ahead by 10.0–12.6 pp (Table 24). The extra groups' isolated gains do not translate into better joint performance.
  • Selection timeline. All seven groups' Base ranges/search limits were set before Core selection, providing broad search bounded by simulator stability (Table 10). Core was fixed using the single-group, pairwise, and three-group development diagnostics before the final joint evaluations. The final joint-evaluation outcomes were not used to revise the selected composition. The common severity path and 91-combination retest were designed afterwards, using the preselection Base-to-limit ranges and all subsets of three to five groups. No ranges or combinations were adjusted for Core. Both face the same test conditions, independent of training frontiers. Common-path CS-AUC compares selected policies; development-stage, Base-DR-calibrated RCS-AUC analyzes group-specific learning needs, with separately defined paths.

AuthorsOpenReview ↗

Response to Reviewer (2/2): Settings and cost

Q3: Settings and cost. Thank you for suggesting that we specify the configuration and training cost. The settings are as follows.

(1) Expansion steps for each parameter. Table 10 gives each parameter's Base range and search limits. Upper and lower boundaries are maintained separately. The coarse proposal follows the paper's equation:

fcand=b−b−f∗ρgrow,ρgrow>1.f_{\mathrm{cand}}=b-\frac{b-f^*}{\rho_{\mathrm{grow}}},\qquad \rho_{\mathrm{grow}}>1.

For continuous parameters, there is no additional fixed physical minimum step or independent maximum-step cap. Fresh proposal displacement is (1−1/ρ)∣b−f∗∣(1-1/\rho)|b-f^*|, with clipping to the search limits. Core uses an initial coarse factor of 1.5 and a lower bound of 1.1. Failure updates the coarse factor to max⁡(1.1,1+0.7(ρ−1))\max(1.1,1+0.7(\rho-1)); outside refinement, success resets the challenged side to 1.5. The refinement ladder is 1.3, 1.4, 1.6, 1.8, 2.0, maintained separately for each side. The first failure marks that side pending; the next success activates the first rung. Subsequent successes advance one rung and failures descend one rung, bounded by the ladder endpoints.

(2) Reduction after failure and stopping conditions. After failure, we retain the accepted–failed bracket. When retrying that side, the smaller proposal follows the paper's equation:

fretry=f⋆+0.4(f−−f⋆)f_{\mathrm{retry}} = f^{\star} + 0.4(f^{-} - f^{\star})

For two-sided parameters, two consecutive failures on the same side trigger a switch to the other unfinished side instead of an immediate interpolated retry; the failed-side information is retained. A parameter side stops upon reaching its search limit or meeting the configured numerical-resolution/failure stopping condition. A group leaves the queue once all its relevant parameter sides have stopped. Global search ends when no expandable group remains, or when the budget is reached and the current phase is completed, returning the latest accepted policy.

For Core, a side stops after four failures that start in the pending state or at the lowest refinement rung; success resets the count. The failure that first creates pending is excluded. At these states, a next retry displacement of at most ϵ=10−6max⁡(1,∣f∗∣)\epsilon=10^{-6}\max(1,|f^*|) also stops the side. Epsilon provides numerical separation for interpolated retries; it is not a fixed physical proposal step. The search-limit tolerance and scalar failed-bracket width threshold are both 10−410^{-4}. The lower actuation side additionally uses a near-limit stopping tolerance of 0.07(c−L)0.07(c-L). Each phase allows at most 1,500 updates, with a 500-update checking window.

(3) Randomization frequency.

ConditionRandomization timing
Actuation/contact32/64 buckets, respectively; resampled at startup and on curriculum transitions, rollback, and retention-comparison resets
Mass, inertia, COM, external force/torque, initial joint stateEvery episode reset
Push velocityEvery 10–15 s
Velocity commandEvery 10 s
Procedural body morphologySampled at scene construction; fixed across resets

(4) Total interactions and training time. For one Core trajectory accumulated to approximately 45k updates, costs are summed across continuation segments:

ItemValue
Parallel environments / collected steps per environment per PPO update4,096 / 24
PPO training interactions4.424×1094.424\times10^9
Retention checks / learner restorations86 paired comparisons / 14 restorations
Total training and in-run checking interactions≤5.65×109\leq5.65\times10^9
Hardware / cumulative runtimeOne RTX 5090 / approximately 38.1 h

Interactions count environment control steps; checking interactions are accumulated using at most 1,000 steps per rollout. Runtime includes initialization, retention checks, model restoration and I/O, and in-run diagnostics. It excludes downtime between continuation segments and final post-training evaluation.

Thank you for helping us distinguish final checkpoint selection, restoration during training, and configuration comparisons under different training allocations.

Thanks a lot for the detailed response. My queries have been answered appropriately and I am satisfied to retain my recommendation of acceptance.

Thank you for carefully considering our response and for your positive assessment of the revised work. We appreciate your constructive questions and will use your feedback to guide and improve our future research.

Reviewer_F3bYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

HORIZON is a recoverability-aware curriculum that jointly governs physical-domain expansion and learner-state updates. HORIZON couples domain expansion with explicit checks of both candidate-domain feasibility and retention of performance on previously accepted domains. When an expansion fails, the method restores the previously accepted learning state, retracts the candidate physical ranges, and refines the failed boundary for subsequent attempts.

The authors evaluate HORIZON against fixed and scheduled randomization, OpenAI Automatic Domain Randomization (ADR), DORAEMON, Active Domain Randomization, and GRAM, alongside ablations of rollback, domain composition, and multi-expert distillation. HORIZON achieves 36.8% mean success on the joint OOD-all evaluation and 62.5% CS-AUC, outperforming DORAEMON (18.9% and 54.8%, respectively) under the reported unified comparison protocol. Controlled restoration experiments further show that restoring the accepted learning state generally improves subsequent learning compared with retracting the sampling ranges alone. Additional experiments examine physical-parameter composition, and further show that broader physical coverage need not improve joint generalization under the evaluated training allocations

Critical strengths

1. Strong empirical performance against relevant adaptive domain randomization baselines.

HORIZON substantially outperforms DORAEMON in joint OOD-all success (36.8% versus 18.9%) and CS-AUC (62.5% versus 54.8%). DORAEMON and ADR are appropriate baselines, as they represent established approaches to adaptive domain randomization based on task-success constraints and boundary-based range adjustment, respectively. The appendix further supports the validity of the comparison by evaluating these methods under a setup, with the same policy architecture, learner, randomized Core groups, and approximately matched PPO-update budgets. Overall, the results provide convincing empirical evidence of HORIZON’s effectiveness

2.Targeted ablations provide evidence for the value of learning-state restoration.

Appendix D.2 provides a controlled evaluation of learner-state restoration by comparing two branches that revert to the same accepted physical ranges but differ in whether the previously accepted learner state is restored. Under matched continuation budgets, restoration improves endpoint CS-AUC by 7.60 percentage points for an actuation-collapse event and 5.21 points for a disturbance failure without immediate collapse. Across six parent failure events, CS-AUC improves in five, with a mean paired gain of 3.55 percentage points. Although the benefit is not universal and the number of failure events is limited, this analysis offers targeted evidence for the value of learner-state restoration, strengthening the empirical justification for a central component of HORIZON.

Critical weaknesses

The algorithmic novelty relative to existing adaptive domain randomization and recovery mechanisms remains incremental.

HORIZON builds on established ideas, including performance-constrained distribution expansion, domain contraction after failure, checkpoint restoration, and retention monitoring. For example, DORAEMON already adapts dynamics distributions under task-success constraints and responds to constraint violations. HORIZON's more distinctive contribution is the integration of candidate feasibility, retention on accepted domains, learning-state restoration, and failed-boundary refinement into a single physical-domain expansion procedure. However, the paper does not fully establish that this combination constitutes a fundamentally new algorithmic principle rather than a carefully engineered integration of existing mechanisms. Although the controlled restoration ablations support the value of a key design choice, the paper could more clearly distinguish its algorithmic contribution from existing adaptive randomization and recovery methods. A more explicit mechanism-level comparison with prior curriculum and continual-learning methods would help clarify the extent of the novelty.

Questions for authors

1. Clarification — Algorithmic novelty.

Could the authors clarify which aspects of HORIZON are algorithmically novel beyond existing performance-constrained domain adaptation and recovery mechanisms? A clearer mechanism-level distinction would help assess the methodological novelty.

2. Clarification — Evidence for learner-state restoration.

Appendix D.2 provides controlled continuation experiments across six parent failure events, but the results are based on a limited number of parent trajectories, with multiple continuation seeds sharing the same parent checkpoint in P1–P2. How robust is the observed restoration benefit across independently trained parent policies and different failure types?

2 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2): Algorithmic contribution

Thank you for recognizing the targeted design of the matched restoration experiments and for asking us to clarify the algorithmic contribution and evidence across parent trajectories and failure types.

Your weakness W1 and Q1 concern the same issue, so we address them together below. Q2, in the next comment, addresses parent independence and failure-type coverage.

Q1 / W1: Algorithmic novelty and differences from existing mechanisms. Thank you for your insightful comments on the contribution and its relationship to existing mechanisms. We agree that distribution contraction, checkpoint restoration, and retention monitoring each have precedents. HORIZON's contribution is to define physical-domain expansion as a joint commit or restoration of the training distribution and learner state. The new ranges and learner are committed only when candidate feasibility LkL_k and retention on accepted ranges CkC_k both pass. Otherwise, both are restored, while the failed boundary remains in the search state to refine the next proposal.

The transaction view describes a concrete expansion-update rule. Changing physical ranges changes the trajectories subsequently collected by an on-policy learner. Contracting the distribution alone after failure therefore leads to a different learning process from also restoring the actor, critic, optimizer, learning rate, and enabled normalizers. Beyond a checkpoint restart, the rule specifies when to accept new ranges, when to check retained capability, and how failure informs the next expansion. Controlled experiments test the two gates and state restoration separately (§4.2, Table 1, Appendices D.1–D.2).

Specifically, HORIZON jointly tests feasibility on candidate ranges and relative performance on the latest accepted ranges. The restoration reference is the most recent checkpoint accepted by both gates within the same curriculum, and the failed boundary informs subsequent proposals. The distinctions are thus in the acceptance signals, restoration reference, and search-state update rules.

Appendix C.1, Figure 6 compares robotics and curriculum methods at the mechanism level, as reproduced below. A dash indicates no physical failed-bracket mechanism.

MethodUpdated objectSignal/referenceLearner mechanismFailed-boundary memory
OpenAI ADRPhysical rangesBoundary success rateContinue updating—
DORAEMONDistributionCurrent success rateContinue updating—
GRAMPolicy + adapterContext/robustnessContext conditioning—
SPDLTask distributionValue/KL to target distributionContinue updating—
CURROTTask distributionReturn constraintContinue updating—
HORIZONRanges + learnerLkL_k and CkC_kRestore accepted stateFailed bracket + retry

AuthorsOpenReview ↗

Response to Reviewer(2/2): Parent trajectories and failure types

Q2: Different parent trajectories and failure types. Thank you for asking about robustness across parent trajectories and failure types. Restoration gains are predominantly positive across curriculum trajectories and both failure types, with event-dependent magnitudes. Nine of eleven events improve CS-AUC (+2.47 ± 2.73 pp). P3–P11 come from nine separately reseeded HORIZON curriculum trajectories; within this subset, seven of nine improve, with a mean gain of +1.60 pp (General response, Table G2). By failed group, six of eight actuation events improve (+2.55 ± 2.97 pp), as do all three disturbance events (+2.27 ± 2.55 pp). By immediate collapse, five of seven collapse events improve (+2.49 ± 3.20 pp), as do all four non-collapse events (+2.45 ± 2.11 pp). All eleven first-failure events eligible under the paper's rule involve actuation or disturbance; none involves mass or COM.

We update Table G2 from six to eleven events by including P7, which meets the same selection rule in the P3–P6 batch, and new parents P8–P11, selected under the same rule registered before launch (Table G2). The five continuation-training seeds for each of P1 and P2 characterize variation conditional on that parent. P3–P11 instead share a 1,500-update Base warm start, then use different random seeds to produce subsequent curriculum trajectories, each contributing its first natural Core-group failure satisfying the paper's rule. The additions therefore test replication across curriculum trajectories rather than simply adding continuations of one parent.

Each Restore/Range-only pair starts from the same parent failure, retracts to the same accepted ranges, and uses the same continuation budget before evaluation at a fixed endpoint. CS-AUC sweeps only the failed group from Base to its search limits, keeping the other groups at Base. Means and sample SDs summarize event-level paired differences; Table G2 reports the full results.

The paired comparison under the same physical ranges and continuation budget shows that restoring learner state improves subsequent learning. Non-collapse events have no immediate collapse at the branch point, yet restoration still yields a positive mean gain at the continuation endpoint. This agrees with the mechanism: retracting physical ranges does not automatically undo learner updates made during the failed phase.

Both negative cases, P5 and P10, remain in the table and every summary, showing variation across events. Mean gains are positive across curriculum trajectories and for failures with and without immediate collapse. This also connects to Q1: physical ranges and learner state jointly determine the starting point for learning after recovery.

Thank you for helping us clarify the algorithmic mechanism, controlled evidence, and scope.


Reviewer_14BcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HORIZON, a recoverable curriculum learning framework designed to mitigate policy degradation caused by aggressive domain randomization (DR) in RL. To prevent irreversible collapse during domain expansion, HORIZON uses a checkpointing mechanism to audit target feasibility, safely roll back failed expansions, and refine parameter boundaries. HORIZON shows consistent generalization gains over strong DR baselines.

Critical strengths
  1. The paper highlights an interesting issue in adaptive domain randomization: restoring the training distribution does not necessarily recover a degraded policy. The matched restoration experiments provide evidence that restoring the learning state improves subsequent training.

  2. The domain composition experiments reveal that expanding more physical parameter groups does not necessarily improve joint generalization. The comparison between Core and Full provides useful insights into the interactions among physical variations.

Critical weaknesses
  1. The success threshold of 0.4m/s for planar velocity tracking, while excluding yaw error, may not adequately distinguish robust locomotion from conservative stabilization. It is unclear whether the reported improvements would hold under stricter evaluation criteria.

  2. The 4-group Core curriculum is manually selected based on preliminary experiments, while Core and Full use different training budgets. It remains unclear whether the observed advantage comes from the selected domain combination or the specific experimental setup.

  3. The rollback effect is not fully isolated. HORIZON returns the latest accepted checkpoint, whereas the no-rollback variant returns the terminal learner. The additional restoration study covers only six failure events, with one negative CS-AUC result.

  4. The generalization evidence is limited. Core is manually selected on the same task, Core and Full use different training budgets, and hardware evaluation covers only one robot while comparing against Base-DR rather than stronger adaptive baselines.

Questions for authors
  1. Can the authors compare HORIZON and no-rollback using the same returned-checkpoint rule, so that restoration during training is separated from final checkpoint selection?

  2. How sensitive are the results to δe=δR=0.05\delta_e=\delta_R=0.05, the LkL_k thresholds, the 500-update window, and the 1,500-update phase length?

  3. Is Core selected using a held-out protocol, and is there evidence that the same composition rule transfers to another task or robot without repeating the manual diagnostics?

4 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2) : Questions

Thank you for your detailed review. This comment addresses Q1–Q3; the next addresses W1–W4.

Q1: The same return rule. Thank you for highlighting this distinction. The original Table 2 compared complete procedures at Core's training budget, with w/o rollback returning terminal. Its terminal and latest accepted checkpoints were close in training time, with little later expansion, so accepted was not initially reported separately. With latest accepted for both, HORIZON retains advantages of 9.0 pp in OOD-all success, 1.6 pp in CS-AUC, and 0.084 m/s lower Base tracking error (General, G1). Switching the ablation to accepted improves these metrics by 1.5 pp, 0.5 pp, and 0.025 m/s. Selection explains part of the gap; the checkpoint was selected from its own gate logs before OOD evaluation. Appendix D.2 separately tests training-time restoration with the same parent failure, retracted ranges, continuation budget, and endpoint (W3).

Q2: Sensitivity. Thank you for this valuable question. Beyond the submitted α\alpha study (Appendix Table 19) and whole-gate ablations (Tables 1, 15–16), Appendix D.4 adds one-factor branches from the same accepted checkpoint at 49k, each trained for 4,000 updates (R2). Seven of nine variants return the Reference policy.

  • δe=δR\delta_e=\delta_R (2%/5%/10%). The 10% and 5% settings return the same policy; 2% changes Base success/CS-AUC by −0.1/−1.7 pp. Proposed groups differ, but no phase is rejected by CkC_k at any of these values.
  • LkL_k. Length thresholds 0.85 (default) and 0.90 return the same policy; relaxing to 0.80 reduces Base success/CS-AUC by 6.5/1.2 pp. Tracking threshold 0.55 also returns Reference.
  • Window/phase cap. Changing the default 500-update window to 250/1,000 or the 1,500-update cap to 1,000/2,000 leaves the returned policy unchanged.

Relaxing LkL_k length threshold admits Base-degrading expansions, illustrating its purpose.

Table R2. One-factor branches; differences from Reference (pp). Returned = latest accepted; † = starting checkpoint when no expansion is accepted. Budget-end = undecided learner, which HORIZON does not return. CS-AUC uses the joint seven-group path, 1,024 episodes per point.

SettingAccepted / phasesReturned: ΔBase / ΔCS-AUCBudget-end: ΔBase / ΔCS-AUC
Reference0 / 20 / 0 †0 / 0
δe=δR=2%\delta_e=\delta_R=2\%1 / 2−0.1 / −1.7−1.8 / +1.2
δe=δR=10%\delta_e=\delta_R=10\%0 / 20 / 0 †+1.5 / +1.6
LkL_k length threshold 0.802 / 3−6.5 / −1.2−2.1 / +0.7
LkL_k length threshold 0.900 / 20 / 0 †−2.7 / −0.2
LkL_k tracking-error threshold 0.550 / 20 / 0 †−2.7 / −0.5
Candidate window 2500 / 20 / 0 †−5.6 / −2.8
Candidate window 1,0000 / 20 / 0 †−6.2 / −2.4
Maximum phase length 1,0000 / 30 / 0 †−1.3 / 0.0
Maximum phase length 2,0000 / 10 / 0 †−0.8 / −0.7

Q3: Core selection and transfer. Thank you for asking about the selection process. All seven groups' Base ranges/search limits were set first (Table 10). Core was fixed using the single-group, pairwise, and three-group development diagnostics before the final joint evaluations. The final joint-evaluation outcomes were not used to revise the selected composition (§4.4, Appendix E). The common severity path and exhaustive 91-combination retest (all subsets of three to five groups) were designed afterwards using those pre-existing ranges. No ranges or combinations were adjusted for Core; both configurations face the same physical conditions.

The diagnostics identify the Core bottlenecks: in Full, accepted actuation/disturbance/COM ranges reach only 46.3%/54.7%/42.8% of their limits (§4.3), and single-group curricula peak at 56.6–84.6% limit success for the four Core groups (Appendix E.1). Base-DR RCS-AUC is 81.3–88.3% on non-Core groups versus 52.1–64.1% on Core groups (Figure 4a). Core leaves non-Core groups at Base yet reaches 91.0–94.5% isolated success; Full reaches 97.8–99.0% there, but trails Core by 10.0–12.6 pp when Core shifts are added (Tables 13, 24). Across five training seeds, Core/Full achieve 36.8%/28.1% OOD-all success and 62.5%/59.1% CS-AUC. On the fixed-checkpoint 91-combination retest, Core leads on 57 combinations (mean 48.94%/45.57%).

The development diagnostics motivate the selected Core, and the subsequent joint evaluations test this fixed composition. The cross-robot rollouts in Appendix F.2/Figure 9 additionally illustrate transfer of the resulting policy across seven quadrupeds. They do not establish that the same composition remains preferable to Full across robots or tasks. Cross-task validation, including manipulation, remains future work.


AuthorsOpenReview ↗

Response to Reviewer (2/2): Weaknesses

W1: Evaluation metric. Thank you for your constructive suggestion. Original success requires ≥950/1,000 control steps and mean planar error <0.4 m/s. Yaw tracking is included in the training objective (Table 9). Our primary success criterion measures sustained survival and planar command tracking, while yaw is assessed separately. In the fixed-policy diagnostic, Core and DORAEMON have similar observed mean yaw MAE (0.580 and 0.583 rad/s); we therefore make no claim of improved yaw-tracking accuracy. Under the stricter 0.2 m/s planar threshold, Core retains the highest mean OOD-all success among the compared methods (6.0%), although Base-DR and Direct-wide have higher strict CS-AUC (38.1%/36.7% versus Core's 32.7%). These results support improved endpoint robustness under severe joint physical shifts, with a trade-off in tracking precision across the severity range.

Core's Base tracking error is 0.106 m/s versus 0.076/0.079 m/s for Base-DR/Direct-wide. Joint-gate ablations and matched restoration tests assess the mechanism separately.

Table R1. Success/CS-AUC are recomputed from Table 2's original joint seven-group episodes; 0.4 m/s values are unchanged. Yaw uses one fixed checkpoint per method over evaluation seeds 322–326 (Linear: 14k checkpoint) and is excluded from success.

MethodOOD-all success (%), 0.4 / 0.2 m/sCS-AUC (%), 0.4 / 0.2 m/sOOD-all yaw MAE ↓ (rad/s)
HORIZON (Core)36.8 / 6.062.5 / 32.70.580
Base-DR17.4 / 3.441.4 / 38.10.632
Direct-wide DR10.3 / 3.844.9 / 36.70.620
OpenAI ADR19.8 / 3.661.5 / 30.50.628
HORIZON + Active-DR14.7 / 0.528.7 / 3.30.681
Linear†22.0 / 3.251.4 / 18.90.861
DORAEMON18.9 / 4.254.8 / 29.90.583
GRAM22.7 / 2.334.3 / 4.90.772

† Planar results use the last successful pre-expansion checkpoint, as in Table 2.

W2: Different budgets. Thank you for asking about the allocation. Core/Full use approximately 45k/60k updates. Despite 15k more updates, Full scores lower: 28.1%/36.8% OOD-all and 59.1%/62.5% CS-AUC (Table 2). Core's late expansion had largely converged. At a common nominal budget of approximately 45k updates, each of five independent training runs per configuration returns its accepted checkpoint closest to 45k. Full obtains 22.9 ± 1.8% OOD-all success and 58.1 ± 0.4% CS-AUC, with mean Base tracking error of 0.122 m/s. Core obtains 36.8 ± 2.5% and 62.5 ± 1.2%, respectively; all ± values denote sample SD across the five trained models. The ordering therefore persists at a common budget.

Both finish the current phase and return latest accepted. Core training records through 49k show 11/13 phases accepted in the first 15k, with normalized difficulty increasing from 0.10 to 0.35. After 30k, only 3/14 phases are accepted: the 32k actuation expansion and increments of 0.002/0.0004, with none between approximately 34.5k and 49k. Smaller retries can still fail candidate mastery or retention, explaining the late slowdown and allocation.

W3: Isolation and negative cases. Thank you for these detailed comments. Q1/G1 addresses final selection; Appendix D.2 isolates restoration with matched parent failures, accepted ranges, continuation budgets, and endpoints. P1–P6 (5 positive, +3.55 ± 2.74 pp) expand to all eleven eligible events (9 positive, +2.47 ± 2.73 pp): collapse 5/7, +2.49 ± 3.20; non-collapse 4/4, +2.45 ± 2.11. P7 shares the P3–P6 batch; P8–P11 are preregistered new parents. Both negative cases, P5/P10, remain included (General, G2). Immediately after retraction, P1 Base success is 2.87% retaining the failed learner versus 98.11% with Restore. For non-collapse P2, the CS-AUC gap grows from 3.37 pp immediately to 5.21 pp at the endpoint. Restoration prevents failed-state carryover and improves subsequent learning under matched ranges/budgets.

W4: Hardware. Thank you for your suggestions. Hardware tests cover one platform against Base-DR; revision §6 states this scope. We plan evaluation on manipulation and more complex locomotion tasks, with adaptive baselines on hardware.


After reading the rebuttal and revised manuscript, I believe the authors have substantially addressed my main experimental concerns. Some limitations remain, including the incremental nature of the contribution, sensitivity to the evaluation threshold, and limited validation across tasks and hardware platforms. Nevertheless, these concerns now primarily limit the scope rather than invalidate the method. I therefore update my rating from 2 to 3.

Thank you for carefully reviewing our rebuttal and revised manuscript, and for recognizing that we have substantially addressed your main experimental concerns. We appreciate your constructive feedback and will use it to guide future research, particularly on evaluation criteria and validation across tasks and hardware platforms.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful reviews. We have revised the PDF to address the main concerns; changes are marked in blue.

  • Restoration and the return rule (14Bc, KmkS). Table 2 now uses the latest accepted checkpoint for w/o rollback. Table G1 below compares its terminal and latest accepted checkpoints; this comparison remains in the rebuttal rather than the paper appendix.
  • Evaluation metric (14Bc). We add an analytical reference for an ideal policy with zero translation. Appendix Table 11 reports stricter planar thresholds and fixed-policy yaw diagnostics, with one sentence in the main text pointing to these results (Table R1 in our response).
  • Restoration across parent trajectories (F3bY, 14Bc, KmkS). We update Table G2 with all five additional events satisfying the selection rule (P7–P11), beyond P1–P6. Four are from new parents selected under a rule registered before launch (Table G2 below; §4.2 and Appendix D.2, Table 18).
  • Core/Full training budgets (14Bc, KmkS). Appendix B.4/Table 12 specifies budgets and return rules; our responses below add the five-training-seed common-budget comparison.
  • Sensitivity (14Bc). We add one-factor gate-parameter branches from the same accepted HORIZON checkpoint (Appendix D.4, Table 20).
  • Settings and cost (KmkS). Q3 specifies expansion, failure refinement, stopping conditions, and randomization frequency, and reports interactions and runtime for a Core trajectory accumulated to approximately 45k updates. The search rules and randomization timing appear in Appendices A.2/B.1; the cost example remains in the rebuttal.
  • Novelty (F3bY). We add the transaction view in §3 and a mechanism comparison in Appendix C.1, Figure 6.

Table G1. Complete training procedures under two return rules. Evaluation follows Table 2; entries are mean ± SD. The w/o rollback latest accepted checkpoint is selected solely from its own gate logs, before any OOD evaluation.

Method and returned checkpointOOD-all success (%)Base tracking error ↓ (m/s)CS-AUC (%)
HORIZON (Core), latest accepted36.8 ± 2.50.106 ± 0.00262.5 ± 1.2
w/o rollback, terminal26.3 ± 2.60.215 ± 0.00760.4 ± 1.1
w/o rollback, latest accepted27.8 ± 1.50.190 ± 0.00460.9 ± 0.7

Returning the latest accepted checkpoint improves w/o rollback by 1.5 percentage points (pp) in OOD-all success and 0.5 pp in CS-AUC, and reduces its Base tracking error by 0.025 m/s. Under the same return rule, HORIZON retains a 9.0 pp advantage in mean OOD-all success, a 1.6 pp advantage in CS-AUC, and 0.084 m/s lower Base tracking error. Final checkpoint selection therefore explains part of the original gap. The separate contribution of restoring state during training is supported by the matched continuations in Appendix D.2.

Table G2. Matched restoration results for all eligible parent events. Within each event, Restore and Range-only use the same continuation budget and are evaluated at the matched endpoint. Entries are endpoint ΔCS-AUC, Restore − Range-only (pp). CS-AUC sweeps the failed group from Base to its search limits, with all other groups at Base; this differs from G1's joint seven-group severity path. P8–P11 come from new parents selected under a rule registered before launch.

EventFailed groupStratumΔCS-AUC
P1actuationcollapse+7.60
P2disturbanceno collapse+5.21
P3actuationcollapse+1.95
P4actuationno collapse+2.99
P5actuationcollapse−0.38
P6actuationcollapse+3.94
P7disturbanceno collapse+0.96
P8–P10actuationcollapse+2.04 / +4.35 / −2.09
P11disturbanceno collapse+0.63
P1–P6, mean ± SD+3.55 ± 2.74
P1–P11, mean ± SD (9/11 positive)+2.47 ± 2.73
Collapse, n = 7 (5 positive)+2.49 ± 3.20
No collapse, n = 4 (4 positive)+2.45 ± 2.11

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.