acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Reinforcement Learning over Adversarial Skill Priors for Long-Horizon Humanoid Navigation

Abstract

Long-horizon humanoid navigation couples dynamic balance with goal-directed exploration across large environments. Contrastive reinforcement learning (CRL) enables goal-conditioned exploration without reward engineering, but end-to-end training on a 28-DoF humanoid forces joint-space exploration, spending most of the sample budget discovering basic locomotion rather than navigating. Pretrained motor priors provide stable gaits from decoded latents, yet they are typically driven by policy-gradient methods that struggle with long horizons. We propose ASE-CRL, which pairs a frozen Adversarial Skill Embedding (ASE) prior learned from motion capture with a contrastive goal-conditioned learner. Neither tier requires reward shaping: ASE trains via adversarial imitation, and CRL learns purely through goal identification. The advantage of the contrastive objective over reward-driven learning grows with maze scale once exploration operates over skills rather than joint targets. ASE-CRL solves every maze layout in our suite, exceeding 97% success on the largest, whereas reward-driven and end-to-end baselines collapse as navigation horizons grow. On whole-body pushing, ASE-CRL is the only goal-conditioned method that transfers from character to object goals, where hindsight-relabeled SAC fails. Finally, we release HGC-NavBench, an Isaac Lab benchmark suite for online humanoid goal reaching.

Primary area:
reinforcement learning
Keywords:
Contrastive Reinforcement Learning, Adversarial Skill Embeddings (ASE), Hierarchical Reinforcement Learning, Goal-Conditioned Reinforcement Learning, Humanoid Navigation, Motor Skill Priors, Long-Horizon Exploration, Whole-Body Manipulation, Isaac Lab, Hindsight Relabeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9QEMOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies long-horizon humanoid goal reaching by separating locomotion from high-level decision making. Its method, ASE-CRL, uses a frozen ASE motion prior and trains a contrastive goal-conditioned policy to select latent skills. It also introduces HGC-NavBench, with four navigation mazes and two object-pushing tasks. Experiments over five seeds compare against end-to-end PPO and CRL, as well as PPO and SAC+HER using the same prior. ASE-CRL shows substantial gains on the larger mazes and open-arena pushing, although maze pushing remains difficult. The appendix includes controls for network depth, observations, control frequency, and evaluation time. The evaluation uses fixed starts and the training goal sets, so the results mainly establish performance within these task settings.

Critical strengths
  1. The results on the larger mazes are convincing. On Giant, ASE-CRL reaches 97.4% success, compared with 82.1% for SAC+HER and 37.3% for PPO using the same prior. This is a useful practical result, and the five-seed evaluation gives it more weight than a few successful examples would.

  2. I appreciate the additional controls and the benchmark contribution. The authors check several fairly obvious alternative explanations, including SAC depth, proprioceptive observations, and decision frequency. Together with the task suite and supplied implementation, this makes the paper useful even if the algorithm itself is mostly a combination of existing ideas.

Critical weaknesses
  1. I am not sure there is much new in the learning algorithm itself. It mostly seems to be deep CRL on top of ASE. The comparisons could also be stronger. I appreciate the recent deep-CRL and JaxGCRL configurations, and the deeper SAC control, but increasing SAC's depth is not quite the same as testing a stronger recent reward-driven method, such as an appropriately adapted SimBaV2. The results show that this combination works well against the chosen baselines; its position relative to recent alternatives is less clear.

  2. Some explanations seem stronger than the experiments support. CRL and SAC+HER differ in future-goal sampling and relabeling as well as the critic objective, so I am not sure the gap can be attributed to the objective alone. Also, the policies are trained and evaluated on the same goal sets from fixed starts. That is still a useful evaluation, but it leaves open how much of the result carries over to new goals or starts.

Questions for authors
  1. Beyond combining deep CRL with ASE, what is the main methodological contribution? Is the intended contribution mainly the empirical finding and benchmark?

  2. How does the method compare with a stronger recent reward-driven learner, for example an appropriately adapted SimBaV2, using the same frozen prior? The deeper SAC control is useful, but does not fully answer this.

  3. How much do future-goal sampling and the relabeling ratio affect SAC+HER? Can the claimed benefit of the contrastive objective be separated from these choices?

  4. Is there any evaluation on held-out goals or different starts? If not, could the claims about goal-reaching generalization be limited to what the current protocol tests?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9QEM

Thank you for the balanced assessment.

I am not sure there is much new in the learning algorithm itself. It mostly seems to be deep CRL on top of ASE.

Novelty of the algorithm. We agree, and this is deliberate: ASE-CRL modifies neither the ASE prior nor the contrastive learner, so that any effect is attributable to their pairing and not to an added ingredient. We do not claim a new learning algorithm; what we claim instead is answered under Q1.

  1. Beyond combining deep CRL with ASE, what is the main methodological contribution? Is the intended contribution mainly the empirical finding and benchmark?

Q1. Methodological contribution. The contribution is the finding in G1 (in our general response), and the combination is built to isolate it: modifying neither component makes the effect attributable to the pairing rather than to a new ingredient. The paper now states the finding as the primary contribution, with the benchmark that lets others test it against further objectives.

  1. How does the method compare with a stronger recent reward-driven learner, for example an appropriately adapted SimBaV2, using the same frozen prior? The deeper SAC control is useful, but does not fully answer this.

Q2. A stronger recent reward-driven learner. We agree that deepening SAC is not the same as testing a stronger learner, and ran SimBaV2 over the same frozen prior with ASE-SAC+HER's sparse reward and relabeling, at the same data and gradient budget. On Giant it reaches 83.6 ± 5.8, against 82.1 ± 5.6 for ASE-SAC+HER and 97.4 ± 0.5 for ASE-CRL, and 53.9 against 92.1 on the longest routes (details under VQPH Q8). It is now part of Table 1.

CRL and SAC+HER differ in future-goal sampling and relabeling as well as the critic objective, so I am not sure the gap can be attributed to the objective alone.

Attributing the gap. We agree that the submission's wording went further than its experiments. The two arms differed in critic, in hindsight-goal sampling and relabeling ratio, and in depth. We have now controlled relabeling (Q3) in addition to depth, and both together, which leaves the critic, and the wording is corrected in the introduction, the results section and the conclusion (G2).

  1. How much do future-goal sampling and the relabeling ratio affect SAC+HER? Can the claimed benefit of the contrastive objective be separated from these choices?

Q3. Future-goal sampling and relabeling ratio. We ran this control. ASE-SAC+HER with geometric future sampling and every transition relabeled, matching ASE-CRL and changing nothing else, reaches 76.8 ± 4.3 on Giant (default relabeling: 82.1 ± 5.6; ASE-CRL: 97.4 ± 0.5). The relabeling scheme does not explain the gap: given to the TD learner it lowers success, if anything. Combined with depth 32, so that only the critic differs from ASE-CRL, it reaches 55.3 ± 7.7 and is still slowly improving at the end of the budget. With depth also ruled out, this points to the critic (G2).

  1. Is there any evaluation on held-out goals or different starts? If not, could the claims about goal-reaching generalization be limited to what the current protocol tests?

Q4. Held-out goals or different starts. See G3. Unseen goal positions are solved at the trained rate, and on pushing ASE-CRL reaches 86.7% on goals beyond the trained range. New maze starts are solved at close to the trained rate on trained routes and at about half off them, for all methods. Because hindsight relabeling covers every visited position, the off-centre test measures interpolation, so we also withheld whole goal cells. ASE-CRL trained on Hard with 9 of the 45 goal cells never commanded keeps 99.3% on the trained cells, reaches 94.4 ± 3.9 on withheld corridor cells (which relabeling still covers on the way to other goals) and 73.3 ± 25.6 on withheld dead ends, against 100% for runs trained on every cell. A goal never commanded and never passed is therefore reached most of the time but not reliably, and we state this as a limit. On starts, ASE-CRL trained with randomized start cells reaches 98.2% from new starts off the trained routes (53.1% with the fixed start; G3), and ASE-SAC+HER trained the same way 89.2% (46.1%), so the drop off-route belongs to the fixed-start protocol.

ASE-CRL shows substantial gains on the larger mazes and open-arena pushing, although maze pushing remains difficult.

Maze-embedded pushing. The new breakdown shows where it fails: success falls with the number of turns on the block's route (ASE-CRL 55.3 ± 26.3 on straight routes, 18.8 ± 15.2 with two or more turns), since a block pushed past a corner cannot be pulled back.

Thanks for the extra experiments. I am still a bit unsure about the “unseen starts” claim. If training now samples all free cells, aren’t those evaluation starts already covered? The 98.2% seems to show that broader coverage helps, rather than resolving the generalization concern.

The SimBaV2 result helps too, though running it well below its default update ratio leaves me wondering how much the comparison depends on that choice.

I still lean weak accept, but will lower my confidence from 2 to 1, as I am less certain how strongly these experiments support the revised claims.

AuthorsOpenReview ↗

Response to Reviewer 9QEM

Thank you for the follow-up. Both points are fair, and we answer them directly.

If training now samples all free cells, aren’t those evaluation starts already covered? The 98.2% seems to show that broader coverage helps, rather than resolving the generalization concern.

Yes. With randomized starts every free cell is a training start, so those starts are no longer unseen, and the 98.2% is not evidence of generalization to unseen starts. What it shows is narrower: the off-route drop of the fixed-start runs is a coverage effect and not a capacity limit of the contrastive learner or of direct destination-to-skill selection, since the same learner reaches all start-goal pairs once its data covers them.

The evidence on genuinely unseen starts is the fixed-start evaluation, and there the answer is mixed. New starts on a trained route are reached at close to the trained rate (ASE-CRL: 96.1 against 99.7 on Hard, 92.9 against 97.0 on Giant), while starts off the trained routes roughly halve for every method (ASE-CRL 53.1 on Hard and 42.8 on Giant; ASE-SAC+HER 46.1 and 34.5). The off-route drop is one of training coverage, and zero-shot generalization to starts off the trained routes remains limited for all methods.

Generalizing well to unseen starts, and to unseen mazes, would require a different high-level policy: one that receives the goal relative to the character, carries a recurrent memory or an explicit map as context, and is trained on a variety of environments. This goes beyond the scope of our paper, and we leave it to future work. Our focus, as in prior contrastive RL work, is reliability over long horizons in a given layout, here in combination with ASE; generalization across starts and layouts is a natural next step.

The SimBaV2 result helps too, though running it well below its default update ratio leaves me wondering how much the comparison depends on that choice.

We matched the update-to-data ratio across all high-level arms, ASE-CRL included, so that each receives the same data and the same number of gradient steps. Our result should therefore be read as: at matched data and gradient budget, the stronger architecture does not close the gap (83.6 against 97.4). Whether SimBaV2 closes it at its own default ratio remains open.

Neither point affects our central claim, which concerns the start the policy trains on: there, ASE-CRL stays above 92% on every route length on Giant, while ASE-SAC+HER and ASE-SimBaV2+HER fall to 76.4% and 53.9% on the longest routes.

We appreciate your continued engagement.

Reviewer_VQPHOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a hierarchical framework for humanoid goal-directed navigation that combines a frozen Adversarial Skill Embedding (ASE) prior with contrastive reinforcement learning. The authors evaluate the framework on simulated maze navigation and whole-body pushing tasks and report improvements over the compared baselines.

Critical strengths

The methodology in the paper is worthwhile to be considered, showing some novelty. The benchmark could also be a useful resource for online humanoid goal-conditioned learning.

Critical weaknesses

The motivation and principal contribution are unclear. The experiments show that ASE-CRL works as a combination but do not isolate the causal contribution of contrastive learning. Baseline coverage is inadequate, particularly planning with low-level control, other hierarchical goal-conditioned methods, and stronger end-to-end methods. These gaps prevent a convincing assessment of the claimed contribution and explain the rejection recommendation.

Questions for authors
  1. I read the introduction, but I found that the motivation does not make sense. In particular, when discussing progress in humanoid research, the existing studies mentioned in the paper are all from before 2023. However, to my knowledge, current humanoid research has only gained momentum over the past three years. Therefore, I ask the authors to carefully verify the insights they propose on this basis.

  2. The experiments only demonstrate that the ASE-CRL combination is effective, but do not sufficiently establish whether the performance improvements arise from the contrastive learning mechanism emphasized by the authors or from other factors that change simultaneously. Please provide additional experiments and evidence to establish this.

  3. The paper claims to adopt a hierarchical end-to-end architecture. However, existing hierarchical control and planning methods can also achieve goal-directed navigation, and this paper has not demonstrated the advantages of ASE-CRL over these approaches in success rate, path efficiency, training cost, information requirements, or generalization. Please clarify whether the paper’s primary contribution is navigation capability, reduced reward design, or new insight into the interaction between skill priors and CRL, and provide evidence directly corresponding to that contribution.

  4. I noticed that the authors cite Wang’s paper several times near the beginning of the introduction. However, because the authors have not provided hyperlinks for each reference, I was unable to locate this paper in the reference list. Please verify this.

  5. Why does the main body of the paper need to devote space to experimental settings and environment configurations? If the hardware configuration does not directly support conclusions about training time or computational efficiency, please move the relevant information to the appendix.

  6. The authors claim that HGCNavBench has been released, but I could not find a corresponding link in the paper. Please carefully verify this.

  7. The paper’s end-to-end policy clearly considers the humanoid’s pose at the current or future time step. However, the six BEV images in the experiments do not show any pose information, or even the robot. Do the authors believe that these details can be omitted from the visualizations and therefore do not need to be shown?

  8. Regarding the experimental baselines, I noticed that the paper does not cover representative methods combining explicit planning with low-level motion control, nor does it sufficiently compare other hierarchical goal-conditioned methods. I also suggest using better-performing end-to-end methods as comparison baselines here, such as end-to-end FastTD3, ASE-FastTD3, and ASE-SimbaV2-SAC+HER.

  9. I noticed that the reference list contains five consecutive papers by Xue Bin Peng. Are all of these citations necessary? Please verify this.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VQPH (1/3)

Thank you for the detailed list. The three substantive points first (Q2, Q3, Q8), then the remainder in order.

  1. The experiments only demonstrate that the ASE-CRL combination is effective, but do not sufficiently establish whether the performance improvements arise from the contrastive learning mechanism emphasized by the authors or from other factors that change simultaneously. Please provide additional experiments and evidence to establish this.

Q2. Isolating the contribution of contrastive learning. See G2. In summary: depth is ruled out (82.1 to 83.1); the compact interface is ruled out (adding the 140-dimensional proprioceptive state leaves ASE-CRL at 97.9 and halves ASE-PPO); decision rate and time budget are controlled in the appendix; relabeling is ruled out (new: ASE-SAC+HER with ASE-CRL's relabeling reaches 76.8 ± 4.3, below its default 82.1), and so are both together (depth 32 plus ASE-CRL's relabeling: 55.3 ± 7.7). A stronger recent TD learner over the same prior does not close the gap either (SimBaV2, 83.6 ± 5.8; Q8).

  1. The paper claims to adopt a hierarchical end-to-end architecture. However, existing hierarchical control and planning methods can also achieve goal-directed navigation, and this paper has not demonstrated the advantages of ASE-CRL over these approaches in success rate, path efficiency, training cost, information requirements, or generalization. Please clarify whether the paper’s primary contribution is navigation capability, reduced reward design, or new insight into the interaction between skill priors and CRL, and provide evidence directly corresponding to that contribution.

Q3. Primary contribution. One claim: over a frozen skill prior, a reward-free contrastive learner stays reliable over a longer navigation horizon from the start it trains on than reward-driven learners given the same prior, and the margin grows with the horizon. Navigation capability and the absence of reward design are consequences of this claim, not separate ones. The evidence, on Giant from the training start with no map and no waypoints:

Success (%)All routes (Table 1)6 or more turnsLongest quartile (48 to 62 m)
ASE-PPO37.319.1 ± 2.910.6 ± 1.5
ASE-SAC+HER82.180.7 ± 10.876.4 ± 14.6
ASE-CRL97.495.7 ± 1.592.1 ± 3.3

In planner terms, ASE-CRL reaches with no waypoints (97.0) the level ASE-SAC+HER reaches only when a map-based planner hands it a waypoint every 2 m (97.0). The axes you list point the same way:

  • Success rate: the table above.
  • Path efficiency (new): median walked distance over shortest route is 1.00 ± 0.01 for ASE-CRL, 0.99 ± 0.02 for ASE-SAC+HER and 1.24 ± 0.05 for ASE-PPO.
  • Training cost: measured wall-clock hours for every run, including all new controls, in Appendix A.15.
  • Information requirements: position, heading and goal; no map and no task reward at either tier.
  • Generalization (new): unseen goal positions are reached at the trained rate (G3). Off the trained routes every method roughly halves, and there the TD critic tolerates sparser waypoints (Q8), which is why the claim names the start the policy trains on.
AuthorsOpenReview ↗

Response to Reviewer VQPH (2/3)

  1. Regarding the experimental baselines, I noticed that the paper does not cover representative methods combining explicit planning with low-level motion control, nor does it sufficiently compare other hierarchical goal-conditioned methods. I also suggest using better-performing end-to-end methods as comparison baselines here, such as end-to-end FastTD3, ASE-FastTD3, and ASE-SimbaV2-SAC+HER.

Q8. Baselines.

  • Stronger off-policy learners over the prior. We added SimBaV2 (Lee et al., 2025) as a high-level learner over the same frozen prior, with the sparse goal reward and hindsight relabeling of ASE-SAC+HER. Architecture, categorical critic, optimizer and entropy settings follow the official defaults; data, batch size and gradient budget are matched to the other high-level arms. On Giant it reaches 83.6 ± 5.8, against 82.1 ± 5.6 for ASE-SAC+HER and 97.4 ± 0.5 for ASE-CRL (five seeds, Table 1 protocol). The stronger architecture does not change the ordering, and on the longest quartile of routes (48 to 62 m) it falls to 53.9 ± 12.4, against 76.4 for ASE-SAC+HER and 92.1 for ASE-CRL, so the margin is again largest on the longest routes. It is now in Table 1, Figure 3 and the route-length table of Appendix A.10. One caveat: matching the gradient budget keeps SimBaV2 at a much lower update-to-data ratio than its own default. FastTD3 was not run, and we make no claim about it.
  • Planning with low-level control. This uses a map that our learners do not receive, so it answers a different question. We now report it as a reference (Appendix A.10): a shortest-path planner on the maze grid that feeds waypoints one cell ahead to the same frozen high-level policies reaches 95.8 ± 1.9 (ASE-CRL) and 97.0 ± 2.0 (ASE-SAC+HER) on Giant from the training start, and 93.7 ± 2.1 and 96.9 ± 1.1 from new starts off the trained routes. The planner (a shortest-path search on the grid, equivalent to A* there) hands its waypoints to the learned high-level policy and not to the ASE low level directly: the frozen prior is driven by a skill latent, not by a target position, so a learned policy is needed to turn a waypoint into skill choices. This does not make the navigation policy redundant. It is still what moves the character between waypoints, and this result needs a waypoint every 2 m: at three cells of lookahead off-route success falls to 71.1 ± 4.8 for ASE-CRL and 84.5 ± 7.1 for ASE-SAC+HER. A policy that is reliable over a longer horizon needs sparser waypoints and less of the map, which matters in environments where dense planning is costly or the map is incomplete. On that axis the learners differ in both directions: on the trained routes ASE-CRL reaches the planner's level with no waypoints at all (97.0 from the training start, against 86.1 for ASE-SAC+HER), while off them ASE-SAC+HER tolerates sparser waypoints.
  • Other hierarchical goal-conditioned methods. The planner experiment above bounds what a subgoal method can add over the same frozen policies: nothing from the training start (ASE-CRL 97.0 direct, 95.8 with waypoints every cell), and up to 93.7 from starts off the trained routes, where direct selection reaches 42.8. That off-route regime is where a learned subgoal method should be compared, and the released benchmark supports it.
AuthorsOpenReview ↗

Response to Reviewer VQPH (3/3)

  1. I read the introduction, but I found that the motivation does not make sense. In particular, when discussing progress in humanoid research, the existing studies mentioned in the paper are all from before 2023. However, to my knowledge, current humanoid research has only gained momentum over the past three years. Therefore, I ask the authors to carefully verify the insights they propose on this basis.

Q1. We added recent humanoid and motor-prior work (MaskedMimic, Tessler et al., 2024; CLoSD, Tevet et al., 2025; OmniH2O, He et al., 2024; HOVER, He et al., 2025; HumanoidBench, Sferrazza et al., 2024; FastTD3, Seo et al., 2025; SimBaV2, Lee et al., 2025) and narrowed the motivating claim to high-level policies that act in a latent skill space.

  1. I noticed that the authors cite Wang’s paper several times near the beginning of the introduction. However, because the authors have not provided hyperlinks for each reference, I was unable to locate this paper in the reference list. Please verify this.

Q4. Citations are now hyperlinked and coloured. The paper is Wang et al. (2025), "1000 Layer Networks for Self-Supervised RL" (NeurIPS 2025); it is listed under K. Wang, next to the unrelated T. Wang et al. (2023), which may have caused the confusion. The entry now carries a link to the proceedings.

  1. Why does the main body of the paper need to devote space to experimental settings and environment configurations? If the hardware configuration does not directly support conclusions about training time or computational efficiency, please move the relevant information to the appendix.

Q5. Simulator lineage, hardware and bootstrap details moved to the appendix; the space now holds the paragraph on what the comparison isolates.

  1. The authors claim that HGCNavBench has been released, but I could not find a corresponding link in the paper. Please carefully verify this.

Q6. The code is in the anonymous supplementary material, now updated with the code and configurations of every new experiment and the training logs of every reported run, from which the success tables and learning curves can be regenerated without retraining. The paper now also links an anonymized repository: https://github.com/hgc-navbench-anonsub/HGC-NavBench.

  1. The paper’s end-to-end policy clearly considers the humanoid’s pose at the current or future time step. However, the six BEV images in the experiments do not show any pose information, or even the robot. Do the authors believe that these details can be omitted from the visualizations and therefore do not need to be shown?

Q7. At 24 m by 32 m the character is a few pixels in a top-down view. The figure is re-rendered with the humanoid at its start cell in every panel and, for the five larger arenas, a close-up inset of it.

  1. I noticed that the reference list contains five consecutive papers by Xue Bin Peng. Are all of these citations necessary? Please verify this.

Q9. They are five distinct works, each cited for a different role: DeepLoco (a hierarchical locomotion controller, as related work on hierarchy), DeepMimic (per-clip tracking objectives), AMP (the adversarial imitation objective ASE builds on), ASE (the prior we use) and MimicKit (the framework, motion data and character our implementation is built on). We kept all five.

We hope these additions address your concerns. If they do, we would be grateful if you would consider updating your assessment; if any remain, we are happy to provide further analyses during the discussion period.

Reviewer_GtTdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether contrastive reinforcement learning becomes more effective for long-horizon humanoid navigation when it operates over a pretrained locomotion controller. The proposed ASE-CRL combines a frozen Adversarial Skill Embedding prior with a goal-conditioned contrastive learner. The results are strong within the evaluated maze environments, particularly on the largest layout. The benchmark and experiments across five seeds are useful contributions.

It also compares using the same motor prior, along with the ablations on proprioception (not the original bio-mechanical and neuroscience definition of it), network depth, and control frequency. The authors also acknowledge several limitations clearly. My main questions concern the interpretation of the hierarchy, the mechanism behind the improvement, and how far the approach extends beyond the demonstrated navigation setting.

Critical strengths

The paper provides strong empirical evidence that a pretrained locomotion controller makes contrastive learning effective for long-horizon humanoid navigation. On the largest maze, ASE-CRL achieves 97.4% success, compared with 82.1% for SAC+HER and 37.3% for PPO using the same motor prior. These comparisons, supported by several ablations, make the combination convincing within the tested setting.

Critical weaknesses

Critical weakness: The experiments establish performance more clearly than they establish the underlying mechanism or generality. The method directly connects distant position goals to short skill commands, but offers limited evidence about how this connection scales beyond familiar layouts and goals. Fixed starting positions, evaluation on the training goal set, and a locomotion-only means leave broader applicability uncertain. The absence of intermediate goals is not itself a flaw. However, the unanswered question is when this direct approach reaches its limits.

Questions for authors

Could the authors explain more concretely how the contrastive learner bridges this gap? In particular:

  • What is the distribution of temporal and spatial distances between sampled states and their hindsight goals, and how does it change during training?
  • Does performance deteriorate systematically with route length, number of turns, or distance to the goal?
  • Would explicit intermediate goals or a subgoal-based baseline help establish when direct destination-to-skill selection is sufficient?

Could the authors evaluate varied starting positions or held-out goal locations within the same layout? More broadly, what would need to change to handle a new layout, given that the high-level policy receives position and orientation but no explicit environmental observation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer GtTd (1/2)

Thank you for the questions. Each now has a measurement, and the new experiments locate the main limitation you raised.

Fixed starting positions, evaluation on the training goal set, and a locomotion-only means leave broader applicability uncertain.

Broader applicability. Each of the three is now tested. Fixed starts: new starts on a trained route are reached at close to the trained rate, starts off them roughly halve for every method, and training with randomized starts shows that this drop is one of coverage (below). Training goal set: unseen goal positions are reached at the trained rate, and withheld goal cells mostly (below). Locomotion only: on open-arena pushing, where the character must move a cube, ASE-CRL reaches 86.7% on goals beyond the trained cube-to-goal range, against 12.0% for ASE-PPO and 0.3% for ASE-SAC+HER (G3).

Could the authors evaluate varied starting positions or held-out goal locations within the same layout?

Varied starts. Every training episode of the reported runs leaves the same cell, so a start off the routes from it requires composing segments that never occurred within one trajectory, the known stitching limitation of learning from a single start (Ghugare et al., 2024; Park et al., 2025). We tested whether this is a property of the protocol or of the learner by training ASE-CRL and ASE-SAC+HER on Hard with each episode starting in a random free cell, and changing nothing else (Hard only, for compute). Success (%), five seeds:

HardTraining startNew start on a trained routeNew start off the trained routes
ASE-SAC+HER, fixed-start training96.7 ± 2.696.8 ± 2.246.1 ± 1.2
ASE-CRL, fixed-start training99.7 ± 0.796.1 ± 1.753.1 ± 3.8
ASE-SAC+HER, randomized-start training94.1 ± 5.795.1 ± 1.889.2 ± 3.8
ASE-CRL, randomized-start training99.0 ± 1.197.7 ± 1.698.2 ± 0.3

For the randomized-start rows these starts are covered in training, so they measure coverage, not generalization. The off-route drop is one of coverage, for both learners. With equal coverage, ASE-CRL remains the more successful learner from every start. With the fixed start, the TD-based ASE-SAC+HER is affected at least as much (on Giant, 34.5 off the trained routes against 42.8 for ASE-CRL).

Held-out goal locations. Goals 0.70 to 0.85 m off the cell centre, outside the 0.5 m success radius, are reached as often as trained goals by every method. Because relabeling covers every visited position, we also withheld 9 of the 45 goal cells on Hard during training. Success on the trained cells is unchanged (99.3%), withheld cells on the way to other goals reach 94.4 ± 3.9, and withheld dead ends, never passed on the way to another goal, reach 73.3 ± 25.6 (9 episodes per seed). We state the latter as a limit.

Does performance deteriorate systematically with route length, number of turns, or distance to the goal?

Route length and turns. On Giant, ASE-CRL stays above 92% for every route length and turn count; the baselines degrade with both. Success (%) from the training start:

0 to 1 turns2 to 34 to 56 or moreLongest quartile (48 to 62 m)
ASE-PPO86.7 ± 11.683.3 ± 5.636.2 ± 6.619.1 ± 2.910.6 ± 1.5
ASE-SAC+HER97.5 ± 2.398.3 ± 2.589.0 ± 15.980.7 ± 10.876.4 ± 14.6
ASE-CRL98.3 ± 2.3100.0 ± 0.098.6 ± 2.195.7 ± 1.592.1 ± 3.3

More broadly, what would need to change to handle a new layout, given that the high-level policy receives position and orientation but no explicit environmental observation?

New layouts. The high level observes absolute position, heading and goal, so the policy is specific to its layout. Generalizing to unseen layouts needs three changes to the high-level observation, none to the frozen prior: a goal relative to the character, exteroception (depth images or a LiDAR scan) so walls are observed rather than memorized, and memory (a recurrent policy or an explicit map the agent updates) so dead ends are remembered. The high level would then train over a distribution of layouts. The limitations section now states this.

What is the distribution of temporal and spatial distances between sampled states and their hindsight goals, and how does it change during training?

Distribution of hindsight goals. The hindsight goal is the state a geometrically distributed number of high-level steps ahead (γ = 0.99 per 6 Hz decision, truncated at the episode end). Untruncated this is a mean of 99 decisions (16.5 s); with truncation the mean offset is about 10 s on Giant and 6 s on Hard, with a long tail up to the full episode, so every batch mixes near and far goals. We did not log how the spatial distribution evolves during training; the revision states this, with the distribution above, in Appendix A.4.

AuthorsOpenReview ↗

Response to Reviewer GtTd (2/2)

However, the unanswered question is when this direct approach reaches its limits.

When does the direct approach reach its limits? From the training start, not within our scales: the first visible erosion is the longest quartile of Giant, where ASE-CRL still reaches 92.1% on routes of 48 to 62 m. The limit appears off the trained routes. With a fixed start, success from new start cells off those routes roughly halves for every method, because reaching the goal from them requires composing segments that never occurred within one trajectory. This is a limit of coverage and not of direct destination-to-skill selection: once those starts are covered in training, ASE-CRL reaches 98.2% from them (Varied starts, comment 1/2). Where coverage cannot be broadened, temporal data augmentation (Ghugare et al., 2024), a temporal-difference contrastive objective (Zheng et al., 2024) and a metric-structured critic (Myers et al., 2024) are reported to improve stitching; their evaluation over the skill prior is future work.

Could the authors explain more concretely how the contrastive learner bridges this gap?

How the contrastive learner bridges the gap. The critic is fitted to pairs of a state and a future state of the same trajectory, so it learns from data alone, without a reward, which skill choices lead toward a distant goal. With a fixed start, it is low along the corridor from the training start to the goal and high elsewhere, even close to the goal (new heatmaps, Appendix A.10). Its rank correlation with walking distance is 0.84 on trained routes but 0.53 over all visited states on Giant, where the TD critic of ASE-SAC+HER reaches 0.84. It therefore encodes progress along the experienced routes rather than a general metric, and we report plainly that ASE-CRL's advantage from the training start does not come from a better general metric. The cause is the data and not the objective: with one start, the training pairs all run outward from one cell. On open-arena pushing, where training randomizes cube and goal, the contrastive critic tracks distance well (0.84, against 0.32 for ASE-SAC+HER), and with randomized maze starts its correlation rises from 0.49 to 0.83, matching the fixed-start TD critic (0.83), with a map that falls off from the goal in every direction. Under equal coverage the TD critic is still the closer distance estimate: ASE-SAC+HER trained with the same randomized starts reaches 0.93, against 0.83 for ASE-CRL, yet its policy reaches 89.2% from starts off the trained routes against 98.2% for ASE-CRL. ASE-CRL's advantage therefore does not come from a better metric, with either start distribution.

Would explicit intermediate goals or a subgoal-based baseline help establish when direct destination-to-skill selection is sufficient?

Intermediate goals or a subgoal baseline. We replaced the final goal in the observation of the same frozen policies by a waypoint from a shortest-path planner on the maze grid (equivalent to A* there; the waypoint goes to the learned high-level policy, since the prior is driven by a skill latent and not a position). Success (%) on Giant from new starts off the trained routes:

Final goal (direct)Waypoint 1 cell (2 m) aheadWaypoint 3 cells ahead
ASE-SAC+HER34.5 ± 6.296.9 ± 1.184.5 ± 7.1
ASE-CRL42.8 ± 2.193.7 ± 2.171.1 ± 4.8

The off-route failure is one of goal selection, not control: the frozen policies walk any local segment in any direction. From the training start, waypoints add nothing for ASE-CRL (97.0 direct, 95.8 with a waypoint every cell), so a subgoal method can only help off the trained routes, which randomized training starts already cover. A planner does not replace the navigation policy: the policy covers the distance between waypoints, and how sparse they can be is set by its reliable horizon. On the trained routes ASE-CRL needs no waypoints over up to 62 m, a level ASE-SAC+HER reaches only with a waypoint every cell (86.1 without, 97.0 with); off them ASE-SAC+HER tolerates sparser waypoints (84.5 against 71.1 at three cells). The planner uses the maze map, which no learner receives, so we report it as a diagnostic and upper reference.

the ablations on proprioception (not the original bio-mechanical and neuroscience definition of it)

"Proprioception". Used in the character-animation sense (joint and body state); now defined at first use.

We hope these measurements address your concerns. If they do, we would be grateful if you would consider updating your assessment; if any remain, we are happy to provide further analyses during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers. Three main concerns were shared; we answer them here and refer back to them below.

G1. The contribution is a finding about how skill priors and contrastive learning interact, and the benchmark to test it. Over a frozen motor prior, a reward-free contrastive learner stays reliable over a longer navigation horizon than reward-driven learners given the same prior, and the margin grows with the horizon: on Giant, 97.4% against 82.1% (ASE-SAC+HER) and 37.3% (ASE-PPO), and 92.1% against 76.4% and 10.6% on its longest routes. ASE-CRL leaves both the ASE prior and the contrastive learner unmodified on purpose, so the effect is attributable to the pairing and not to an added ingredient. The abstract and introduction now state this claim directly.

G2. The gap to ASE-SAC+HER is attributed to the algorithm, not to the objective alone. The two arms differ in critic, in hindsight-goal sampling and relabeling ratio, and in depth. Depth was already controlled (82.1 to 83.1 at depth 32 on Giant). We have now also controlled relabeling. ASE-SAC+HER given ASE-CRL's relabeling (geometric future sampling, every transition relabeled, nothing else changed) reaches 76.8 ± 4.3 on Giant, against 82.1 ± 5.6 with its default relabeling and 97.4 ± 0.5 for ASE-CRL (five seeds each, Table 1 protocol). CRL's relabeling therefore does not carry over to the TD learner: it does not close the gap and, if anything, lowers success. Of the three differences, depth and relabeling are now ruled out individually, which leaves the critic. We also tested the two jointly: ASE-SAC+HER at depth 32 with ASE-CRL's relabeling, so that it differs from ASE-CRL only in the critic, reaches 55.3 ± 7.7 [49.7, 61.9]. Its success is still rising at the end of the budget, so the combination mainly slows the TD learner down, but it widens the gap rather than closing it. This is expected: Wang et al. (2025) find that the depth scaling which benefits the contrastive critic does not carry over to SAC in goal-conditioned RL, which is why the reported ASE-SAC+HER uses its standard depth of 4; depth 32 is a fairness control, not a setting that favours the TD learner. The wording is corrected in the introduction, the results section and the conclusion.

G3. New evaluations beyond the fixed start and trained goals. Using the final checkpoints of all three hierarchical methods, we evaluated start cells never used in training, goal positions never commanded in training, and goals beyond the trained range on pushing (Appendix A.10). Success (%):

Giant mazeTraining startNew start on a trained routeNew start off the trained routesOff-centre goal
ASE-PPO37.3 ± 2.832.3 ± 1.99.5 ± 1.634.8 ± 2.9
ASE-SAC+HER86.1 ± 8.888.9 ± 5.234.5 ± 6.287.4 ± 6.2
ASE-CRL97.0 ± 1.292.9 ± 2.142.8 ± 2.197.0 ± 0.7

The first column repeats the submitted protocol on the final checkpoints and is consistent with Table 1, which averages over the last tenth of training. Hard shows the same pattern (Appendix A.10).

  • Unseen goal positions cost nothing. Goals 0.70 to 0.85 m off the cell centre, outside the 0.5 m success radius, are reached as often as trained goals by every method.
  • On pushing, ASE-CRL extrapolates. In the open arena it is unaffected by a moved agent start (97.5 ± 0.6 against 99.0 ± 0.5) and reaches 86.7 ± 0.5 on goals beyond the trained cube-to-goal range, where ASE-PPO reaches 12.0 ± 3.6 and ASE-SAC+HER 0.3 ± 0.4.
  • New maze starts work on trained routes and roughly halve off them, for every method. All training episodes leave one cell, so a start off those routes requires composing segments that never occurred in one trajectory. This is the known stitching limitation of learning from a single start (Ghugare et al., 2024; Park et al., 2025), it affects the TD-based ASE-SAC+HER at least as much, and it is a matter of training coverage and not a capacity limit of the learner: trained with randomized start cells on Hard and otherwise unchanged, so that these starts are now covered in training, ASE-CRL reaches 98.2 ± 0.3 from them, against 53.1 ± 3.8 with the fixed start, and keeps 99.1 from the fixed start (Appendix A.8). ASE-SAC+HER given the same randomized starts improves too, from 46.1 to 89.2 ± 3.8. Zero-shot generalization to starts off the trained routes therefore remains limited for all methods. The failure is in goal selection, not control: given waypoints one cell ahead from a grid planner, the same frozen policies reach 93.7 ± 2.1 (ASE-CRL) and 96.9 ± 1.1 (ASE-SAC+HER) from these starts on Giant (details under Reviewer GtTd).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.