- Rating
- 3
- Confidence
- 1(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies long-horizon humanoid goal reaching by separating locomotion from high-level decision making. Its method, ASE-CRL, uses a frozen ASE motion prior and trains a contrastive goal-conditioned policy to select latent skills. It also introduces HGC-NavBench, with four navigation mazes and two object-pushing tasks. Experiments over five seeds compare against end-to-end PPO and CRL, as well as PPO and SAC+HER using the same prior. ASE-CRL shows substantial gains on the larger mazes and open-arena pushing, although maze pushing remains difficult. The appendix includes controls for network depth, observations, control frequency, and evaluation time. The evaluation uses fixed starts and the training goal sets, so the results mainly establish performance within these task settings.
Critical strengths
-
The results on the larger mazes are convincing. On Giant, ASE-CRL reaches 97.4% success, compared with 82.1% for SAC+HER and 37.3% for PPO using the same prior. This is a useful practical result, and the five-seed evaluation gives it more weight than a few successful examples would.
-
I appreciate the additional controls and the benchmark contribution. The authors check several fairly obvious alternative explanations, including SAC depth, proprioceptive observations, and decision frequency. Together with the task suite and supplied implementation, this makes the paper useful even if the algorithm itself is mostly a combination of existing ideas.
Critical weaknesses
-
I am not sure there is much new in the learning algorithm itself. It mostly seems to be deep CRL on top of ASE. The comparisons could also be stronger. I appreciate the recent deep-CRL and JaxGCRL configurations, and the deeper SAC control, but increasing SAC's depth is not quite the same as testing a stronger recent reward-driven method, such as an appropriately adapted SimBaV2. The results show that this combination works well against the chosen baselines; its position relative to recent alternatives is less clear.
-
Some explanations seem stronger than the experiments support. CRL and SAC+HER differ in future-goal sampling and relabeling as well as the critic objective, so I am not sure the gap can be attributed to the objective alone. Also, the policies are trained and evaluated on the same goal sets from fixed starts. That is still a useful evaluation, but it leaves open how much of the result carries over to new goals or starts.
Questions for authors
-
Beyond combining deep CRL with ASE, what is the main methodological contribution? Is the intended contribution mainly the empirical finding and benchmark?
-
How does the method compare with a stronger recent reward-driven learner, for example an appropriately adapted SimBaV2, using the same frozen prior? The deeper SAC control is useful, but does not fully answer this.
-
How much do future-goal sampling and the relabeling ratio affect SAC+HER? Can the claimed benefit of the contrastive objective be separated from these choices?
-
Is there any evaluation on held-out goals or different starts? If not, could the claims about goal-reaching generalization be limited to what the current protocol tests?
3 comments
Thanks for the extra experiments. I am still a bit unsure about the “unseen starts” claim. If training now samples all free cells, aren’t those evaluation starts already covered? The 98.2% seems to show that broader coverage helps, rather than resolving the generalization concern.
The SimBaV2 result helps too, though running it well below its default update ratio leaves me wondering how much the comparison depends on that choice.
I still lean weak accept, but will lower my confidence from 2 to 1, as I am less certain how strongly these experiments support the revised claims.
Response to Reviewer 9QEM
Thank you for the follow-up. Both points are fair, and we answer them directly.
If training now samples all free cells, aren’t those evaluation starts already covered? The 98.2% seems to show that broader coverage helps, rather than resolving the generalization concern.
Yes. With randomized starts every free cell is a training start, so those starts are no longer unseen, and the 98.2% is not evidence of generalization to unseen starts. What it shows is narrower: the off-route drop of the fixed-start runs is a coverage effect and not a capacity limit of the contrastive learner or of direct destination-to-skill selection, since the same learner reaches all start-goal pairs once its data covers them.
The evidence on genuinely unseen starts is the fixed-start evaluation, and there the answer is mixed. New starts on a trained route are reached at close to the trained rate (ASE-CRL: 96.1 against 99.7 on Hard, 92.9 against 97.0 on Giant), while starts off the trained routes roughly halve for every method (ASE-CRL 53.1 on Hard and 42.8 on Giant; ASE-SAC+HER 46.1 and 34.5). The off-route drop is one of training coverage, and zero-shot generalization to starts off the trained routes remains limited for all methods.
Generalizing well to unseen starts, and to unseen mazes, would require a different high-level policy: one that receives the goal relative to the character, carries a recurrent memory or an explicit map as context, and is trained on a variety of environments. This goes beyond the scope of our paper, and we leave it to future work. Our focus, as in prior contrastive RL work, is reliability over long horizons in a given layout, here in combination with ASE; generalization across starts and layouts is a natural next step.
The SimBaV2 result helps too, though running it well below its default update ratio leaves me wondering how much the comparison depends on that choice.
We matched the update-to-data ratio across all high-level arms, ASE-CRL included, so that each receives the same data and the same number of gradient steps. Our result should therefore be read as: at matched data and gradient budget, the stronger architecture does not close the gap (83.6 against 97.4). Whether SimBaV2 closes it at its own default ratio remains open.
Neither point affects our central claim, which concerns the start the policy trains on: there, ASE-CRL stays above 92% on every route length on Giant, while ASE-SAC+HER and ASE-SimBaV2+HER fall to 76.4% and 53.9% on the longest routes.
We appreciate your continued engagement.
Response to Reviewer 9QEM
Thank you for the balanced assessment.
Novelty of the algorithm. We agree, and this is deliberate: ASE-CRL modifies neither the ASE prior nor the contrastive learner, so that any effect is attributable to their pairing and not to an added ingredient. We do not claim a new learning algorithm; what we claim instead is answered under Q1.
Q1. Methodological contribution. The contribution is the finding in G1 (in our general response), and the combination is built to isolate it: modifying neither component makes the effect attributable to the pairing rather than to a new ingredient. The paper now states the finding as the primary contribution, with the benchmark that lets others test it against further objectives.
Q2. A stronger recent reward-driven learner. We agree that deepening SAC is not the same as testing a stronger learner, and ran SimBaV2 over the same frozen prior with ASE-SAC+HER's sparse reward and relabeling, at the same data and gradient budget. On Giant it reaches 83.6 ± 5.8, against 82.1 ± 5.6 for ASE-SAC+HER and 97.4 ± 0.5 for ASE-CRL, and 53.9 against 92.1 on the longest routes (details under VQPH Q8). It is now part of Table 1.
Attributing the gap. We agree that the submission's wording went further than its experiments. The two arms differed in critic, in hindsight-goal sampling and relabeling ratio, and in depth. We have now controlled relabeling (Q3) in addition to depth, and both together, which leaves the critic, and the wording is corrected in the introduction, the results section and the conclusion (G2).
Q3. Future-goal sampling and relabeling ratio. We ran this control. ASE-SAC+HER with geometric future sampling and every transition relabeled, matching ASE-CRL and changing nothing else, reaches 76.8 ± 4.3 on Giant (default relabeling: 82.1 ± 5.6; ASE-CRL: 97.4 ± 0.5). The relabeling scheme does not explain the gap: given to the TD learner it lowers success, if anything. Combined with depth 32, so that only the critic differs from ASE-CRL, it reaches 55.3 ± 7.7 and is still slowly improving at the end of the budget. With depth also ruled out, this points to the critic (G2).
Q4. Held-out goals or different starts. See G3. Unseen goal positions are solved at the trained rate, and on pushing ASE-CRL reaches 86.7% on goals beyond the trained range. New maze starts are solved at close to the trained rate on trained routes and at about half off them, for all methods. Because hindsight relabeling covers every visited position, the off-centre test measures interpolation, so we also withheld whole goal cells. ASE-CRL trained on Hard with 9 of the 45 goal cells never commanded keeps 99.3% on the trained cells, reaches 94.4 ± 3.9 on withheld corridor cells (which relabeling still covers on the way to other goals) and 73.3 ± 25.6 on withheld dead ends, against 100% for runs trained on every cell. A goal never commanded and never passed is therefore reached most of the time but not reliably, and we state this as a limit. On starts, ASE-CRL trained with randomized start cells reaches 98.2% from new starts off the trained routes (53.1% with the fixed start; G3), and ASE-SAC+HER trained the same way 89.2% (46.1%), so the drop off-route belongs to the fixed-start protocol.
Maze-embedded pushing. The new breakdown shows where it fails: success falls with the number of turns on the block's route (ASE-CRL 55.3 ± 26.3 on straight routes, 18.8 ± 15.2 with two or more turns), since a block pushed past a corner cannot be pulled back.