- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BVER is an automatic start/goal curriculum for sparse-reward, goal-conditioned RL in resettable simulators. Like Reverse Curriculum (RC), it grows intermediate start states by random walks from the goal. Symmetrically, it grows intermediate goals by random walks from the initial-state distribution. In both directions it keeps states whose return falls in an intermediate band, and it trains one shared PPO policy on both kinds of tasks. Random-walk seeds are chosen with an RRT-style Voronoi bias. Experiments in Isaac Lab cover point-mass mazes, ANYmal box climbing, Franka ring-on-peg transfer and scanned terrains, with 5 seeds. BVER learns faster than vanilla PPO, a random curriculum and RC; it is the only reference-free method that solves the 0.7 m box; and it is competitive with demonstration-based RSI/Backplay. Ablations show that bidirectional expansion beats either direction alone.
Critical strengths
-
A simple, intuitive idea with informative ablations. Which single direction works depends on the task (backwards for climbing up, forwards for climbing down). Bidirectional expansion removes the need to choose in advance, which is practically useful.
-
Results on non-trivial robotics tasks (0.7 m box climbing, scanned terrains) learned from purely sparse rewards without demonstrations, with hyperparameters reused across terrains.
Critical weaknesses
-
The baselines are too narrow. The only reference-free curricula compared are RC and a random curriculum. The paper discusses goal-curriculum methods (Goal-GAN, OUTPACE, Skew-Fit/MEGA, DISCOVER) and Go-Explore-style methods but does not compare against any of them. Most importantly, it does not compare against IBC (Kim et al., ICML 2023), a demonstration-free bidirectional curriculum. Although IBC targets the reset-free setting, it could be run with resets as a baseline, and because it needs weaker assumptions it is a natural point of comparison. As a result, the claim that BVER "learns faster than all reference-free curricula" is not well supported.
-
Strong assumptions with limited testing. The method needs resets to arbitrary simulator states, a known goal state, and a known task-space box with a Euclidean metric. All experiments use 2–3D position goals, with no orientation, object-pose or higher-dimensional goals where Voronoi bias via uniform sampling would be harder. Hyperparameters are tuned per domain, and it is unclear whether the baselines were tuned equally. The random-walk simulation (about 28% extra wall-clock time) is not counted in the x-axis.
Questions for authors
- Can you add goal-curriculum baselines (e.g., OUTPACE or DISCOVER) and IBC?
- How is "success" defined and evaluated in Figs. 4/8/9?
- Can you plot learning curves against total simulated steps, including the random walks?
- Does BVER still work with higher-dimensional or non-positional goal spaces?
3 comments
Thank you for the response, which clarified most of my concerns. Please also include the hyperparameters of RC and GoalGAN in the appendix for reproducibility. Thanks.
Thank you for your follow up question and for pointing out the missing hyperparameters of RC and GoalGAN. We added them to the Table 4 in the appendix.
Response to Reviewer rLEB
We sincerely thank the reviewer for the detailed and constructive review. We added new experiments and clarifications; all references below are to the revised PDF.
Question 1: Can you add goal-curriculum baselines (e.g., OUTPACE or DISCOVER) and IBC?
A: Thank you for this suggestion, and for highlighting IBC, which is closely related to our work.
Question 2: How is "success" defined and evaluated in Figs. 4/8/9?
A: We thank the reviewer for pointing out this ambiguity. We define success as a positive episodic reward on the sparse term. We evaluate N=5 policies, trained with different seeds, on M=512 random initializations of the target task, average over the M rollouts, and then aggregate over the N policies (shaded: standard error). This applies to Figures 4, 10, and 11 of the revision (4, 8, and 9 of the submission).
Question 3: Can you plot learning curves against total simulated steps, including the random walks?
A: Thank you for raising this; the submission was unclear here.
Question 4: Does BVER still work with higher-dimensional or non-positional goal spaces?
A: Yes. In the new Section 5.5 (details in Appendix A.2), we extend the 3-D ring position of ring-on-peg transfer with the ring orientation as a rotation matrix (12-D), with the non-positional arm joint offsets (10-D), or with both (19-D). The Euclidean distance between rotation matrices is monotonic in the geodesic angle, so the nearest-neighbor search stays consistent. Every variant solves the task within ten minutes of training (Figure 8) and the computation times differ by 3-17%; the ring position alone remains the most efficient choice.
Weakness 2: "The method needs resets to arbitrary simulator states, a known goal state, and a known task-space box with a Euclidean metric. [...] Hyperparameters are tuned per domain, and it is unclear whether the baselines were tuned equally."
A:
Thank you again for the valuable feedback. If anything needs further clarification, please do not hesitate to let us know.