acceptodds
Under review as a conference paper at ICLR 2027

Bidirectional Voronoi-biased Exploration Curriculum for Reinforcement Learning

Abstract

Long-horizon tasks with sparse rewards pose an exploration bottleneck for goal-conditioned reinforcement learning: a policy started from the initial state rarely reaches the goal and receives no learning signal. Reference motions, hand-designed curricula, and shaped rewards supply this signal but require demonstrations or task-specific engineering; automatic start-state and goal curricula avoid this but typically expand from one side only, so the full distance to the target must be covered from that side. We propose the Bidirectional Voronoi-biased Exploration curriculum for Reinforcement learning (BVER), which expands from both ends at once. Inspired by bidirectional RRT planning, BVER grows start states outward from the goal and goals outward from the initial state distribution, biases both toward unexplored task space, and steers them toward each other, training one goal-conditioned policy on both. On point-mass mazes, quadrupedal box climbing, and robot-arm ring-on-peg transfer, BVER learns faster than all compared reference-free curricula. On box climbing, it reaches 95% success on a 0.4 m box in roughly 65% fewer iterations than the best of them, is the only one of them to learn to climb a 0.7 m box, and yields a policy robust to start, goal, and yaw variation. Without a demonstration, it approaches the sample efficiency of reference-based curricula on the 0.4 m box and on ring-on-peg transfer. Ablations show that expanding from both ends outperforms either direction alone.

Primary area:
reinforcement learning
Keywords:
curriculum learning, reinforcement learning, goal-conditioned reinforcement learning, sparse rewards, exploration, legged locomotion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rLEBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BVER is an automatic start/goal curriculum for sparse-reward, goal-conditioned RL in resettable simulators. Like Reverse Curriculum (RC), it grows intermediate start states by random walks from the goal. Symmetrically, it grows intermediate goals by random walks from the initial-state distribution. In both directions it keeps states whose return falls in an intermediate band, and it trains one shared PPO policy on both kinds of tasks. Random-walk seeds are chosen with an RRT-style Voronoi bias. Experiments in Isaac Lab cover point-mass mazes, ANYmal box climbing, Franka ring-on-peg transfer and scanned terrains, with 5 seeds. BVER learns faster than vanilla PPO, a random curriculum and RC; it is the only reference-free method that solves the 0.7 m box; and it is competitive with demonstration-based RSI/Backplay. Ablations show that bidirectional expansion beats either direction alone.

Critical strengths
  1. A simple, intuitive idea with informative ablations. Which single direction works depends on the task (backwards for climbing up, forwards for climbing down). Bidirectional expansion removes the need to choose in advance, which is practically useful.

  2. Results on non-trivial robotics tasks (0.7 m box climbing, scanned terrains) learned from purely sparse rewards without demonstrations, with hyperparameters reused across terrains.

Critical weaknesses
  1. The baselines are too narrow. The only reference-free curricula compared are RC and a random curriculum. The paper discusses goal-curriculum methods (Goal-GAN, OUTPACE, Skew-Fit/MEGA, DISCOVER) and Go-Explore-style methods but does not compare against any of them. Most importantly, it does not compare against IBC (Kim et al., ICML 2023), a demonstration-free bidirectional curriculum. Although IBC targets the reset-free setting, it could be run with resets as a baseline, and because it needs weaker assumptions it is a natural point of comparison. As a result, the claim that BVER "learns faster than all reference-free curricula" is not well supported.

  2. Strong assumptions with limited testing. The method needs resets to arbitrary simulator states, a known goal state, and a known task-space box with a Euclidean metric. All experiments use 2–3D position goals, with no orientation, object-pose or higher-dimensional goals where Voronoi bias via uniform sampling would be harder. Hyperparameters are tuned per domain, and it is unclear whether the baselines were tuned equally. The random-walk simulation (about 28% extra wall-clock time) is not counted in the x-axis.

Questions for authors
  1. Can you add goal-curriculum baselines (e.g., OUTPACE or DISCOVER) and IBC?
  2. How is "success" defined and evaluated in Figs. 4/8/9?
  3. Can you plot learning curves against total simulated steps, including the random walks?
  4. Does BVER still work with higher-dimensional or non-positional goal spaces?
3 comments

Response to Reviewer rLEB

We sincerely thank the reviewer for the detailed and constructive review. We added new experiments and clarifications; all references below are to the revised PDF.


Question 1: Can you add goal-curriculum baselines (e.g., OUTPACE or DISCOVER) and IBC?

A: Thank you for this suggestion, and for highlighting IBC, which is closely related to our work.

  • New baseline. We added GoalGAN on the three point-mass mazes, ring-on-peg transfer, and climb box up (0.4 m and 0.7 m)(Figure 4, Section 5.1). After 600 iterations it reaches about 55% success in the U-maze and about 50% in the large maze, no sustained success in the N-maze or on ring-on-peg transfer and 95%/0% on the 0.4/0.7 m box climbing task. BVER reaches above 95% success on 6 four tasks (Figure 4a–d).
  • Why GoalGAN. In our paper, we focus on on-policy learning with PPO, which is widely used in robotic learning in complex environments like quadrupedal locomotion and robotic arm manipulation. Therefore, we chose GoalGAN as a goal-curriculum baseline because we found it to be the closest match in this setting. The mentioned works (OUTPACE, MEGA, DISCOVER, and IBC) are very interesting but all use off-policy RL algorithms like SAC. Our assumptions are not specific to on-policy RL and BVER could also be evaluated with off-policy RL. However, we limit our evaluation to on-policy for simplicity and resource constraints and to isolate the performance of the curricula instead of the RL algorithm.
  • Scope of the claim. "Learns fastest" refers to the curricula we compare against, as worded in the abstract ("all compared reference-free curricula").

Question 2: How is "success" defined and evaluated in Figs. 4/8/9?

A: We thank the reviewer for pointing out this ambiguity. We define success as a positive episodic reward on the sparse term. We evaluate N=5N=5 policies, trained with different seeds, on M=512M=512 random initializations of the target task, average over the MM rollouts, and then aggregate over the NN policies (shaded: standard error). This applies to Figures 4, 10, and 11 of the revision (4, 8, and 9 of the submission).


Question 3: Can you plot learning curves against total simulated steps, including the random walks?

A: Thank you for raising this; the submission was unclear here.

  • Iteration axis. All methods use the same total number of parallel environments and simulation steps per iteration; BVER and RC spend part of them on random walks, so the iteration axis already includes these steps (clarified in Section 5).
  • Wall-clock time. Figure 13 (Appendix A.3) shows the runs of Figure 4 against wall-clock time. Only two things change: the gap to the random curriculum shrinks in the U-maze and the large maze, and on ring-on-peg transfer the reference-based RSI becomes faster than BVER.
  • Corrected overhead. We found that the 28% was computed incorrectly, and we apologize for this error. Table 4 (Appendix A.4) replaces it: the curriculum computations take 5–20% of the time per iteration on the legged tasks and up to 41% on the mazes.

Question 4: Does BVER still work with higher-dimensional or non-positional goal spaces?

A: Yes. In the new Section 5.5 (details in Appendix A.2), we extend the 3-D ring position of ring-on-peg transfer with the ring orientation as a rotation matrix (12-D), with the non-positional arm joint offsets (10-D), or with both (19-D). The Euclidean distance between rotation matrices is monotonic in the geodesic angle, so the nearest-neighbor search stays consistent. Every variant solves the task within ten minutes of training (Figure 8) and the computation times differ by 3-17%; the ring position alone remains the most efficient choice.


Weakness 2: "The method needs resets to arbitrary simulator states, a known goal state, and a known task-space box with a Euclidean metric. [...] Hyperparameters are tuned per domain, and it is unclear whether the baselines were tuned equally."

A:

  • Assumptions. We agree, and we state them in the limitations (Section 6): resets to arbitrary simulator states, random-action rollouts, and a bounded task space with a meaningful distance metric. The new experiment in Question 4 tests the last one beyond low-dimensional positions; learned distances and learned dynamics models could relax them further, which we leave to future work.
  • Tuning. All baselines and BVER use the same PPO hyperparameters and the same environment and step budget (Appendix A.2), tuned for the specific task but not in favor of any specific curriculum algorithm. The hyperparameters for all baselines and BVER were selected in the same way, empirically per domain in preliminary runs. pconnectp_{\text{connect}} and the number of subsamples per walk are shared across all tasks (Table 3).

Thank you again for the valuable feedback. If anything needs further clarification, please do not hesitate to let us know.

Thank you for the response, which clarified most of my concerns. Please also include the hyperparameters of RC and GoalGAN in the appendix for reproducibility. Thanks.

Thank you for your follow up question and for pointing out the missing hyperparameters of RC and GoalGAN. We added them to the Table 4 in the appendix.

Reviewer_JAY3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BVER (Bidirectional Voronoi-Biased Exploration Curriculum for Reinforcement Learning) is a curriculum-learning framework inspired by bidirectional RRT. It expands intermediate states from both the initial and goal states and trains a single policy on tasks sampled from both directions. Voronoi-bias proportional to unexplored volume in task space is introduced to favor less-explored regions, along with a connection step that links the two expansions. The authors evaluate the method in point-mass mazes, quadruped climbing and maneuvering tasks over scanned terrains, and a ring-on-peg transfer task with a Franka manipulator. They report that BVER reaches higher success earlier than vanilla PPO and curriculum-learning baselines. Ablations suggest that both expansion directions and the connection step contribute to performance.

Critical strengths
  1. Well-justified curriculum inspired by bidirectional RRT, combining Voronoi-biased exploration with intermediate state generation from both the start and goal sides. Comparison with curriculum-learning baselines in Figure 4 demonstrate improved sample efficiency and robustness, supporting the effectiveness of the proposed approach. Ablation results in Table 1 highlight the contributions of bidirectional expansion and connection mechanism, providing empirical support for these core components of the method.

  2. Simulation results span across diverse task settings including point-mass mazes, quadruped climbing and locomotion over scanned terrains, and ring-on-peg manipulation with a Franka arm. These tasks involve different embodiments, goal conditions, and exploration challenges, providing a broad assessment of the proposed curriculum. Quadruped climbing with directionally asymmetric dynamics serve as an informative setting for evaluating bidirectional expansion, with ablations highlighting the contribution of the two expansion directions under asymmetry.

Critical weaknesses
  1. Incomplete clarification of interaction and computational costs from random-action rollouts. The learning curves in Figure 4 demonstrate faster learning and higher success rates at comparable training iterations. However, BVER also uses random-action rollouts to generate curriculum candidates. Although the conclusion reports an approximately 28% increase in wall-clock training time, the experimental results do not clearly indicate whether the stated simulator-step budget includes these exploration rollouts or how their computational overhead affects comparisons with the baselines. Clarifying this accounting and reporting success rates against total simulator interactions or wall-clock training time would strengthen the efficiency claims and help assess whether the gains observed per training iteration translate into practical training-time improvements.

  2. Limited justification and sensitivity analysis for the connection probability p_connect. The ablation evaluates 3 values (0,0.5,1) in equation 6 to support the benefit of combining coverage-oriented exploration with connections between the two expansion directions. Evaluating additional intermediate values would help determine whether the reported gains persist across a broader range of settings. This would be particularly informative across tasks with asymmetric dynamics such as quadruped climbing with gravity. Clarifying how the default value 0.5 was selected and whether different values were tested across tasks would further strengthen the evaluation.

Questions for authors

A. Questions that could materially affect the assessment

  1. (Weakness 1) How does overhead from random rollouts vary across tasks and stages of training?
  2. (Weakness 1) Could you provide learning curves against total simulator interactions or wall-clock time to clarify whether BVER’s efficiency gains over the baselines persist when curriculum-generation costs are included?
  3. (Weakness 2) How sensitive is proposed method’s performance to other intermediate values of p_connect?
  4. (Weakness 2) Was proposed default value of p_connect the optimal value across all simulation environments, especially quadruped box climbing?

B. Clarification requests

  1. How was the approximately 28% increase in wall-clock training time measured?
  2. How was default p_connect value (0.5) selected?
2 comments

Response to Reviewer JAY3

We sincerely thank the reviewer for the thorough review and the precise questions, which led us to correct an error and to add a parameter study. All references below are to the revised PDF.


Weakness 1: "[...] the experimental results do not clearly indicate whether the stated simulator-step budget includes these exploration rollouts or how their computational overhead affects comparisons with the baselines."

A: The budget includes them, and we apologize that this was unclear. All methods use the same total number of parallel environments and simulation steps per iteration; BVER and RC spend part of them on random walks, while the other methods use all of them for policy rollouts. The iteration axis in Figure 4 therefore already includes the random-walk interactions (clarified in Section 5). The computational overhead is addressed in Questions A.1 and A.2 below.


Question A.1: (Weakness 1) How does overhead from random rollouts vary across tasks and stages of training?

A: The new Table 4 (Appendix A.4) breaks down the wall-clock time per iteration. The BVER-specific computations (random-walk collection, nearest-neighbor search, and the remaining curriculum computations) take the following share:

PM-UPM-NPM-LRPTCUFB IFB IIRASSF
BVER-specific share22%32%41%34%20%10%10%5%9%
Random-walk overhead4.0%5.8%5.9%17.3%6.6%3.5%3.4%1.8%3.1%

Note that the random-walk overhead is part of the BVER-specific share. The share is the same during all stages of training, since the number of random-walk environments, the walk horizon, and the buffer sizes are fixed (Table 3).


Question A.2: (Weakness 1) Could you provide learning curves against total simulator interactions or wall-clock time to clarify whether BVER’s efficiency gains over the baselines persist when curriculum-generation costs are included?

A: Figure 13 (Appendix A.3) shows the runs of Figure 4 against wall-clock time, which includes all curriculum-generation costs. BVER's gains over the reference-free baselines persist, and only two things change: the gap to the random curriculum shrinks in the U-maze and the large maze, and on ring-on-peg transfer the reference-based RSI becomes faster than BVER.


Question A.3: (Weakness 2) How sensitive is proposed method’s performance to other intermediate values of p_connect?

A: We added a parameter study on ring-on-peg transfer (Figure 7, Section 5.4):

pconnectp_{\text{connect}}00.20.40.50.60.81
Iterations to 75% success150150120120120150not reached

The performance is robust except for purely greedy connecting. In the same study on the large maze (Figure 12c), all seven values reach 75% success within 160 to 240 iterations.


Question A.4: (Weakness 2) Was proposed default value of p_connect the optimal value across all simulation environments, especially quadruped box climbing?

A: We do not claim that 0.5 is optimal everywhere: pconnectp_{\text{connect}} was not tuned per task and is shared across all tasks (Table 3). On the two tasks where we swept it, 0.5 is among the best values: it ties with 0.4 and 0.6 on ring-on-peg transfer (Figure 7) and with 0.6 in the large maze (Figure 12c).


Question B.1: How was the approximately 28% increase in wall-clock training time measured?

A: Thank you for asking. On re-examination we found that the 28% was computed incorrectly, and we apologize for this error. We replaced it with the breakdown in Table 4 and updated the conclusion: BVER only adds a computational overhead of 20% of the wall-clock time per iteration (326 ms), including 6% (112 ms) for the random-walk collection averaged on all tasks.


Question B.2: How was default p_connect value (0.5) selected?

A: Since the performance is robust except for extreme values, we chose 0.5 for simplicity; it weighs coverage and connection equally in Equation 6.


Thank you again for the careful reading. If anything needs further clarification, please do not hesitate to let us know.

Thank you for your response. Additional experiment results and table clarified my worries regarding computational cost and overheads. Your answer on parameter p_connect also resolved my questions on parameter selection.

Reviewer_c9YzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies goal-conditioned reinforcement learning with sparse rewards and proposes BVER, a bidirectional curriculum method. BVER generates intermediate start states by expanding outward from the target goal and intermediate goals by expanding outward from the initial state distribution. A single policy is trained on both types of tasks. Expansion uses Voronoi-biased seed selection for random walks, together with samples from the opposite direction’s solved buffer to guide the two frontiers toward each other. The experiments cover point-mass mazes, quadrupedal box climbing, and robotic ring-on-peg transfer. The authors compare BVER with training without a curriculum, a random curriculum, reverse curriculum generation, and two demonstration-based methods. Ablations examine the expansion directions and connection ratio. The paper also evaluates the reuse of the main curriculum hyperparameters from box climbing on four scanned terrains.

Critical strengths

1.The expansion-direction ablation is the strongest part of the evaluation. The unidirectional variants in Table 1 receive the full policy-rollout budget, so the advantage of bidirectional training cannot simply be attributed to additional training samples. Ring-on-peg transfer provides a clear example: neither unidirectional variant reaches a mean success rate of 75%, whereas the bidirectional method reaches it in 120 iterations. The climb-up and climb-down experiments also show that the two directions have different strengths. These results say more about the value of bidirectional training than a comparison against vanilla PPO alone would. 2.The results extend beyond simple navigation problems. BVER also improves learning in quadrupedal locomotion and arm manipulation, with the main learning curves reported over five random seeds. The scanned-terrain experiments provide further evidence that the curriculum is useful beyond simple geometric environments: reusing the main hyperparameters from box climbing yields final success rates of approximately 60%–91% across four terrains. The task space, episode length, and number of environments are adjusted, but this is still a useful demonstration of applicability.

Critical weaknesses

1.The demonstration-free baseline set is limited. RC is an important comparison, but the paper also emphasizes intermediate-goal generation. Comparing primarily against RC and a random curriculum leaves open how much this part of the method improves upon existing goal curricula. GoalGAN, OUTPACE, and DISCOVER are discussed but not evaluated. The authors’ forward-only variant helps establish whether combining directions is useful, but it does not replace a comparison with existing methods. The scanned-terrain experiments compare only against vanilla PPO and therefore do not resolve this concern. 2.Voronoi bias is a central part of the title and method, yet its contribution is not directly ablated. The “no connect” variant in Table 1 still retains Voronoi-biased seed selection through uniform task-space sampling and nearest-neighbor lookup. That experiment tests the connection mechanism rather than the benefit of Voronoi-biased selection over uniform sampling from the buffer. This distinction matters. On the 0.7 m climb-up task, backward-only BVER and bidirectional BVER both reach 75% success in 3,800 iterations, while RC fails to learn the task. The improvement on this task therefore cannot be attributed entirely to bidirectional training; the effects of the other design changes need to be separated.

Questions for authors

1.Could you include a suitable goal-curriculum baseline on at least one maze and one robotic task? If these methods are difficult to apply in your setting, please explain the specific difficulties. 2.What happens if Voronoi-biased seed selection is replaced with uniform sampling from the buffer, while keeping bidirectional training, the connection mechanism, buffers, and training budget unchanged? Does the connection mechanism also improve performance on the robotic tasks?

1 comment

Response to Reviewer c9Yz

We sincerely thank the reviewer for the careful and constructive review. We ran both requested experiments; all references below are to the revised PDF.


Question 1: Could you include a suitable goal-curriculum baseline on at least one maze and one robotic task? If these methods are difficult to apply in your setting, please explain the specific difficulties.

A: Thank you for this suggestion. We agree that RC and a random curriculum alone left this open.

  • New baseline. We added GoalGAN on the three point-mass mazes, ring-on-peg transfer, and climb box up (0.4 m and 0.7 m) (Figure 4, Section 5.1). After 600 iterations it reaches about 55% success in the U-maze and about 50% in the large maze, no sustained success in the N-maze or on ring-on-peg transfer, and 95%/0% on the 0.4/0.7 m climb box up task. BVER reaches above 95% success on 6 four tasks (Figure 4a–d).
  • Why GoalGAN. In our paper, we focus on on-policy learning with PPO, which is widely used in robotic learning in complex environments like quadrupedal locomotion and robotic arm manipulation. Therefore, we chose GoalGAN as a goal-curriculum baseline because we found it to be the closest match in this setting. The mentioned works (OUTPACE, MEGA, DISCOVER, and IBC) are very interesting but all use off-policy RL algorithms like SAC. Our assumptions are not specific to on-policy RL and BVER could also be evaluated with off-policy RL. However, we limit our evaluation to on-policy for simplicity and resource constraints and to isolate the performance of the curricula instead of the RL algorithm.
  • Scanned terrains. We agree with the reviewer that the scanned terrain experiments do not resolve the concern. We designed them to test whether the box-climbing hyperparameters transfer to more complex geometry.

Question 2: What happens if Voronoi-biased seed selection is replaced with uniform sampling from the buffer, while keeping bidirectional training, the connection mechanism, buffers, and training budget unchanged? Does the connection mechanism also improve performance on the robotic tasks?

A: Thanks to the reviewer for pointing out this gap. The reviewer is right that "no connect" only tested the connection mechanism.

  • Voronoi ablation. The new "no Voronoi" variant is exactly as you propose: seeds are drawn uniformly from the buffer, while bidirectional training, the connection mechanism, the buffers, and the budget are unchanged (Table 1, Section 5.4, Figure 12a).

    Iterations to 75% successPM-UPM-NPM-LRPT
    BVER140800140120
    BVER, no Voronoi440not reached180330

    On all tested tasks, drawing uniformly from the buffer reduces sample efficiency. Specifically, on ring-on-peg transfer the uniform variant needs 2.8 times more iterations, and in the N-maze it does not reach 75% success.

  • Connection on a robotic task. We extended our connection ablation to ring-on-peg transfer, where BVER needs 150 instead of 120 iterations without connecting, and greedy connecting does not reach 75% success (Table 1). Further, we conducted a parameter study for pconnectp_{\text{connect}} on this task (Figure 7). We observed similar behavior in the large maze (Figure 12c).

  • 0.7 m climb-up. We agree: backward-only and bidirectional BVER both need 3800 iterations while RC does not learn the task, so this gain is not due to bidirectional training. The backward-only extends RC with Voronoi-biased coverage (Section 4.1), which the new ablation tests directly. The benefit of combining both directions shows across tasks: each direction alone fails on two of the eight tasks in Table 1, and which one works depends on the task.


Thank you again for the valuable feedback. If anything needs further clarification, please do not hesitate to let us know.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.