- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors proposes ReQRL for offline goal-conditioned reinforcement learning. The main idea is to use the states that a robot can reach after several actions, together with information about obstacle boundaries, to train a better goal-distance value function. The method estimates these constraints from offline data and uses the learned values to train goal-reaching policies. The proposed method exhibits superior performance across several OGBench navigation tasks.
Critical strengths
- The main idea is meaningful. A useful goal-distance function should reflect how the robot can actually move, rather than only how far the goal is on a map.
- The evaluation covers point, ant, humanoid, and car navigation tasks. ReQRL shows strong success rates against several existing GCRL methods.
Critical weaknesses
- The motivation of ReQRL for offline RL is not sufficiently clear. The paper mostly starts from limitations of earlier quasimetric methods. But limited motion and obstacles are also problems in online RL. What specific difficulty of offline learning does ReQRL solve?
- The paper’s section structure is unclear; the core method is scattered across different sections, hindering readers from understanding it as a complete algorithm.
- The experiments tell us that ReQRL works, but not clearly why it works. The paper has value-geometry figures and learning curves, yet it does not show enough real paths, value heatmaps, or decisions near walls. More importantly, there is no clear ablation study separating multi-step reachability, boundary regularization, horizon k, and policy design.
Questions for authors
- What offline-GCRL problem is ReQRL designed to solve? Would the same idea also help if the agent could collect new online data?
- Can you show PointMaze trajectories and value heatmaps, especially near obstacles, to explain how ReQRL changes the policy's behavior?
- How close are the affine-model reachable sets to the robot's actual multi-step reachable sets? How does this change with the horizon k?
- Can you provide controlled ablations for multi-step reachability, boundary regularization, horizon k, and the high-/low-level policy setup?
5 comments
Q3-4. Can you provide controlled ablations for multi-step reachability and horizon k? How close are the affine-model reachable sets to the robot's actual multi-step reachable sets? How does this change with the horizon k?
First, we ablate the finite horizon , which controls the number of steps over which we estimate reachable motions, with . As we discuss in our response to Rev. bkEK’s Q1, smaller favors a local first-order approximation, while longer horizons allow control effects to accumulate. In ReQRL, we use the heuristic rule to choose (Sec. 5 and App. C.2). Here, we assess how goal-reaching performance changes with this choice by varying the reachability and waypoint horizons together, with the same pipeline fixed across . New runs use three seeds (ReQRL results in the paper are starred).
| antmaze-nav | * | ||
|---|---|---|---|
| large | 95 ± 1 | 95 ± 1 | 94 ± 3 |
| giant | 72 ± 5 | 77 ± 6 | 74 ± 5 |
| humanoid-nav | * | ||
|---|---|---|---|
| large | 81 ± 8 | 82 ± 7 | 82 ± 6 |
| giant | 73 ± 1 | 86 ± 8 | 81 ± 5 |
We observe that ReQRL remains effective across a range of horizons, so it does not depend strongly on precise choice of . This choice still matters though, particularly for longer goal-reaching trajectories. This is expected, as longer routes require more waypoint decisions, so the effects of on reachability estimates and waypoint choices can accumulate over trajectories.
To isolate the contribution of multistep reachability, we also replaced the reachability estimate at with a one-step estimate on antmaze-large while keeping the actor’s waypoint horizon fixed to see the effect of multi-step reachability in critic learning. This resulted in success decrease from 95 to 75 over three seeds. This shows that multi-step reachability helps goal reaching even when the actor’s waypoint horizon stays the same.
Separately, we analyzed reachability estimates using directed action-sequence searches at on antmaze. For coverage, our full estimated set in Eq. 14 contains all net state changes produced by the tested -step action sequences. On held-out antmaze trajectories, the propagated center in Eq. 14 predicts observed state changes with of 0.63, 0.54, and 0.47, respectively. Together with the ablations above, these results support that this approximation is practically useful. More faithful reachability estimation with efficient propagation and evaluation remains important but complementary direction for future work.
Q4. Can you provide controlled ablations for boundary regularization and the high-/low-level policy setup?
Yes, we checked the contributions of the policy architecture and boundary regularization separately through two additional ablations. First, we ablate how the high-/low-level policies receive value supervision. We replace the original ReQRL’s reduced-state high-level critic and separate full-state low-level TD value function with a single full-state ReQRL critic shared by both actors. This variant uses full-state reachability while retaining the spatial boundary penalty. This directly tests how ReQRL critic contributes to policy learning without additional TD value, which also addresses Reviewer SKza’s Q1.
For reference, we include Eik-QRL and HJB-QRL (the two offical variants from the EikHiQRL paper), which use Eikonal and Hamilton–Jacobi–Bellman critic constraints, respectively, and also use the hierarchical actors. Both the variant and ReQRL use models trained from scratch across three seeds, with the same training budget and critic-penalty weights. Eik-QRL/HJB-QRL results come from Table 1 of the Eik-HiQRL paper.
| antmaze-nav | Variant | ReQRL | Eik-QRL | HJB-QRL |
|---|---|---|---|---|
| medium | 92 | 94 | 84 | 31 |
| large | 88 | 95 | 74 | 28 |
| giant | 42 | 77 | 8 | 0 |
We observe that the variant outperforms both baselines across all layouts, showing that ReQRL’s critic supervision and resulting values remain effective when shared by both policy levels. Standard ReQRL performs better still, supporting the practical benefit of our value separation, as discussed in our response to Rev. bkEK’s Q2. We would be happy to consider another targeted ablation if you had a different aspect of the setup in mind.
Second, we ablate whether boundary regularization improves goal reaching and how performance depends on its strength. For the new runs, we vary only the boundary-penalty weight while keeping the remaining setup fixed, and report mean success (%) over three seeds. Starred results are our ReQRL results in the paper.
| antmaze-giant-nav | * | EikHiQRL | |||
|---|---|---|---|---|---|
| Success (%) | 64 ± 3 | 73 ± 8 | 77 ± 6 | 77 ± 7 | 56 ± 9 |
| humanoid-giant-nav | * | EikHiQRL | |||
|---|---|---|---|---|---|
| Success (%) | 80 ± 6 | 83 ± 2 | 86 ± 8 | 83 ± 1 | 80 ± 5 |
In both antmaze and humanoidmaze, we see higher mean success with boundary regularization than without it, which shows the benefit of incorporating boundary information into critic learning. We also observe that increasing the weight further need not improve success. An excessive weight can favor reducing the boundary penalty at the expense of satisfying the interior Hamiltonian constraint and maximizing learned distances.
In our response to Rev. bkEK’s W2, we also discuss how this term regularizes the critic near the estimated edge of the data-supported region rather than imposing a hard restriction on the policy.
Thank you for insightful questions and feedback! We clarify each weakness and question together and provide new results below. The revision currently includes the new plots in Sec. 6.3 in red.
The main difficulty in offline GCRL is that the critic must infer goal-reaching costs from fixed offline data, without additional interaction to check if predicted progress/routes are feasible. When available data contains short or incomplete trajectories, this becomes especially challenging. ReQRL addresses this by estimating multi-step reachability and boundary geometry from the available offline data and using these to supervise the critic. When used with the quasimetric structure, these local constraints effectively inform global goal-reaching values as we see in our experiments in Sec. 6.
We agree though, that dynamical and feasibility constraints also matter in online settings. So our formulation is not restricted to offline learning. We focus on offline evaluation here because the inability to collect additional data makes the supervision particularly valuable. Online evaluation is a promising direction for future work, as we mention in App. A.
Thank you for the feedback. The separation across sections was based on our thinking to first develop the general formulation and then show how it can be instantiated from offline data. So, we organized Sec. 3–4 to present the general ReQRL formulation, followed by the practical instantiation in Sec. 5 used in our experiments in Sec. 6. To make this clearer, we will add an end-to-end overview and training pseudocode connecting these sections in our revision. This will show how reachability estimation, boundary estimation, critic training, and policy learning come together, with the general components and their Sec. 5 implementations identified in one place.
We agree that the new visuals will help explain why ReQRL improves goal-reaching performance, separately from success rates in Table 1. So, we additionally provide successful and failed trajectories (Figure 6), close-up trajectories with ReQRL’s critic gradient field (Figure 7), and distance heatmap comparisons against EikHiQRL (Figure 8) on antmaze. We choose antmaze to illustrate both navigation around obstacles and control behavior. These figures and their details are included in the revised draft in Sec. 6.3 in red.
In Fig. 6, ReQRL reaches goals more often, consistent with Table 1, and its successful trajectories are more tightly clustered. EikHiQRL shows greater variation, including failed trajectories near walls. In Fig. 7, the critic’s descent directions −∇dθ(s,g) follow the corridor around the turn, and they provide coherent local guidance toward the goal. Successful and failed rollouts follow similar routes, but the ant loses balance while turning close to the corner in the failed rollout. Hence, a coherent route guidance does not eliminate all execution failures, which we view as an important direction for future work. In Fig. 8, we observe smaller relative deviations from the shortest-path reference for ReQRL. We can see much closer agreement with the reference distances along routes around obstacles for ReQRL than EikHiQRL.
Overall, these results support that ReQRL’s reachability and boundary constraints help learn value geometry that better reflects routes around obstacles and supports more consistent goal-reaching behavior.