acceptodds
Under review as a conference paper at ICLR 2027

Finite-Horizon Hamiltonians for Value Geometry in Goal-Conditioned RL

Abstract

In goal-conditioned reinforcement learning (GCRL), quasimetric learning models goal-reaching costs as quasimetric distances, connecting local constraints to global value geometry. Its local constraints, however, should reflect the direction-dependent effects of control composition over a finite horizon together with environmental feasibility. We propose *ReQRL*, which constrains the critic's value gradients through finite-horizon reachability. Drawing on state-constrained optimal control, we decouple dynamical reachability from boundary geometry, estimating both from data. On *OGBench*, our method outperforms or rivals existing quasimetric approaches and other offline GCRL methods.

Primary area:
reinforcement learning
Keywords:
Goal-conditioned RL, offline RL, quasimetric learning, local dynamics, Hamilton–Jacobi equations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gXAQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposes ReQRL for offline goal-conditioned reinforcement learning. The main idea is to use the states that a robot can reach after several actions, together with information about obstacle boundaries, to train a better goal-distance value function. The method estimates these constraints from offline data and uses the learned values to train goal-reaching policies. The proposed method exhibits superior performance across several OGBench navigation tasks.

Critical strengths
  1. The main idea is meaningful. A useful goal-distance function should reflect how the robot can actually move, rather than only how far the goal is on a map.
  2. The evaluation covers point, ant, humanoid, and car navigation tasks. ReQRL shows strong success rates against several existing GCRL methods.
Critical weaknesses
  1. The motivation of ReQRL for offline RL is not sufficiently clear. The paper mostly starts from limitations of earlier quasimetric methods. But limited motion and obstacles are also problems in online RL. What specific difficulty of offline learning does ReQRL solve?
  2. The paper’s section structure is unclear; the core method is scattered across different sections, hindering readers from understanding it as a complete algorithm.
  3. The experiments tell us that ReQRL works, but not clearly why it works. The paper has value-geometry figures and learning curves, yet it does not show enough real paths, value heatmaps, or decisions near walls. More importantly, there is no clear ablation study separating multi-step reachability, boundary regularization, horizon k, and policy design.
Questions for authors
  1. What offline-GCRL problem is ReQRL designed to solve? Would the same idea also help if the agent could collect new online data?
  2. Can you show PointMaze trajectories and value heatmaps, especially near obstacles, to explain how ReQRL changes the policy's behavior?
  3. How close are the affine-model reachable sets to the robot's actual multi-step reachable sets? How does this change with the horizon k?
  4. Can you provide controlled ablations for multi-step reachability, boundary regularization, horizon k, and the high-/low-level policy setup?
5 comments

Thank you for insightful questions and feedback! We clarify each weakness and question together and provide new results below. The revision currently includes the new plots in Sec. 6.3 in red.

W1. The motivation of ReQRL for offline RL is not sufficiently clear. The paper mostly starts from limitations of earlier quasimetric methods. But limited motion and obstacles are also problems in online RL. What specific difficulty of offline learning does ReQRL solve?

Q1. What offline-GCRL problem is ReQRL designed to solve? Would the same idea also help if the agent could collect new online data?

The main difficulty in offline GCRL is that the critic must infer goal-reaching costs from fixed offline data, without additional interaction to check if predicted progress/routes are feasible. When available data contains short or incomplete trajectories, this becomes especially challenging. ReQRL addresses this by estimating multi-step reachability and boundary geometry from the available offline data and using these to supervise the critic. When used with the quasimetric structure, these local constraints effectively inform global goal-reaching values as we see in our experiments in Sec. 6.

We agree though, that dynamical and feasibility constraints also matter in online settings. So our formulation is not restricted to offline learning. We focus on offline evaluation here because the inability to collect additional data makes the supervision particularly valuable. Online evaluation is a promising direction for future work, as we mention in App. A.

W2. The paper’s section structure is unclear; the core method is scattered across different sections, hindering readers from understanding it as a complete algorithm.

Thank you for the feedback. The separation across sections was based on our thinking to first develop the general formulation and then show how it can be instantiated from offline data. So, we organized Sec. 3–4 to present the general ReQRL formulation, followed by the practical instantiation in Sec. 5 used in our experiments in Sec. 6. To make this clearer, we will add an end-to-end overview and training pseudocode connecting these sections in our revision. This will show how reachability estimation, boundary estimation, critic training, and policy learning come together, with the general components and their Sec. 5 implementations identified in one place.

W3-1. The experiments tell us that ReQRL works, but not clearly why it works. The paper has value-geometry figures and learning curves, yet it does not show enough real paths, value heatmaps, or decisions near walls.

Q2. Can you show PointMaze trajectories and value heatmaps, especially near obstacles, to explain how ReQRL changes the policy's behavior?

We agree that the new visuals will help explain why ReQRL improves goal-reaching performance, separately from success rates in Table 1. So, we additionally provide successful and failed trajectories (Figure 6), close-up trajectories with ReQRL’s critic gradient field (Figure 7), and distance heatmap comparisons against EikHiQRL (Figure 8) on antmaze. We choose antmaze to illustrate both navigation around obstacles and control behavior. These figures and their details are included in the revised draft in Sec. 6.3 in red.

In Fig. 6, ReQRL reaches goals more often, consistent with Table 1, and its successful trajectories are more tightly clustered. EikHiQRL shows greater variation, including failed trajectories near walls. In Fig. 7, the critic’s descent directions −∇dθ(s,g)-\nabla d_\theta (s,g) follow the corridor around the turn, and they provide coherent local guidance toward the goal. Successful and failed rollouts follow similar routes, but the ant loses balance while turning close to the corner in the failed rollout. Hence, a coherent route guidance does not eliminate all execution failures, which we view as an important direction for future work. In Fig. 8, we observe smaller relative deviations from the shortest-path reference for ReQRL. We can see much closer agreement with the reference distances along routes around obstacles for ReQRL than EikHiQRL.

Overall, these results support that ReQRL’s reachability and boundary constraints help learn value geometry that better reflects routes around obstacles and supports more consistent goal-reaching behavior.

Q3-4. Can you provide controlled ablations for multi-step reachability and horizon k? How close are the affine-model reachable sets to the robot's actual multi-step reachable sets? How does this change with the horizon k?

First, we ablate the finite horizon kk, which controls the number of steps over which we estimate reachable motions, with τ=kΔt\tau=k\Delta t. As we discuss in our response to Rev. bkEK’s Q1, smaller τ\tau favors a local first-order approximation, while longer horizons allow control effects to accumulate. In ReQRL, we use the heuristic rule to choose kk (Sec. 5 and App. C.2). Here, we assess how goal-reaching performance changes with this choice by varying the reachability and waypoint horizons together, with the same pipeline fixed across kk. New runs use three seeds (ReQRL results in the paper are starred).

antmaze-navk=8k=81212*1616
large95 ± 195 ± 194 ± 3
giant72 ± 577 ± 674 ± 5
humanoid-navk=32k=324848*6464
large81 ± 882 ± 782 ± 6
giant73 ± 186 ± 881 ± 5

We observe that ReQRL remains effective across a range of horizons, so it does not depend strongly on precise choice of kk. This choice still matters though, particularly for longer goal-reaching trajectories. This is expected, as longer routes require more waypoint decisions, so the effects of kk on reachability estimates and waypoint choices can accumulate over trajectories.

To isolate the contribution of multistep reachability, we also replaced the reachability estimate at k=12k=12 with a one-step estimate on antmaze-large while keeping the actor’s waypoint horizon fixed to see the effect of multi-step reachability in critic learning. This resulted in success decrease from 95 to 75 over three seeds. This shows that multi-step reachability helps goal reaching even when the actor’s waypoint horizon stays the same.

Separately, we analyzed reachability estimates using directed action-sequence searches at k∈{8,12,16}k \in \lbrace 8,12,16\rbrace on antmaze. For coverage, our full estimated set in Eq. 14 contains all net state changes produced by the tested kk-step action sequences. On held-out antmaze trajectories, the propagated center in Eq. 14 predicts observed state changes with R2R^2 of 0.63, 0.54, and 0.47, respectively. Together with the ablations above, these results support that this approximation is practically useful. More faithful reachability estimation with efficient propagation and evaluation remains important but complementary direction for future work.

Q4. Can you provide controlled ablations for boundary regularization and the high-/low-level policy setup?

Yes, we checked the contributions of the policy architecture and boundary regularization separately through two additional ablations. First, we ablate how the high-/low-level policies receive value supervision. We replace the original ReQRL’s reduced-state high-level critic and separate full-state low-level TD value function with a single full-state ReQRL critic shared by both actors. This variant uses full-state reachability while retaining the spatial boundary penalty. This directly tests how ReQRL critic contributes to policy learning without additional TD value, which also addresses Reviewer SKza’s Q1.

For reference, we include Eik-QRL and HJB-QRL (the two offical variants from the EikHiQRL paper), which use Eikonal and Hamilton–Jacobi–Bellman critic constraints, respectively, and also use the hierarchical actors. Both the variant and ReQRL use models trained from scratch across three seeds, with the same training budget and critic-penalty weights. Eik-QRL/HJB-QRL results come from Table 1 of the Eik-HiQRL paper.

antmaze-navVariantReQRLEik-QRLHJB-QRL
medium92948431
large88957428
giant427780

We observe that the variant outperforms both baselines across all layouts, showing that ReQRL’s critic supervision and resulting values remain effective when shared by both policy levels. Standard ReQRL performs better still, supporting the practical benefit of our value separation, as discussed in our response to Rev. bkEK’s Q2. We would be happy to consider another targeted ablation if you had a different aspect of the setup in mind.

Second, we ablate whether boundary regularization improves goal reaching and how performance depends on its strength. For the new runs, we vary only the boundary-penalty weight while keeping the remaining setup fixed, and report mean success (%) over three seeds. Starred results are our ReQRL results in the paper.

antmaze-giant-navwb=0w_b=0100100300300*10001000EikHiQRL
Success (%)64 ± 373 ± 877 ± 677 ± 756 ± 9
humanoid-giant-navwb=0w_b=0300300500500*10001000EikHiQRL
Success (%)80 ± 683 ± 286 ± 883 ± 180 ± 5

In both antmaze and humanoidmaze, we see higher mean success with boundary regularization than without it, which shows the benefit of incorporating boundary information into critic learning. We also observe that increasing the weight further need not improve success. An excessive weight can favor reducing the boundary penalty at the expense of satisfying the interior Hamiltonian constraint and maximizing learned distances.

In our response to Rev. bkEK’s W2, we also discuss how this term regularizes the critic near the estimated edge of the data-supported region rather than imposing a hard restriction on the policy.

Reviewer_SKzaOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces ReQRL, an algorithms for offline Goal-conditioned Reinforcement Learning that overcomes some of the limitations of EikHiQRL by constraining the ciritc with a finite-horizon hamiltonian and boundary conditions.

Critical strengths
  1. The paper is well motivated and executed.
  2. It provides substantial empirical improvements compared to the other baselines.
Critical weaknesses
  1. Missing experiments in manipulation tasks.
Questions for authors
  1. A substantial part of ReQRL is inherited from Eik-HiQRL. However, the motivation for the hierarchical decomposition in Eik-HiQRL stems from the use of the Eikonal constraint, which assumes isotropic dynamics. Given that the proposed Hamiltonian does not rely on this assumption, is the state-space reduction for the high-level policy still necessary? Have the authors tried training a hierarchical actor with a single critic, as in HIQL? I would expect this to perform better than the Eik-QRL or HJB-QRL formulations in the Eik-HiQRL paper, due to the relaxation of the assumptions on the task dynamics.

  2. No robotic manipulation experiments are reported in the main text. Is manipulation still a significant limitation of this approach, as in Eik-HiQRL? Do the authors have any insights into how such limitations could be overcome?

  3. How is δ\delta in Eq. (9) computed, and why does the formulation of the Hamiltonian constraint change in Eq. (12)?

  4. I also did not fully understand how Eq. (12) is obtained from data. How are the boundary states identified in practice?

2 comments

Thank you for your positive feedback and thoughtful questions! We answer each question below. The current revision adds the new plots in Sec. 6.3 in red.

Q1. A substantial part of ReQRL is inherited from Eik-HiQRL. However, the motivation for the hierarchical decomposition in Eik-HiQRL stems from the use of the Eikonal constraint, which assumes isotropic dynamics. Given that the proposed Hamiltonian does not rely on this assumption, is the state-space reduction for the high-level policy still necessary? Have the authors tried training a hierarchical actor with a single critic, as in HIQL? I would expect this to perform better than the Eik-QRL or HJB-QRL formulations in the Eik-HiQRL paper, due to the relaxation of the assumptions on the task dynamics.

Thanks for this question, and yes, state reduction is not required for ReQRL's finite-horizon formulation. The formulation in Sec. 3–4 can use full-state critic and reachability coordinates, pairing the full-state critic gradient with estimated full state changes in the Hamiltonian. The reduced representation in Sec. 5 is our practical choice.

We clarify an important point though, that removing the isotropic dynamics assumption does not remove our core motivation for separating (a) goal-level geometry from (b) full-state control. Reaching a goal is defined in the goal-relevant coordinates, but the actions needed to reach a waypoint depend on the full dynamical state. In ReQRL, our high-level critic is thus tasked to learn shared quasimetric geometry in those coordinates as an effective approximation. And the low-level value and policy use the full state to learn how to reach there locally. This separation can remain useful even when the Hamiltonian accommodates more general dynamics.

To test this and answer Q1 directly, we additionally trained a full-state variant with a single full-state ReQRL critic shared by hierarchical actors, using full-state reachability and the spatial boundary penalty for the same training steps, with the same critic-penalty weights. Both the full-state variant and ReQRL use models trained from scratch across 3 seeds. The results for Eik-/HJB-QRL (that use the hierarchical actors as well) are taken from Tab. 1 of the Eik-HiQRL paper.

antmaze-navfull-state variantReQRLEik-QRLHJB-QRL
medium92948431
large88957428
giant427780

The full-state variant largely outperforms the reported Eik-QRL/HJB-QRL results across all settings, showing that ReQRL is effective without state reduction. But our standard ReQRL architecture performs better still, with the difference particularly visible on giant (77 vs 42). So, while our Hamiltonian formulation allows a single full-state critic, the experiments support our choice to separate goal-level value geometry from full-state control.

W1. Missing experiments in manipulation tasks.

Q2. No robotic manipulation experiments are reported in the main text. Is manipulation still a significant limitation of this approach, as in Eik-HiQRL? Do the authors have any insights into how such limitations could be overcome?

We note that our evaluation already includes a task involving object interaction through contact. In Appendix D (Table 8), we evaluated ReQRL on antsoccer from OGBench. Here, the goal reaching requires moving a ball through contact, and ReQRL achieves higher mean success than baselines in both settings.

We also want to clarify the distinction between the applicability of our finite-horizon formulation and the particular reachability estimator used in our experiments. ReQRL described in Sec. 3-4 is a general GCRL framework. Specifically, contact does not inherently prevent defining finite-horizon reachable sets or quasimetric goal-reaching costs. As we discuss in Lem. B1 (App. B), in a Markov state space, the triangle inequality for finite minimum-time costs follows from stitching feasible trajectories with contact effects already included in the dynamics and feasibility constraints. So, the formulation is not inherently restricted to locomotion or obstacle navigation.

However, we acknowledge that broader manipulation may require adapting our estimator in Sec. 5. This is because the same action can produce substantially different motions depending on the robot–object configuration and contact mode. One promising direction is to use richer object/contact representations and local dynamics models. But, the challenge will be to retain efficient reachability propagation and Hamiltonian evaluation under these contact-heavy dynamics. This remains an important direction for future work, as we mention in App. A.

Q3. How is δ\delta in Eq. (9) computed, and why does the formulation of the Hamiltonian constraint change in Eq. (12)?

In Eq. 9, δ\delta is a net state change over the horizon τ\tau. The constraint involves maximizing over the reachable set from Eq. 8, rather than computing individual displacements. In practice, we thus estimate the set of all these state changes by fitting the dynamics model to offline transition data and propagating it over the horizon (Sec. 5), which gives the zonotope in Eq.14. Hence, we do not need to compute/enumerate individual δ\delta or action sequences, since Eq.15 already evaluates the maximum over this estimated set in closed form exactly.

The interior Hamiltonian constraint does not change from Eq. 9 to Eq. 12. By definition, Hτdyn=sup⁡δ⟨−∇d,δ⟩/τ−1H_\tau^{\mathrm{dyn}}=\sup_\delta\langle-\nabla d,\delta\rangle/\tau-1, so setting it to zero (as in line 257) is exactly Eq. 9. Eq. 12 combines this with the distance-maximization objective and a boundary condition. No additional approximation is introduced in rewriting Eq. 9 as the interior constraint in Eq. 12.

Q4. I also did not fully understand how Eq. (12) is obtained from data. How are the boundary states identified in practice?

Eq. 12 gives the critic constraints, which we instantiate using two estimates from offline data (Sec 5). For the interior constraint, we fit an affine dynamics model to observed transitions and propagate it over kk steps to get the estimated reachable set in Eq. 14. Its closed-form support function gives the critic constraint in Eq. 15, as discussed in Q3 above.

For the boundary, we estimate the data-supported region from offline observations and take its edge as the estimated boundary. We obtain outward normals from a smoothed signed-distance field and apply the soft boundary penalty to observed states within a narrow band along this boundary (Sec. 5 and App. C.2). These then give the practical critic-training objective in Eq. (16), with soft penalties for both constraints.

Reviewer_bkEKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ReQRL for offline goal-conditioned RL, building on QRL and EikHiQRL. Instead of assuming equal movement speed in all directions, it uses the robot’s reachable states over kk steps to guide critic learning. It also adds a boundary constraint based on state-constrained Hamilton–Jacobi theory to account for obstacles.

Critical strengths
  • The paper presents a practical way to use finite-horizon reachable motions in critic learning. The closed-form constraint avoids searching over action sequences, making the approach more efficient for larger tasks.
  • It provides a practical way to select the horizon kk based on the robot’s dynamics, rather than tuning it separately for each task.
Critical weaknesses
  • The author discusses nonlinear robot dynamics but uses a single linear model for the entire dataset. It is unclear how this model captures heading-dependent movements.
  • The boundary estimation uses only offline data and may mistake unexplored areas for obstacles when data coverage is limited.
  • In Eq. (15), the model treats fitting errors GwG_w as if they were movements the robot could control. Since the robot cannot control these errors, the estimated reachable set may include movements that are not actually possible, raising concerns about the correctness of the proposed method.
  • I could not find any rollout videos in the supplementary material. Including videos of successful and failed runs would help assess how the policy handles obstacles and where it struggles. These behaviors are difficult to judge from success rates alone.

Minor

  • "This paper. We ask: "--Why does this line start like this? (line -54)
  • The reproducibility section is too brief and should include more details on the experimental setup, training process, and evaluation procedure to help readers reproduce the results.
  • There are a few typos in the paper. In Appendix C.4, “qausimetric” should be “quasimetric,” and “humanoind” should be “humanoidmaze.”
Questions for authors
  • How reliable is the first-order approximation over longer horizons such as k=48k=48? Could you provide results with smaller values of kk?
  • The critic only uses position, while the reachability model also considers velocity and heading (Appendix C.2). How does the critic handle states with the same position but different velocities or headings?
  • How much of the predicted progress comes from the residual generators, and how do the results change when these generators are removed or rescaled?

Additional Questions

  • The AI Use Statement (lines 488–490) mentions assistance with figure generation. Could you clarify which figures involved AI and whether it was used to write plotting or drawing code, or to generate the images themselves?
8 comments

Thank you for your feedback and questions! Below, we clarify on weaknesses and answer each question together. The current revision PDF has the new plots in Sec. 6.3 in red.

W1. The author discusses nonlinear robot dynamics but uses a single linear model for the entire dataset. It is unclear how this model captures heading-dependent movements.

Yes, we use a global affine model for estimating finite-horizon reachable motions. However, we clarify that the global affine model does not mean our estimated reachable sets are state-independent. The propagated drift term is obtained as Arkrs+∑j<kArjbrA_r^k r_s+\sum_{j<k}A_r^j b_r (see Sec. 5 and App. C.2), which leads to the csc_s term in Eq. (14). So, we can see that the velocity/heading information is still inside the predicted reachability through ArkrsA_r^k r_s, and shared model coefficients do not result in identical reachable sets across different dynamical states. The propagated action generators (ArjBrA_r^jB_r) also retain the accumulated effects of controls over multiple steps. In Eq. 15, we incorporate this state-dependent term directly into the critic constraint.

The affine coefficients themselves are shared across states by design, as you pointed out, and state dependence in the estimated reachable set is through the propagated drift. So, while the model does not capture arbitrary nonlinear heading–action interactions, it does retain state-dependent predicted motion and multistep control effects.

This is a practical approximation we found working well across the settings in Sec. 6, while allowing efficient reachable-set construction and exact closed-form Hamiltonian evaluation in Eq. 15. The finite-horizon formulation in ReQRL itself does not require this specific instantiation, and more expressive models can also be used, although reachability computation and Hamiltonian evaluation may become more involved.

W2. The boundary estimation uses only offline data and may mistake unexplored areas for obstacles when data coverage is limited.

Yes, we distinguish recovering the true physical boundary from using offline data to provide useful boundary supervision. If a region has no samples in our offline data, we cannot tell if the region is outside the unknown feasible domain or simply unvisited by the behavior policy. The two possibilities are not distinguishable from available observations, but they can imply different goal-reaching costs. Without additional assumptions, we cannot resolve this from the same offline data. This uncertainty is a general issue in offline learning, including QRL, EikHiQRL, and other baselines, though the extent of its impact depends on the method.

In ReQRL, we can view the region supported by the data as the region where we have signals for value learning. As mentioned above, outside of this region, an unseen area could either be feasible or not, and either option could agree with all the observed data at hand but give different route cost/values. So, our estimated boundary is simply meant to mark where our data coverage ends, rather than strict obstacles. Near this boundary, our soft penalty discourages the critic's descent direction from pointing into unobserved regions where goal-reaching costs cannot be reliably inferred from data alone (see lines 285-288).

The relevant practical question is whether such supervision improves goal-reaching performance. In our response to Q4 for Rev. gXAQ, our new ablations show that removing boundary regularization reduces success from 77% to 64% on antmaze-giant and from 86% to 80% on humanoid-giant. So, recovering the exact physical boundary is not necessary for the estimated boundary to provide useful critic supervision. Please also refer to our response to Rev. gXAQ's W1, where we also discuss ReQRL in online settings.

W3. In Eq. (15), the model treats fitting errors GwG_w as if they were movements the robot could control. Since the robot cannot control these errors, the estimated reachable set may include movements that are not actually possible, raising concerns about the correctness of the proposed method.

Q3. How much of the predicted progress comes from the residual generators, and how do the results change when these generators are removed or rescaled?

The main distinction is between (a) using residuals to account for model mismatch in an estimated reachable-set envelope and (b) claiming that every motion in this envelope is independently executable. Our instantiation does (a) without claiming (b).

First, we should clarify that GwG_w is there to account for the observed mismatch between the fitted affine model and the dynamics, which includes nonlinear/state-dependent effects and fitting inaccuracies (see lines 316-319). These residuals can contain systematic control effects (not merely noise). We propagate their empirical envelope together with the control parts to estimate our reachable set (Eq. 14 and below). We do this intentionally to account for the mismatches. But, as you noted in W3, independently propagating these residual generators relaxes their dependence on particular states and actions. So, the estimated set can include motions that are not individually executable. This is indeed an approximation of reachability (which retains model mismatches and realizes efficient computations). Such propagation of error envelopes is a known approach in reachability analysis [1].

There are also practical reasons why this instantiation is useful for ReQRL. In particular,

  1. It keeps reachability estimation and its Hamiltonian evalution tractable, since affine propagation gives a zonotope, and Eq. 15 can evaluate its largest predicted linear cost decrease exactly in closed form. Explicitly modeling residual dependences would require more involved training and muti-step reachbaility computation.
  2. When Eq. 15 is satified, every actual motion covered by the estimated set has predicted decrease at most 1. Enlarging the set makes this upper-bound constraint more restrictive, and underestimating it can leave actual motions unconstrained. So, although the maximizing estimated motion need not be executable, overestimation is conservative, specifically for the upper-bound side of the critic constraint.

Our main experiments in Sec. 6 show ReQRL's practical effectiveness compared to baselines. This does not mean that the full equality constraint is unaffected by this overestimate, so we further examine the residual contribution and provide the ablation requested in Q3 to see if accounting for the model discrepancies gives useful supervision.

We first used our trained critics and measured residual- and control-generator support contributions in Eq. 15 on held-out states and goals. These are respectively 0.87 and 0.43 on antmaze-giant and 0.95 and 0.13 on humanoid-giant. We observe substantial residual contributions, indicating that the mismatch envelope plays a significant role in the critic’s reachability supervision, as we discussed above. The residual contribution is not negligible, so it is useful to test whether it helps or hurts learning.

To this end, we also tested whether this contribution helps goal reaching by removing or halving the residual generators and retraining the critic and actors. We trained the new critics across three seeds with other components fixed. For residual scales {0,0.5,1}\lbrace 0, 0.5, 1\rbrace, success is respectively 73, 78, and 77 on antmaze-giant, and 70, 82, and 86 on humanoidmaze-giant. Including the residual generators improves mean success on both layouts (particularly for humanoidmaze). Half-scale residuals perform similarly to full-scale residuals on antmaze-giant, which suggests that the benefit does not rely on some careful choice of the scale. Thus, we see the practical utility of accounting for model mismatch, despite the relaxation identified in the review.

Learning tighter, fully state/action-dependent reachability estimates while retaining efficient construction and evaluation remains an important complementary direction for future work. The finite-horizon formulation is not tied to our specific reachability estimator.

[1] Alanwar et al. Data-Driven Reachability Analysis Using Matrix Zonotopes. L4DC, 2021; and Data-Driven Reachability Analysis From Noisy Data. IEEE TAC, 2023.

W4. I could not find any rollout videos in the supplementary material. Including videos of successful and failed runs would help assess how the policy handles obstacles and where it struggles. These behaviors are difficult to judge from success rates alone.

Thank you for this suggestion. We additionally provide successful and failed trajectories (Fig. 6), close-up trajectories with ReQRL’s critic gradient field (Fig. 7), and distance heatmap comparisons against EikHiQRL (Fig. 8). These figures and their details are now included in the revised draft in Sec. 6.3 in red. Rollout videos will definitely be included in our full code/data release on GitHub.

In Fig. 6, ReQRL reaches goals more often, consistent with Table 1, and its successful trajectories are more tightly clustered. EikHiQRL shows greater variation, including failed trajectories near walls. In Fig. 7, the critic’s descent directions −∇dθ(s,g)-\nabla d_\theta (s,g) follow the corridor around the turn, and they provide coherent local guidance toward the goal. Successful and failed rollouts follow similar routes, but the ant loses balance while turning close to the corner in the failed rollout. This shows that coherent route guidance alone does not eliminate failures during execution.

In Fig. 8, we observe smaller relative deviations from the shortest-path reference for ReQRL. We can see much closer agreement with the reference distance along routes around obstacles for ReQRL than EikHiQRL. Overall, these visuals help us relate the improved success rates to our explanation that better learned goal-reaching geometry around obstacles can help the policy navigate more consistently.

W5. There are some typoes etc.

Thanks for pointing out the typos. We will fix these in the draft.

W6. The reproducibility section is too brief and should include more details on the experimental setup, training process, and evaluation procedure to help readers reproduce the results.

Thanks for pointing this out. Experimental and evaluation details are provided in Appendix C.1, estimation procedures in C.2, and training settings and hyperparameters in C.5, with additional details in Appendix D. We will make these pointers and details more explicit in the reproducibility statement.

Q1. How reliable is the first-order approximation over longer horizons such as k=48k=48? Could you provide results with smaller values of kk?

First-order approximation error depends on the net reachable displacement in the critic's coordinates, not only on the number of steps (kk) alone. The horizon τ=kΔt\tau=k\Delta t balances accumulating control effects against keeping the critic’s first-order approximation local (Sec. 3, lines 186–193). Under the smoothness assumption in Thm. B.4, the linearization error is bounded by L∥δ∥2/2L\Vert\delta\Vert^2/2, where δ\delta is the net state change over τ\tau and LL is the Lipschitz constant of the gradient. Shorter horizons thus favor local accuracy, while longer ones capture accumulated control effects but can increase approximation error. In practice, we choose kk heuristically (Sec. 5 and App. C.2). Here, we provide an ablation over the values of kk additionally. We vary k∈{8,10,12,16}k\in\lbrace 8,10, 12,16\rbrace for antmaze and k∈{32,48,64}k\in\lbrace 32,48,64\rbrace for humanoidmaze on the large and giant (three seeds) with the pipeline fixed, changing the reachability and waypoint horizons together.

antmaze-navk=8k=810101212*1616
large95 ± 194 ± 195 ± 194 ± 3
giant72 ± 573 ± 877 ± 674 ± 5
humanoid-navk=32k=324848*6464
large81 ± 882 ± 782 ± 6
giant73 ± 186 ± 881 ± 5

We observe good goal-reaching performance across the tested horizons, which suggests some flexibility in choosing kk. Shorter horizons do not consistently yield better performance. Our chosen horizons achieve the highest success, and its choice is still relevant. In particular, the choice matters more on the giant layouts, consistent with differences in reachability estimates and waypoint selection accumulating over longer routes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.