acceptodds
Under review as a conference paper at ICLR 2027

Guiding Reachability Learning with Pontryagin's Maximum Principle

Abstract

Hamilton–Jacobi (HJ) reachability provides a framework for analyzing the safety of autonomous systems under worst-case disturbances. Specifically, its value function identifies states from which failure is unavoidable, while its spatial gradient determines how a controller should act to preserve safety. Computing this function with traditional grid-based solvers is prohibitively expensive in high dimensions. Neural methods such as DeepReach improve scalability, but training on the HJ partial differential equation residual alone provides limited direct supervision for the gradient, while existing methods that generate additional value labels can be costly. We propose a sample-efficient method that supplements residual training with value and gradient labels generated using Pontryagin’s Maximum Principle (PMP). At intervals aligned with the training curriculum, we initialize state–costate pairs from the network’s current level sets and integrate the PMP characteristic equations over a short horizon to generate supervision beyond the curriculum frontier. At the same time, since the network provides the initial conditions, generating these labels avoids solving the expensive two-point boundary-value problems associated with PMP, thereby enabling efficient training. In experiments with a 13-dimensional quadrotor and an eight-dimensional racing vehicle, the method reduces training time and improves the accuracy of the learned safety value function in challenging settings. We further deploy the learned value function as a safety filter on a full-scale electric vehicle driven near the friction limit and against an adversarial human driver instructed to leave the track.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Hamilton-Jacobi-Reachability, Sobolev training, Pontryagin's maximum principle, optimal control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W6bDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a Pontryagin's Maximum Principle (PMP)-guided learning framework for neural Hamilton–Jacobi (HJ) reachability analysis. The authors aim to address the limitations of existing neural HJ methods, particularly the lack of direct gradient supervision when training solely with PDE residuals and the computational cost of generating additional supervision. The proposed approach is evaluated on a 13-dimensional quadrotor system and an 8-dimensional racing vehicle.

Critical strengths
  1. The paper is well-written and clearly organized.

  2. The paper addresses an important and practically relevant problem in safety-critical control. Improving the accuracy and computational efficiency of neural Hamilton–Jacobi reachability methods, particularly the estimation of value functions and their gradients, is meaningful for developing reliable safety controllers for high-dimensional dynamical systems

Critical weaknesses

1. Limited Novelty and Insufficient Motivation

The novelty of this work is questionable, as both the central motivation and several key methodological components have already been extensively investigated in prior studies, particularly [1] and [2].

First, the authors argue that conventional HJ PDE residual-based learning provides insufficient supervision for value gradients, potentially leading to inaccurate safety controllers. However, this issue has already been explicitly identified and investigated in [1], where the authors combine PDE residual learning with PMP-derived value and costate supervision. More importantly, [1] provides empirical evidence demonstrating that accurate costate estimation is essential for reliable closed-loop safety performance. Therefore, the motivation for introducing additional gradient supervision is not new.

Second, the authors emphasize the computational expense of conventional PMP-based supervision due to solving two-point boundary-value problems (BVPs), and present their method as an efficient alternative. However, this limitation has already been addressed in [2], which proposes a Pontryagin Neural Operator (PNO) that generates self-supervised value and costate signals through inexpensive forward and backward ODE rollouts, without requiring BVP solvers or externally generated supervisory data. Furthermore, [2] already combines PDE residual minimization with value and costate consistency losses, which is conceptually very similar to the proposed objective in Equations (15)–(16).

The main distinction of the present work appears to lie in its level-set-based initialization of PMP characteristics and short-horizon propagation aligned with the training curriculum. While these implementation choices may offer computational advantages, it remains unclear whether they constitute a substantial methodological contribution beyond existing PMP-guided neural HJ learning frameworks, rather than an incremental modification of trajectory initialization and supervision generation.

More concerningly, neither [1] nor [2] is discussed or cited in the manuscript, despite their direct relevance to the paper's motivation and methodology. This omission makes the novelty claims insufficiently justified. The authors should clearly distinguish their contributions from these prior methods and provide convincing evidence that the proposed PMP-guided supervision offers fundamental advantages beyond existing costate-based learning approaches.

[1] Zhang, L., Ghimire, M., Zhang, W., Xu, Z., and Ren, Y. Value Approximation for Two-Player General-Sum Differential Games with State Constraints. IEEE Transactions on Robotics.

[2] Zhang, L., Ghimire, M., Xu, Z., Zhang, W., and Ren, Y. Pontryagin Neural Operator for Solving General-Sum Differential Games with Parametric State Constraints. Learning for Dynamics and Control (L4DC), 2024.

2. Unclear Time Direction in PMP Characteristic Propagation

There appears to be a potential inconsistency in the time parameterization of the PMP characteristic equations. In Section 3.1, Equations (8)–(10) are derived with respect to the physical time (t), where the state and costate dynamics follow the standard PMP characteristic equations. However, in Section 3.2, the authors state that these equations are integrated forward in the time-to-go variable (\tau=T-t), from (\tau_c) to (\tau_c+h). Since (d\tau/dt=-1), integrating forward in (\tau) requires reversing the signs of both the state and costate dynamics. This transformation is not explicitly discussed in the manuscript. If Equation (10) is directly integrated using increasing (\tau), the resulting trajectories would not correspond to the intended characteristics.

3. Limited Baseline Coverage and Insufficient Ablation Studies

First, the number of baselines evaluated within each benchmark is relatively limited. For example, the single-cylinder experiment only compares the proposed method against DeepReachMPC, while other relevant methods are omitted from this setting. More importantly, prior PMP-based costate learning approaches, particularly [1] and [2], are not adequately discussed or evaluated.

Second, the paper lacks several important ablation studies. The proposed framework combines multiple components, including level-set initialization, PMP-based value supervision, gradient supervision, consistency filtering, and periodic reseeding. However, their individual contributions are not sufficiently isolated. In particular, it is unclear how much of the improvement comes from value supervision versus gradient supervision, or whether level-set-based initialization provides meaningful advantages over simpler trajectory sampling strategies.

4. Limited Benchmark Diversity

The highest-dimensional system considered in this paper has only 13 dimensions, while the other benchmark involves an 8-dimensional vehicle model. Moreover, the benchmark scenarios are largely toy-like. The quadrotor experiments only consider avoidance of one or four fixed cylindrical obstacles, while the vehicle experiments are conducted on a predefined oval track. These relatively simple and highly structured environments do not adequately reflect the complexity of realistic safety-critical control problems.

Questions for authors

Please refer to the weaknesses discussed above for the main concerns.

Also, how does the proposed method ensure that the generated PMP trajectories and costate labels remain valid when the safety constraint becomes active? Since Equation (10) is derived under the assumption that the obstacle constraint is inactive, it is unclear how the method handles trajectories that reach the safety boundary, where the value function may become nonsmooth. Is the proposed heuristic filtering sufficient to ensure the validity of the generated gradient supervision, or should additional constraint-related terms or boundary conditions be considered?

7 comments

Dear Reviewer W6bD,

Thank you for your detailed review and for the time you have invested in our work. We are glad that you find the paper well written and clearly organized, and that you consider the problem important and practically relevant. Below, we address your concerns and questions.

Critical Weaknesses

  1. [Limited novelty and insufficient motivation] Thank you for identifying these closely related works. Both are now discussed in the introduction (highlighted in blue). We agree that [1] combines PDE residual learning with value and costate supervision, and that PNO [2] generates such supervision without BVP solves. Our contribution concerns how approximate supervision is generated and screened for reachability, with two key differences.

    1. Short-horizon propagation. Unlike PNO's supervisory rollouts to the terminal time and the offline BVP-generated trajectories of [1], we initialize PMP characteristics from the learned value and gradient at the current curriculum frontier, propagate them for a short horizon, and regenerate labels as training advances.
    2. Label filtering. We screen the trajectories for obstacle consistency, bounded gradient norms, and agreement between their assigned values and policy rollouts. During propagation, we truncate characteristics once their transported values violate the necessary obstacle condition V≤ℓV\leq \ell.

    Quantitative evidence. We adapted the released PNO [2] implementation to our eight-second reachability problem. We compare both methods at approximately three hours of training and additionally evaluate PNO with a substantially longer training schedule:

    MethodTPRFPRTraining time (h)
    PNO, matched budget0.024±0.0550.024 \pm 0.0550.0003±0.00060.0003 \pm 0.00063.09±0.013.09 \pm 0.01
    PNO, longer training0.198±0.0890.198 \pm 0.0890.0024±0.00130.0024 \pm 0.001325.8±1.225.8 \pm 1.2
    Ours0.907±0.0110.907 \pm 0.0110.0034±0.00060.0034 \pm 0.00063.08±0.103.08 \pm 0.10

    While PNO has a lower FPR, it identifies substantially fewer safe states. To distinguish value ranking from the choice of classification threshold, we also evaluate the ROC AUC against a common empirical rollout reference. Our method achieves 0.975±0.0020.975\pm 0.002, compared with 0.649±0.1230.649\pm 0.123 for matched-budget PNO and 0.765±0.0200.765\pm 0.020 for longer-trained PNO.

    Qualitative evidence. PMP characteristics propagate value and gradient information only under smoothness and optimality assumptions. In minimum-over-time reachability, the active minimum or optimal solution branch can change along an extended characteristic, potentially making the true value nonsmooth and the propagated costate invalid. Short propagation limits how far we extrapolate local information before reseeding from the updated network. Filtering addresses a complementary problem: even a short characteristic may carry incorrect labels. For example, if a characteristic transports the value qiq_i to a state where ℓ(xk)<qi\ell(\boldsymbol{x}_k)<q_i, Proposition~2 proves that this label cannot represent the true reachability value, even if the numerical integration remains smooth.

    In summary, short propagation limits how far we extrapolate information that may cease to represent the optimal, differentiable solution, while our filtering pipeline rejects detectable inconsistencies before they enter the replay buffer.

    We hope these clarifications and additional experiments address your concerns regarding the methodological contribution and its practical significance. We would appreciate hearing whether any specific concerns remain.

  2. [Unclear time direction in PMP characteristic propagation] The reviewer is right that propagating forward in time-to-go reverses the signs of the physical-time PMP equations. Our code already does this: rk4_step in src/pmp/integrate.py (lines 16-22 of the anonymized code) integrates the physical-time equations with step −Δt-\Delta t. This is equivalent to integrating dx/dτ=−f(x,u,d)\mathrm{d}x/\mathrm{d}\tau = -f(x,u,d) and dp/dτ=(∂f/∂x)⊤p\mathrm{d}p/\mathrm{d}\tau = (\partial f/\partial x)^\top p forward in τ\tau. Hence, the concern arises from an ambiguity in the presentation rather than an error in the method. Section 3.1 now states the time convention, and Eq.(10) is written in time-to-go.

Critical Weaknesses (continued)

  1. [Limited Baseline Coverage and Insufficient Ablation Studies] Table 1 now compares five baselines (ExactDeepReach, Steer2Reach, DeepReachPMP, RL baseline, and DeepReachMPC) on both the single- and the multiple-cylinder tasks. On the vehicle, Table 2 also includes Steer2Reach.

    Neither [1] nor [2] applies to our problem as published, since both address general-sum games rather than reachability. Both are now discussed in the introduction. We nevertheless adapted PNO [2] to reachability and included it in the new ablation study (Section 6, Table 5).

    Table 5 also ablates each component the reviewer lists: value vs. gradient supervision, level-set initialization, consistency filtering, and periodic reseeding.

  2. [Limited Benchmark Diversity] We agree that cluttered, dynamic, and multi-agent scenes are an important next step. We respectfully note, though, that the difficulty of both tasks lies in the dynamics rather than in the scene geometry.

    • Quadrotor. At 13 dimensions, the system is far beyond the five or six that grid-based solvers can handle. In the multiple-cylinder task, the 0.2 m gap requires a roll of at least 54∘54^\circ, so the safe set is shaped by near-saturated braking and swerving maneuvers.
    • Vehicle. The dynamics are highly nonlinear and stiff, with a brush tire model operated at the friction limit, and the problem has an 8 s horizon and adversarial disturbance. Friction and disturbance scale are context inputs, so a single network covers all grip levels.
    • Hardware. The learned value function runs as a 200 Hz safety filter on a full-scale vehicle, both at the friction limit and against a human driver trying to leave the track. In the counterfactual evaluation, 78 % of the driver's requests would have taken the car off the track within 2 seconds, yet the car never left it.

    If the reviewer has a specific benchmark in mind, we would be glad to consider it for the final version.

Questions

  1. We agree that Eq. (10) applies only where the value is smooth and the obstacle constraint is inactive (V<ℓV<\ell, Section 3.1). To mitigate invalid supervision, we apply three safeguards:

    • Reject seeds whose gradient norms are outside prescribed bounds or whose assigned values are inconsistent with policy rollouts.
    • Discard all subsequent labels once ℓ(xk)<qi−εℓ\ell(\boldsymbol{x}_k)<q_i -\varepsilon_{\ell}, using the necessary condition established by Proposition 2.
    • Restrict propagation to short intervals and regenerate labels as the curriculum advances.

    Importantly, these checks test necessary conditions only and do not guarantee label validity. Characteristics may become suboptimal or encounter nonsmooth regions without violating the obstacle condition. We therefore treat the generated labels as approximate supervision, combined with the HJ variational-inequality residual, rather than as exact labels.

References

[1] Zhang, Lei, et al. "Value approximation for two-player general-sum differential games with state constraints." IEEE Transactions on Robotics 40 (2024): 4631-4649.

[2] Zhang, Lei, et al. "Pontryagin neural operator for solving general-sum differential games with parametric state constraints." 6th Annual Learning for Dynamics & Control Conference. PMLR, 2024.

Reviewer_W6bDOpenReview ↗

Reply to authors

Thank you for the detailed rebuttal. I appreciate the authors' efforts, particularly the clarification of the PMP time parameterization and the inclusion of additional baselines and ablations. However, several of my major concerns remain unresolved.

1. Limited methodological novelty

I still find the proposed contribution rather incremental, particularly when considering the existing work in [1] and [2].

Both prior studies have already investigated PMP-based value and costate supervision for neural HJ learning. In particular, [2] addresses the computational burden of generating such supervision without expensive BVP solvers. It also incorporates curriculum learning and adaptive sampling of PMP trajectories.

2. Comparisons with [1] and [2]

I understand that [1] and [2] originally address general-sum differential games, whereas the present work focuses on single-agent optimal control.

However, both prior approaches already consider state constraints, value approximation, and costate-based learning. Reducing a multi-player formulation to a single-controller setting does not appear to introduce a fundamental obstacle to adapting the underlying learning mechanisms.

In fact, the authors have already adapted PNO [2] to the proposed reachability benchmark, which demonstrates that such a comparison is technically feasible.

In particular, [2] already incorporates state-constraint handling through its costate dynamics and associated losses. Could the authors clarify why and how PNO formulation was modified to adapt to reachability?

3. Limited benchmark complexity

The 13-D quadrotor contains relatively simple dynamics. DeepReachMPC has covered up to 40-D publisher-subscriber systems.

For hardware experiments involving safety-critical control, I would expect more substantial supporting evidence, such as experimental videos or detailed intervention records. I could not identify an accompanying experimental video in the provided materials.

Reviewer_XvGyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes a method for training neural Hamilton-Jacobi reachability models using additional value and gradient supervision from Pontryagin’s Maximum Principle (PMP). At selected stages of training, it samples states on level sets of the current value network and uses the network’s gradients to initialize short state-costate trajectories. These trajectories generate training labels beyond the current time horizon without requiring two-point boundary-value solves. The labels are filtered and used alongside the HJ residual loss. The paper evaluates the method on quadrotor and racing-vehicle tasks and demonstrates a learned safety filter on a full-scale electric vehicle.

Critical strengths
  1. The demonstration of the learned safety filter on a full-scale electric vehicle is a strength, particularly the tests near the friction limit and with an adversarial human driver.
  2. The experiments cover two different high-dimensional systems, a 13D quadrotor and an 8D vehicle, providing broader empirical evidence than a single task.
Critical weaknesses
  1. The methodological contribution beyond prior work needs clearer explanation. The method builds on DeepReach's neural HJ formulation, while using PMP-based value and costate information to guide learning is already explored by Zhang et al. (2024) in their Pontryagin Neural Operator (PNO), which is not currently discussed in the paper. The level-set initialization, short propagation beyond the training horizon, and label filtering distinguish the proposed procedure, but the paper should explain what these choices add beyond earlier PMP-guided learning methods. Without this comparison, the extent of the methodological advance is unclear.

  2. The hardware implementation is valuable. However, I was not able to find any visual supplement (e.g., a video or a picture with frames) showcasing the method in action. Such material would help readers understand the experimental conditions and results much easily.

Questions for authors
  1. How does the proposed supervision scheme differ from PNO (Zhang et al., 2024)? Would it be possible to discuss this related work and clarify the differences in how value and costate training targets are generated and used.
  2. What evidence isolates the benefit of level-set initialization, propagation beyond the current training horizon, and label filtering over simpler PMP-based supervision?

References

Zhang, L., Ghimire, M., Xu, Z., Zhang, W. & Ren, Y.. (2024). Pontryagin neural operator for solving general-sum differential games with parametric state constraints. Proceedings of the 6th Annual Learning for Dynamics & Control Conference, in Proceedings of Machine Learning Research 242:1728-1740 Available from https://proceedings.mlr.press/v242/zhang24f.html.

5 comments

Dear Reviewer XvGy,

Thank you for your review and for the time you have invested in our work. We are glad that you appreciated the experiments on two high-dimensional systems, and in particular the hardware experiments on the full-scale vehicle. Below, we address your concerns and questions.

Weaknesses

  1. [Methodological Contribution] We now discuss PNO [1] in the revised introduction (highlighted in blue). Both methods generate value and costate supervision without solving BVPs. Our contribution concerns how approximate supervision is generated and screened for reachability, with two key differences.

    • Short-horizon propagation. PNO generates its supervisory targets using rollouts to the terminal time. We instead initialize PMP characteristics on level sets of the learned value at the current curriculum frontier, propagate them for a short horizon, and regenerate labels as training advances. This places the supervision near the estimated boundary of the reachable tube and limits error accumulation.
    • Label filtering. We screen the trajectories for obstacle consistency, bounded gradient norms, and agreement between their assigned values and policy rollouts. During propagation, we truncate characteristics once their transported values violate the necessary obstacle condition V≤ℓV \le \ell.

    Quantitative evidence. PNO was originally developed for general-sum differential games rather than minimum-over-time reachability, so we adapted the authors' released implementation to our eight-second vehicle problem. We use the minimum safety margin along each rollout as the value target and reset the backward-propagated costate whenever a new minimum becomes active. We compare both methods at approximately three hours of training and additionally evaluate PNO with a substantially longer training schedule.

    MethodTPRFPRTraining time (h)
    PNO, matched budget0.024±0.0550.024 \pm 0.0550.0003±0.00060.0003 \pm 0.00063.09±0.013.09 \pm 0.01
    PNO, longer training0.198±0.0890.198 \pm 0.0890.0024±0.00130.0024 \pm 0.001325.8±1.225.8 \pm 1.2
    Ours0.907±0.0110.907 \pm 0.0110.0034±0.00060.0034 \pm 0.00063.08±0.103.08 \pm 0.10

    While PNO has a lower FPR, it identifies substantially fewer safe states. To distinguish value ranking from the choice of classification threshold, we also evaluate the ROC AUC against a common empirical rollout reference. Our method achieves 0.975±0.0020.975\pm 0.002, compared with 0.649±0.1230.649\pm 0.123 for matched-budget PNO and 0.765±0.0200.765\pm 0.020 for longer-trained PNO.

    Qualitative evidence. PMP characteristics propagate value and gradient information only under smoothness and optimality assumptions. In minimum-over-time reachability, the active minimum or optimal solution branch can change along an extended characteristic, potentially making the true value nonsmooth and the propagated costate invalid.

    Short propagation limits how far we extrapolate local information before reseeding from the updated network. Filtering addresses a complementary problem: even a short characteristic may carry incorrect labels. For example, if a characteristic transports the value qiq_i to a state where ℓ(xk)<qi\ell(\boldsymbol{x}_k)<q_i, Proposition 2 proves that this label cannot represent the true reachability value, even if the numerical integration remains smooth.

  2. [Visual Supplement] Thank you for this suggestion. We have added Figure 5 (Appendix D), which shows four onboard camera frames from an adversarial-driving run on the vehicle, synchronized with the vehicle log. While entering a left-hand corner, the human driver keeps the steering wheel turned to the right, toward the outer track boundary. The filter first passes the request through and then overrides it: while the driver still steers right, it steers up to 10∘10^\circ to the left (front-wheel angle), cuts the drive torque and brakes, and the car follows the corner, staying more than 2 m inside the track boundary. Below the frames, the figure plots the driver's requests, the applied steering and pedal commands, and the learned value V(x)V(x) over time. Here, V(x)V(x) is the network's estimate of the worst-case distance to the track boundary. It drops to about zero shortly after the override begins and stays within ±0.08\pm 0.08 m of zero while the driver keeps steering out, i.e., the filter keeps the car on the edge of the learned safe set. The figure also shows a top view of the driven path, together with where the driver's request alone would have led: off the track within about 1.2 s.

    We also recorded onboard videos of several adversarial and autonomous driving runs, as well as drone videos (bird's-eye and side-tracking views). The onboard camera moves during fast cornering, so the videos show the driver and other identifying details. We therefore cannot share them without breaking anonymity, but we plan to release them with the de-anonymized version of the paper and link them from it.

Questions

  1. We have addressed the methodological differences from PNO and provided a direct experimental comparison in our response to the first weakness above. We have also added the quantitative results to our new ablation study in Section 6.

  2. Simpler PMP-based supervision does not learn a usable tube on the vehicle. Networks trained on labels generated over the full horizon claim no state as safe, PMP-based collocation alone is unstable (TPR 0.44±0.490.44\pm0.49), and networks trained on PNO's rollout labels claim almost no state as safe. Table 5 isolates each of our components by removing it from the full method (five seeds each).

    • Level-set initialization sets each label's value. Since the value is constant along an optimal characteristic (Lemma 1), a seed on the level set {Vθ=qi}\{V_\theta = q_i\} labels its whole characteristic with qiq_i. We can therefore choose which values are supervised, e.g., by concentrating labels near the zero level set. With uniformly sampled seeds, the label values follow the sampling distribution instead. Table 5 compares this with other initializations, namely uniform seeds and seeds on the zero level set.
    • Propagation beyond the training horizon places labels at times-to-go that the residual curriculum has not yet reached. Labels inside the horizon mostly restate what the network already fits.
    • Filtering lowers the false-positive rate and enlarges the probabilistic certified volume. Without the Proposition 2 filter, the FPR rises from 3.4×10−33.4 \times 10^{-3} to 4.1×10−34.1 \times 10^{-3} and the probabilistic certified volume falls from 0.71%0.71\% to 0.14%0.14\%. Without the consistency check, the FPR rises to 3.9×10−33.9\times10^{-3} and the volume falls to 0.35%0.35\%.

References

[1] Zhang, Lei, et al. "Pontryagin neural operator for solving general-sum differential games with parametric state constraints." 6th Annual Learning for Dynamics & Control Conference. PMLR, 2024.

Dear Reviewer XvGy,

Thank you again for the constructive feedback. We have addressed your concerns in our rebuttal and would appreciate any follow-up on whether our response resolves the issues raised or whether any concerns remain.

Best,

The Authors

Reviewer_6p32OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper augments neural Hamilton–Jacobi reachability training with value and spatial-gradient supervision generated through Pontryagin’s Maximum Principle (PMP). At selected stages of the time curriculum, the current network initializes state–costate pairs on its level sets, and short characteristic integrations then generate approximate labels beyond the curriculum frontier, avoiding two-point boundary-value solves. These labels supplement the HJI residual, with consistency checks and periodic reseeding intended to mitigate errors. Validation includes a 13-dimensional quadrotor, an eight-dimensional vehicle model, and full-scale vehicle experiments involving aggressive autonomous driving and an adversarial human driver.

Critical strengths
  1. The proposed method has a well-motivated and practical supervision mechanism. It exploits the connection between PMP costates and value gradients to provide explicit supervision. Initializing propagation from the current network offers a computationally attractive alternative to generating labels through boundary-value solves or sampling-based MPC.

  2. Meaningful evidence of efficiency and practical utility is provided. The experiments demonstrate competitive performance with substantially lower training time than MPC-based supervision in the evaluated settings. Validation on high-dimensional systems and a full-scale vehicle strengthens the practical relevance of the proposed approach.

Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution of gradient supervision. The residual-only comparison removes both value and gradient labels, while comparisons with MPC supervision and Dallas et al. also change label generation, sampling, and temporal coverage. Vehicle-specific false-positive fine-tuning further complicates attribution. Consequently, the results support the effectiveness of the overall pipeline more clearly than the claim that direct gradient supervision is a key source of improvement. A controlled comparison retaining the value labels and training protocol while removing only the gradient loss would materially strengthen this central claim.

  2. The methodological contribution is not clearly established. PMP-based characteristic propagation [1] and value-gradient supervision [2] have both been studied previously. It remains unclear what substantive methodological advance the proposed training procedure provides beyond combining these existing ideas. The experiments demonstrate the effectiveness of the complete pipeline, but do not establish which specific design accounts for an advance over straightforward extensions of prior methods.

[1] Nakamura-Zimmerer et al., Adaptive Deep Learning for High-Dimensional Hamilton–Jacobi–Bellman Equations, SIAM Journal on Scientific Computing, 2021. [2] Dallas et al., Boundary Sampling to Learn Predictive Safety Filters via Pontryagin’s Maximum Principle, arXiv:2604.13325, 2026.

Questions for authors
  1. With the label-generation and training protocol held fixed, how do PDE-only, PDE-plus-value, and PDE-plus-value-and-gradient training compare? For the vehicle task, how much improvement is attributable to the subsequent false-positive fine-tuning?

  2. The reported total training times are informative. Could the authors additionally break down the time spent on seed projection, consistency rollouts, PMP integration, and network optimization, and report replay-buffer memory usage? How significant is the requirement to compute dynamics Jacobians for the method’s computational cost and applicability?

  3. Appendix A.3 states that the deployed filter uses only value evaluations. Under the same filter and training setup, does adding gradient supervision improve closed-loop performance compared with value-only supervision?

5 comments

Dear Reviewer 6p32,

Thank you for your review and for the time you have invested in our work. We are glad that you find the supervision mechanism well motivated and practical, and that you value the evidence of its efficiency on high-dimensional systems and on a full-scale vehicle. Below, we address your concerns and questions.

Weaknesses

  1. [Ablations] We have added a new section, "Ablation Study" (Sec. 6), where we present Table 5. The table removes one component of our method at a time: the gradient loss, the value loss, the level-set initialization, the consistency check, the Proposition 2 filter, and the stage-wise regeneration (labels generated once over the full horizon), and measures the performance degradation. It also includes DeepReachPMP [1] and PNO [2] baselines at our compute budget and at eight times that budget, and MADR [3] restricted to our data-generation time. Each variant is trained with five seeds, and in total Table 5 reports over 100 training and fine-tuning runs.

    The rows of Table 5 that isolate the gradient supervision on the vehicle are reproduced below.

    TrainingBefore FT: TPR ↑Before FT: FPR ↓After FT: TPR ↑After FT: FPR [10⁻³] ↓After FT: μ_ε [%] ↑
    PDE only0.0000.0000.000n/a0.00
    PDE + value (w_g = 0)0.915 ± 0.0590.060 ± 0.0140.900 ± 0.0194.0 ± 1.20.55 ± 0.60
    PDE + gradient (w_v = 0)0.935 ± 0.0370.066 ± 0.0090.917 ± 0.0244.1 ± 1.10.41 ± 0.31
    PDE + value + gradient (ours)0.951 ± 0.0160.058 ± 0.0080.907 ± 0.0113.4 ± 0.60.71 ± 0.58

    Without PMP labels, the network claims no state as safe, so the labels themselves have the largest effect. Adding the gradient loss to the value labels raises the TPR before fine-tuning from 0.9150.915 to 0.9510.951 and reduces the spread across seeds. After fine-tuning, it gives the lowest FPR (0.00340.0034 vs. 0.00400.0040) and the largest certified volume με\mu_{\varepsilon} (0.710.71% vs. 0.550.55%) at a similar TPR.

    The remaining rows of Table 5 show three things. First, the way the labels are generated matters most. Labels generated once over the full horizon, PMP used only for collocation [1], PNO [2], and MADR [3] at our budget all fail or are unstable, with a TPR between 00 and 0.440.44, compared with 0.950.95 for our method. Second, the label filters matter for certification. Removing the consistency check halves the certified volume (from 0.710.71% to 0.350.35%), and removing the Proposition 2 filter reduces it by a factor of five (to 0.140.14%). Third, the choice of initialization has a smaller effect. Uniform and zero-level-set seeds reach a TPR similar to ours, but zero-level-set seeds give a higher FPR after fine-tuning (5.0×10−35.0\times 10^{-3} vs. 3.4×10−33.4\times 10^{-3}).

    We hope that the comprehensive study in Section 6, with over 100 runs, sufficiently justifies the effectiveness of our method and clarifies where each benefit comes from.

  2. [Methodological Contribution] [4] also generate labels during training, but each label requires a two-point boundary-value solve. [1] use PMP only to place collocation points, once, before training. We instead initialize the PMP equations from the network's current value and gradient at the curriculum frontier, so each label is a short initial-value integration. Because PMP gives only necessary conditions, characteristics integrated over long horizons may be only locally optimal, and their labels can degrade the quality of the learned value function, especially when the safe set is relatively small. Propagating each characteristic briefly and regenerating labels online along the curriculum limits that risk. Indeed, on the 8 s racing vehicle problem, generating PMP labels once over the full horizon fails completely (TPR 0), and the approach of [1] is unstable across seeds (TPR 0.44±0.490.44\pm0.49), whereas our method reaches a TPR of 0.95±0.020.95\pm0.02 before fine-tuning (Sec. 6, Table 5).

Questions

Thank you for these valuable questions; we are happy to answer them.

  1. We report this comparison, including the effect of fine-tuning, in our response above (Sec. 6, Table 5 of the paper).

    • Time. Appendix E.2 (Table 7) now breaks down the training time for all three tasks, averaged over five seeds. Label generation (projection, consistency rollouts, PMP integration, and buffer handling) takes 1.0 min of the single-cylinder run (0.5 %), 1.1 min of the multiple-cylinder run (0.4 %), and 27.6 min of the vehicle run (14.9 %). The vehicle's share is larger because its 8 s horizon needs 79 label stages, compared with 9 for the 1 s quadrotor tasks. All components are implemented in PyTorch, and each run uses a single GPU (NVIDIA RTX A5000 for the quadrotors, RTX 3090 for the vehicle).
    • Memory. The replay buffer stores (x,p,τ,V)(x, p, \tau, V) in fp32 in host memory, i.e., (2n+2)⋅4(2n+2)\cdot 4 bytes per row. It grows to 22.9±0.522.9\pm0.5 M rows (1.65 GB) on the vehicle and to about 0.9 M rows (0.10 GB) on the quadrotor tasks.
    • Jacobians. Dynamics Jacobians enter only through the costate equation. We differentiate the Hamiltonian with automatic differentiation, which gives (∂f/∂x)⊤p(\partial f / \partial x)^\top p directly, so the n×nn\times n Jacobian is never formed or derived by hand. The method needs a known model that is differentiable in the state, as assumed in Section 2, so in principle it also applies to differentiable simulators.
  2. Yes. We deployed both value-only and value + gradient networks in simulation with the same filter. Gradient supervision does improve closed-loop performance, such as a higher mean speed and a lower lap time. Since our method provides the gradient, it can also be paired with gradient-based filters such as a CBF-QP. We have therefore added results for both networks with a CBF-QP filter to the table below. The driving performance is similar to that of the sampling-based filter, but the control jerk is reduced by more than an order of magnitude for both networks.

    MetricSampling: Value onlySampling: Value + gradientCBF-QP: Value onlyCBF-QP: Value + gradient
    Collisions0/1000/1000/1000/100
    Filter interventions [%]71.6 ± 2.871.9 ± 2.871.1 ± 2.770.5 ± 2.8
    Mean speed [m/s]11.26 ± 0.1411.64 ± 0.1511.29 ± 0.1311.67 ± 0.14
    Corner speed [m/s]10.64 ± 0.3611.21 ± 0.4110.68 ± 0.3511.24 ± 0.41
    Lap time [s]19.72 ± 0.1919.25 ± 0.2419.70 ± 0.1919.23 ± 0.24
    Min. track clearance [m]2.49 ± 0.242.72 ± 0.402.52 ± 0.252.73 ± 0.39
    Max. sideslip |β| [rad]0.061 ± 0.0390.044 ± 0.0370.065 ± 0.0480.047 ± 0.046
    Jerk0.013 ± 0.0010.012 ± 0.0010.0005 ± 0.00050.0004 ± 0.0001

References

[1] Dallas, James, et al. "Boundary Sampling to Learn Predictive Safety Filters via Pontryagin's Maximum Principle." arXiv preprint arXiv:2604.13325 (2026).

[2] Zhang, Lei, et al. "Pontryagin neural operator for solving general-sum differential games with parametric state constraints." 6th Annual Learning for Dynamics & Control Conference. PMLR, 2024.

[3] Teoh, Ryan, et al. "MADR: MPC-guided adversarial DeepReach." 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.

[4] Nakamura-Zimmerer, Tenavi, Qi Gong, and Wei Kang. "Adaptive deep learning for high-dimensional Hamilton--Jacobi--Bellman equations." SIAM Journal on Scientific Computing 43.2 (2021): A1221-A1247.

Dear Reviewer 6p32,

Thank you again for the constructive feedback. We have addressed your concerns in our rebuttal and would appreciate any follow-up on whether our response resolves the issues raised or whether any concerns remain.

Best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.