- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a Pontryagin's Maximum Principle (PMP)-guided learning framework for neural Hamilton–Jacobi (HJ) reachability analysis. The authors aim to address the limitations of existing neural HJ methods, particularly the lack of direct gradient supervision when training solely with PDE residuals and the computational cost of generating additional supervision. The proposed approach is evaluated on a 13-dimensional quadrotor system and an 8-dimensional racing vehicle.
Critical strengths
-
The paper is well-written and clearly organized.
-
The paper addresses an important and practically relevant problem in safety-critical control. Improving the accuracy and computational efficiency of neural Hamilton–Jacobi reachability methods, particularly the estimation of value functions and their gradients, is meaningful for developing reliable safety controllers for high-dimensional dynamical systems
Critical weaknesses
1. Limited Novelty and Insufficient Motivation
The novelty of this work is questionable, as both the central motivation and several key methodological components have already been extensively investigated in prior studies, particularly [1] and [2].
First, the authors argue that conventional HJ PDE residual-based learning provides insufficient supervision for value gradients, potentially leading to inaccurate safety controllers. However, this issue has already been explicitly identified and investigated in [1], where the authors combine PDE residual learning with PMP-derived value and costate supervision. More importantly, [1] provides empirical evidence demonstrating that accurate costate estimation is essential for reliable closed-loop safety performance. Therefore, the motivation for introducing additional gradient supervision is not new.
Second, the authors emphasize the computational expense of conventional PMP-based supervision due to solving two-point boundary-value problems (BVPs), and present their method as an efficient alternative. However, this limitation has already been addressed in [2], which proposes a Pontryagin Neural Operator (PNO) that generates self-supervised value and costate signals through inexpensive forward and backward ODE rollouts, without requiring BVP solvers or externally generated supervisory data. Furthermore, [2] already combines PDE residual minimization with value and costate consistency losses, which is conceptually very similar to the proposed objective in Equations (15)–(16).
The main distinction of the present work appears to lie in its level-set-based initialization of PMP characteristics and short-horizon propagation aligned with the training curriculum. While these implementation choices may offer computational advantages, it remains unclear whether they constitute a substantial methodological contribution beyond existing PMP-guided neural HJ learning frameworks, rather than an incremental modification of trajectory initialization and supervision generation.
More concerningly, neither [1] nor [2] is discussed or cited in the manuscript, despite their direct relevance to the paper's motivation and methodology. This omission makes the novelty claims insufficiently justified. The authors should clearly distinguish their contributions from these prior methods and provide convincing evidence that the proposed PMP-guided supervision offers fundamental advantages beyond existing costate-based learning approaches.
[1] Zhang, L., Ghimire, M., Zhang, W., Xu, Z., and Ren, Y. Value Approximation for Two-Player General-Sum Differential Games with State Constraints. IEEE Transactions on Robotics.
[2] Zhang, L., Ghimire, M., Xu, Z., Zhang, W., and Ren, Y. Pontryagin Neural Operator for Solving General-Sum Differential Games with Parametric State Constraints. Learning for Dynamics and Control (L4DC), 2024.
2. Unclear Time Direction in PMP Characteristic Propagation
There appears to be a potential inconsistency in the time parameterization of the PMP characteristic equations. In Section 3.1, Equations (8)–(10) are derived with respect to the physical time (t), where the state and costate dynamics follow the standard PMP characteristic equations. However, in Section 3.2, the authors state that these equations are integrated forward in the time-to-go variable (\tau=T-t), from (\tau_c) to (\tau_c+h). Since (d\tau/dt=-1), integrating forward in (\tau) requires reversing the signs of both the state and costate dynamics. This transformation is not explicitly discussed in the manuscript. If Equation (10) is directly integrated using increasing (\tau), the resulting trajectories would not correspond to the intended characteristics.
3. Limited Baseline Coverage and Insufficient Ablation Studies
First, the number of baselines evaluated within each benchmark is relatively limited. For example, the single-cylinder experiment only compares the proposed method against DeepReachMPC, while other relevant methods are omitted from this setting. More importantly, prior PMP-based costate learning approaches, particularly [1] and [2], are not adequately discussed or evaluated.
Second, the paper lacks several important ablation studies. The proposed framework combines multiple components, including level-set initialization, PMP-based value supervision, gradient supervision, consistency filtering, and periodic reseeding. However, their individual contributions are not sufficiently isolated. In particular, it is unclear how much of the improvement comes from value supervision versus gradient supervision, or whether level-set-based initialization provides meaningful advantages over simpler trajectory sampling strategies.
4. Limited Benchmark Diversity
The highest-dimensional system considered in this paper has only 13 dimensions, while the other benchmark involves an 8-dimensional vehicle model. Moreover, the benchmark scenarios are largely toy-like. The quadrotor experiments only consider avoidance of one or four fixed cylindrical obstacles, while the vehicle experiments are conducted on a predefined oval track. These relatively simple and highly structured environments do not adequately reflect the complexity of realistic safety-critical control problems.
Questions for authors
Please refer to the weaknesses discussed above for the main concerns.
Also, how does the proposed method ensure that the generated PMP trajectories and costate labels remain valid when the safety constraint becomes active? Since Equation (10) is derived under the assumption that the obstacle constraint is inactive, it is unclear how the method handles trajectories that reach the safety boundary, where the value function may become nonsmooth. Is the proposed heuristic filtering sufficient to ensure the validity of the generated gradient supervision, or should additional constraint-related terms or boundary conditions be considered?
7 comments
Critical Weaknesses (continued)
-
[Limited Baseline Coverage and Insufficient Ablation Studies] Table 1 now compares five baselines (ExactDeepReach, Steer2Reach, DeepReachPMP, RL baseline, and DeepReachMPC) on both the single- and the multiple-cylinder tasks. On the vehicle, Table 2 also includes Steer2Reach.
Neither [1] nor [2] applies to our problem as published, since both address general-sum games rather than reachability. Both are now discussed in the introduction. We nevertheless adapted PNO [2] to reachability and included it in the new ablation study (Section 6, Table 5).
Table 5 also ablates each component the reviewer lists: value vs. gradient supervision, level-set initialization, consistency filtering, and periodic reseeding.
-
[Limited Benchmark Diversity] We agree that cluttered, dynamic, and multi-agent scenes are an important next step. We respectfully note, though, that the difficulty of both tasks lies in the dynamics rather than in the scene geometry.
- Quadrotor. At 13 dimensions, the system is far beyond the five or six that grid-based solvers can handle. In the multiple-cylinder task, the 0.2 m gap requires a roll of at least , so the safe set is shaped by near-saturated braking and swerving maneuvers.
- Vehicle. The dynamics are highly nonlinear and stiff, with a brush tire model operated at the friction limit, and the problem has an 8 s horizon and adversarial disturbance. Friction and disturbance scale are context inputs, so a single network covers all grip levels.
- Hardware. The learned value function runs as a 200 Hz safety filter on a full-scale vehicle, both at the friction limit and against a human driver trying to leave the track. In the counterfactual evaluation, 78 % of the driver's requests would have taken the car off the track within 2 seconds, yet the car never left it.
If the reviewer has a specific benchmark in mind, we would be glad to consider it for the final version.
Questions
-
We agree that Eq. (10) applies only where the value is smooth and the obstacle constraint is inactive (, Section 3.1). To mitigate invalid supervision, we apply three safeguards:
- Reject seeds whose gradient norms are outside prescribed bounds or whose assigned values are inconsistent with policy rollouts.
- Discard all subsequent labels once , using the necessary condition established by Proposition 2.
- Restrict propagation to short intervals and regenerate labels as the curriculum advances.
Importantly, these checks test necessary conditions only and do not guarantee label validity. Characteristics may become suboptimal or encounter nonsmooth regions without violating the obstacle condition. We therefore treat the generated labels as approximate supervision, combined with the HJ variational-inequality residual, rather than as exact labels.
References
[1] Zhang, Lei, et al. "Value approximation for two-player general-sum differential games with state constraints." IEEE Transactions on Robotics 40 (2024): 4631-4649.
[2] Zhang, Lei, et al. "Pontryagin neural operator for solving general-sum differential games with parametric state constraints." 6th Annual Learning for Dynamics & Control Conference. PMLR, 2024.
Reply to authors
Thank you for the detailed rebuttal. I appreciate the authors' efforts, particularly the clarification of the PMP time parameterization and the inclusion of additional baselines and ablations. However, several of my major concerns remain unresolved.
1. Limited methodological novelty
I still find the proposed contribution rather incremental, particularly when considering the existing work in [1] and [2].
Both prior studies have already investigated PMP-based value and costate supervision for neural HJ learning. In particular, [2] addresses the computational burden of generating such supervision without expensive BVP solvers. It also incorporates curriculum learning and adaptive sampling of PMP trajectories.
2. Comparisons with [1] and [2]
I understand that [1] and [2] originally address general-sum differential games, whereas the present work focuses on single-agent optimal control.
However, both prior approaches already consider state constraints, value approximation, and costate-based learning. Reducing a multi-player formulation to a single-controller setting does not appear to introduce a fundamental obstacle to adapting the underlying learning mechanisms.
In fact, the authors have already adapted PNO [2] to the proposed reachability benchmark, which demonstrates that such a comparison is technically feasible.
In particular, [2] already incorporates state-constraint handling through its costate dynamics and associated losses. Could the authors clarify why and how PNO formulation was modified to adapt to reachability?
3. Limited benchmark complexity
The 13-D quadrotor contains relatively simple dynamics. DeepReachMPC has covered up to 40-D publisher-subscriber systems.
For hardware experiments involving safety-critical control, I would expect more substantial supporting evidence, such as experimental videos or detailed intervention records. I could not identify an accompanying experimental video in the provided materials.
Dear Reviewer W6bD,
Thank you for your detailed review and for the time you have invested in our work. We are glad that you find the paper well written and clearly organized, and that you consider the problem important and practically relevant. Below, we address your concerns and questions.
Critical Weaknesses
[Limited novelty and insufficient motivation] Thank you for identifying these closely related works. Both are now discussed in the introduction (highlighted in blue). We agree that [1] combines PDE residual learning with value and costate supervision, and that PNO [2] generates such supervision without BVP solves. Our contribution concerns how approximate supervision is generated and screened for reachability, with two key differences.
Quantitative evidence. We adapted the released PNO [2] implementation to our eight-second reachability problem. We compare both methods at approximately three hours of training and additionally evaluate PNO with a substantially longer training schedule:
While PNO has a lower FPR, it identifies substantially fewer safe states. To distinguish value ranking from the choice of classification threshold, we also evaluate the ROC AUC against a common empirical rollout reference. Our method achieves 0.975±0.002, compared with 0.649±0.123 for matched-budget PNO and 0.765±0.020 for longer-trained PNO.
Qualitative evidence. PMP characteristics propagate value and gradient information only under smoothness and optimality assumptions. In minimum-over-time reachability, the active minimum or optimal solution branch can change along an extended characteristic, potentially making the true value nonsmooth and the propagated costate invalid. Short propagation limits how far we extrapolate local information before reseeding from the updated network. Filtering addresses a complementary problem: even a short characteristic may carry incorrect labels. For example, if a characteristic transports the value qi to a state where ℓ(xk)<qi, Proposition~2 proves that this label cannot represent the true reachability value, even if the numerical integration remains smooth.
In summary, short propagation limits how far we extrapolate information that may cease to represent the optimal, differentiable solution, while our filtering pipeline rejects detectable inconsistencies before they enter the replay buffer.
We hope these clarifications and additional experiments address your concerns regarding the methodological contribution and its practical significance. We would appreciate hearing whether any specific concerns remain.
[Unclear time direction in PMP characteristic propagation] The reviewer is right that propagating forward in time-to-go reverses the signs of the physical-time PMP equations. Our code already does this: rk4_step in src/pmp/integrate.py (lines 16-22 of the anonymized code) integrates the physical-time equations with step −Δt. This is equivalent to integrating dx/dτ=−f(x,u,d) and dp/dτ=(∂f/∂x)⊤p forward in τ. Hence, the concern arises from an ambiguity in the presentation rather than an error in the method. Section 3.1 now states the time convention, and Eq.(10) is written in time-to-go.