acceptodds
Withdrawn from ICLR 2027

Data-Driven Hamiltonian for Neural Optimal Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) has emerged as a prominent data-driven approach to optimal control, but often requires substantial amounts of data and provides limited guarantees when encountering out-of-distribution state-action pairs. In this paper, we propose an alternative approach grounded in the well-established Hamilton-Jacobi framework in optimal control theory. Rather than requiring an explicit analytical model of the system dynamics to evaluate the Hamiltonian, we replace this with a data-driven Hamiltonian (DDH), which we generalize to a broad class of optimal control problems. In contrast to standard RL approaches, the DDH provides a principled mechanism, requiring only minimal prior system information such as a Lipschitz constant, for quantifying and accounting for uncertainty in regions not covered by the data, enabling the derivation of controllers with performance guarantees. Additional prior system knowledge, when available, can be readily incorporated to improve performance. For high-dimensional systems, we further introduce a physics-informed neural approximation of the value function, trained using a novel procedure that generates worst-case fictitious rollouts evaluated through the DDH. On an LQR problem, the DDH reaches near-optimal cost on every dataset we test, including random data on which offline RL methods remain far from optimal, and needs far fewer samples than online RL. On a nonlinear lunar lander benchmark, the DDH achieves an optimality gap to a trajectory-optimization reference more than smaller than CQL and FQL, and matches the performance of SAC with approximately fewer samples. These results establish the DDH as a promising framework for data-driven optimal control, particularly in settings where available data are limited in quantity or quality.

Primary area:
reinforcement learning
Keywords:
Data-Driven Optimal Control, Reinforcement Learning, Hamilton-Jacobi-Bellman, Physics-Informed Neural Networks, Adversarial Rollouts

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2zq5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates data-driven optimal control by extending the data-driven Hamiltonian of prior work (Choi et al, 2025) in terms of running and terminal costs of the state trajectory and leveraging this within the Hamilton-Jacobi-Bellman framework. Furthermore, the paper uses a physics-informed neural net approximation of the value function to cope with high-dimensional control tasks.

The paper presents itself mainly as a theory paper, but also provides two experimental validations: The first one is mainly to verify the theory by considering a simple two-dimensional linear dynamical system in standard form and comparing against LQR and other baselines. The second experiment investigates a 6-dimensional variant of the lunar lander problem.

Critical strengths

The paper is well written, clean, structured, presented and motivated. A RL paper on solid theoretical grounds has merits on its own. Also bridging classical control theoretical concepts and learning-based methods (RL, PINN) is beneficial to the field.

Critical weaknesses

Novelty

While overall the paper is interesting, looking the prior work in (Choi et al, 2025) where the DDH machinery was mainly developed, the question is raised what the contribution of this paper is relative to the prior state of the art.

To this end, I recommend to reorganize Introduction into motivation, prior art, identification of research gaps, and contribution to fill in the gaps, probably as dedicated subsections.

Section 3 is supposed to be original work of this paper. I would recommend to insert a subsection 3.x prior the current section 3.1. This section 3.x, however, is mainly already developed by (Choi et al, 2025); this submission mainly refers to their paper, but does not provide clearly marked theorems and proofs original to this submission. This, however, makes it difficult to position this paper is a theory contribution.

Positioning of the paper

Theory versus experimental.

This paper is positioned primarily a theory paper. It provides experiments too, but the first mainly serves the theory and the second is a single demonstration based on the lunar lander. However, on the other hand, we miss the strong and clearly marked theoretic contribution in section 3 to make this submission a strong theory submission.

High dimensions.

The paper's second contribution (PINN approximation of value function) is targeted for high-dimensional systems. However, neither of the two experiments provided investigates high-dimensional tasks. A systematic investigation how the proposed technique behaviors with increasing dimensionality would be indicated.

The authors could expand the experiments to various well-known MuJoCo environments, like Humanoid, Ant, and the like.

Minor aspects.

l.042: Distribution shift is a different issue than insufficient representation of areas of the state-action space in the training distribution.

Other prior work

After a quick literature research, there seems to be quite some prior work related to the proposed method. Two examples:

  • Calliess et al, "Lazily Adapted Constant Kinky Inference for nonparametric regression and model-reference adaptive control"

  • Milanese and Novara, "Set Membership identification of nonlinear systems"

We ask the author to revisit the prior work and reevaluate how their work is put into context of existing literature, also concept-wise.

On l.117 you assume that the state velocity is known exactly. But there is (Bajelani et al, 2026), apparently a successor of (Choi et al, 2025), which handles noise already and which you also cite in the appendix.

Mathematical formalities and exact language

l.087: "two-dimensional" -> It is unclear to which dimensions the authors refer. State space? Action space? Do you distinguish between state and observation space? Maybe you can formalize the problem model first.

The notation sections should be explicit from what vector space the vectors v\mathbf{v} are taken.

l.103: ℓ2\ell_2 ball --> ℓ2\ell_2-ball.

When you speak of a Lipschitz constant of a map f ⁣:X→Yf \colon X \to Y then you need to specify the metric of both, XX and YY.

On l.115: Why is ff only assumed to be Lipschitz continuous in the state?

You use two notations for a function as argument: u(⋅)u(\cdot) and uu, e.g., l.129, l.137. Stick to one

The E\mathcal{E} and u(i)\mathbf{u}^{(i)} in l.159 is not defined.

l.160: "Our approach generalizes" --> in what sense?

l.165: "conservative value function" --> could be possibly read as a conservative function in the mathematical sense, i.e., where closed integrals are zero.

Experiments

Seeding.

Experiments involving randomness shall be run a couple of times (5 or more) with different seeds, and then mean and stddev should be reported on the numbers. The paper sometimes uses only one seed, or only three seeds, mostly does not report mean and stddev, sometimes only max.

Tuning effort.

Can you comment in what terms you invested equal tuning effort to all competitors in the experimental evaluation?

Somehow related, but beyond tuning: Offline RL on lunar lander would benefit from expert data, but you only provided random data. This limits the insight from your comparison between different methods.

More baselines.

In l.272 you note that your learning process can be applied to (Choi et al, 2025). So you could design experiment and compare to (Choi et al, 2025) and hence demonstrate how you improved the prior state of the art?

Minor comments.

The experiments on the DDH variants are often called "ablation studies" in the machine learning domain.

Discussion

Limitations are insufficiently discussed.

Questions for authors

In addition to the points addressed above:

  • Out of curiosity, why use ℓ1\ell_1-loss instead of ℓ2\ell_2-loss in l.237?
Reviewer_i3UnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this submission, the authors propose a data-driven framework to estimate and solve the Hamilton-Jacobi-Bellman equation for optimal control. First, building on prior work, they present a data-driven method to learn the Hamiltonian from arbitrary, suboptimal trajectory observations, establishing a conservative upper bound on the value function. Second, they introduce an augmented loss function to learn the value function using this data-driven Hamiltonian. Experiments are conducted on a Linear Quadratic Regulator system and the Lunar Lander environment. The results suggest that the proposed method uses collected data more efficiently than traditional offline reinforcement learning baselines, achieving superior performance with fewer samples.

Critical strengths

The idea of approximating the Hamiltonian within a tractable search space is original and promising.

The theoretical background on the Hamilton-Jacobi-Bellman formulation is clearly presented.

The empirical results demonstrate that the proposed approach can efficiently learn optimal policies in an offline reinforcement learning setting.

Critical weaknesses

1. Method Applicability

Crucially, the method relies on prior knowledge of the Lipschitz constants of the dynamics with respect to both the state and control variables. However, these constants are typically unknown in real-world settings. In the Lunar Lander experiment, they are derived analytically because the exact dynamics model is available. In practical applications, it is unclear how such constants could be estimated and with what degree of accuracy. In my view, this reliance severely limits the method's applicability, and the authors should explicitly address how this limitation can be handled in practice.

2. Novelty and Soundness

The submission presents two somewhat independent contributions: an extension of the Data-Driven Hamiltonian (DDH) and a neural surrogate for the value function that utilizes this DDH. Because multiple approximations are introduced, each brings a distinct source of error. These individual error sources should be evaluated separately to understand their relative contributions to final performance.

Regarding the first contribution, the method is framed as an extension of prior work by Choi et al. However, the manuscript does not clearly delineate what was already established in prior work versus what is newly introduced. In particular, it is unclear whether Choi et al.'s method could serve as an experimental baseline, and if not, why. Furthermore, the claim in the introduction that the proposed DDH interpolates the Hamiltonian at arbitrary states requires context: the interpolation relies on sampled uncertain state points, but since no accuracy bounds are provided for this approximation, the reliability of this interpolation needs further discussion.

Secind, the neural value function is trained using an augmented loss that adds a residual term to the standard partial differential equation learning loss. To isolate the impact of each design choice, their effects should be disentangled. For example, it would be insightful to evaluate the performance of the DDH alone using a traditional grid-based solver (which should be computationally feasible on these low-dimensional tasks), and, conversely, evaluate the neural solver independently against standard solvers.

Currently, the experimental section only appears to include an ablation study on the neural value function loss component, which shows very little performance difference. Without a targeted analytical or empirical evaluation of each component, and uncertainty analysis of the approximations, it is difficult to measure the true novelty, individual contribution, and practical impact of the proposed framework.

3. Clarity

The paper suffers from several clarity issues:

Key concepts are introduced without sufficient background or definitions, such as the "adversarial dynamics" mentioned in Section 3.1. As a result, the methodological development is difficult to follow.

The experimental section relies heavily on dense tables and collections of numbers without adequate high-level takeaways, making it hard to grasp the main findings. Additionally, the various neural Hamiltonian baselines compared in the paper are not adequately introduced.

Questions for authors
  1. What does the neural value function solver yield when the Hamiltonian is known analytically? Conversely, how does the proposed framework perform when the value function can be solved exactly given the estimated Hamiltonian?
  2. What does the acronym "DDH SI" stand for?
  3. What is the technical or domain-specific motivation for using an exponential cost function in the Lunar Lander experiment?
Reviewer_gJK2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends the data-driven Hamiltonian (DDH) framework to optimal control problems with running costs. To mitigate the curse of dimensionality associated with solving the HJB PDE, the paper trains a neural value function that approximates HJB-PDE solutions. The DDH enables PDE residual evaluation at arbitrary states, providing a training signal for scaling to high-dimensional systems. The proposed method is evaluated on 2D and 6D benchmark systems.

Critical strengths

None

Critical weaknesses

The paper builds heavily upon the framework proposed in [R1] and distinguishes itself primarily through claims of broader applicability to optimal control problems with running costs and improved scalability to higher-dimensional systems. However, the evidence provided does not sufficiently substantiate these claims.

  1. It is unclear why the framework in [R1] cannot be naturally extended to accommodate running costs, making the novelty and necessity of the proposed generalization difficult to assess.

  2. The motivation and theoretical implications of upper-bounding the data-driven Hamiltonian are not adequately justified. The authors should clarify why an upper bound is preferred over a lower bound and establish how this choice affects the resulting value-function approximation and its theoretical guarantees.

  3. While the paper claims improved scalability relative to [R1], the experimental evaluation is limited to a 6D Lunar Lander benchmark and does not provide a systematic assessment of computational complexity, scalability limitations, or empirical comparisons against [R1]. As a result, the central claims regarding both generality and scalability remain insufficiently supported.

Other comments:

  1. Several equations are unnumbered, including those on lines 112 and 183. Please number equations that are referenced in the text.

  2. Equation (4) in the manuscript defines the data-driven Hamiltonian using a min–max formulation, whereas Eq. (16) in [R1] uses a max–min formulation. The authors should clarify this difference and revise the attribution to [R1] accordingly.

  3. Figure 2 is very hard to interpret.

Questions for authors
  1. Can the author please clarify how their framework of data-driven Hamiltonian based on [R1] generalizes to a broader class of optimal control problems? If the primary contribution of the proposed method is extending the framework in [R1] to settings with running costs and to higher-dimensional systems, it is unclear why the framework in [R1] cannot already be adapted to accommodate running costs.

  2. What guarantee does an upper bound on the data-driven Hamiltonian provide for the resulting value function, and why is an upper bound preferred over a lower bound in this formulation? Could you please clarify?

  3. What are the scalability limitations of the proposed approach? It’s claimed that compared to [R1], the proposed method is scalable to high-dimensional systems, but only a 6D Lunar Lander benchmark is presented without any discussion of scalability limitations beyond 6D?

  4. Can the authors please benchmark their method against the method in [R1]? Can the authors compare the safety violations observed with the proposed method, which upper-bounds the data-driven Hamiltonian, against those obtained by the method in [R1], which instead relies on lower-bounding the Hamiltonian?

References

[R1] Jason J Choi, Christopher A Strong, Koushil Sreenath, Namhoon Cho, and Claire J Tomlin. Data-driven Hamiltonian for direct construction of safe set from trajectory data. In 2025 IEEE 64th Conference on Decision and Control (CDC), pp. 2951–2958. IEEE, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.