- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates data-driven optimal control by extending the data-driven Hamiltonian of prior work (Choi et al, 2025) in terms of running and terminal costs of the state trajectory and leveraging this within the Hamilton-Jacobi-Bellman framework. Furthermore, the paper uses a physics-informed neural net approximation of the value function to cope with high-dimensional control tasks.
The paper presents itself mainly as a theory paper, but also provides two experimental validations: The first one is mainly to verify the theory by considering a simple two-dimensional linear dynamical system in standard form and comparing against LQR and other baselines. The second experiment investigates a 6-dimensional variant of the lunar lander problem.
Critical strengths
The paper is well written, clean, structured, presented and motivated. A RL paper on solid theoretical grounds has merits on its own. Also bridging classical control theoretical concepts and learning-based methods (RL, PINN) is beneficial to the field.
Critical weaknesses
Novelty
While overall the paper is interesting, looking the prior work in (Choi et al, 2025) where the DDH machinery was mainly developed, the question is raised what the contribution of this paper is relative to the prior state of the art.
To this end, I recommend to reorganize Introduction into motivation, prior art, identification of research gaps, and contribution to fill in the gaps, probably as dedicated subsections.
Section 3 is supposed to be original work of this paper. I would recommend to insert a subsection 3.x prior the current section 3.1. This section 3.x, however, is mainly already developed by (Choi et al, 2025); this submission mainly refers to their paper, but does not provide clearly marked theorems and proofs original to this submission. This, however, makes it difficult to position this paper is a theory contribution.
Positioning of the paper
Theory versus experimental.
This paper is positioned primarily a theory paper. It provides experiments too, but the first mainly serves the theory and the second is a single demonstration based on the lunar lander. However, on the other hand, we miss the strong and clearly marked theoretic contribution in section 3 to make this submission a strong theory submission.
High dimensions.
The paper's second contribution (PINN approximation of value function) is targeted for high-dimensional systems. However, neither of the two experiments provided investigates high-dimensional tasks. A systematic investigation how the proposed technique behaviors with increasing dimensionality would be indicated.
The authors could expand the experiments to various well-known MuJoCo environments, like Humanoid, Ant, and the like.
Minor aspects.
l.042: Distribution shift is a different issue than insufficient representation of areas of the state-action space in the training distribution.
Other prior work
After a quick literature research, there seems to be quite some prior work related to the proposed method. Two examples:
-
Calliess et al, "Lazily Adapted Constant Kinky Inference for nonparametric regression and model-reference adaptive control"
-
Milanese and Novara, "Set Membership identification of nonlinear systems"
We ask the author to revisit the prior work and reevaluate how their work is put into context of existing literature, also concept-wise.
On l.117 you assume that the state velocity is known exactly. But there is (Bajelani et al, 2026), apparently a successor of (Choi et al, 2025), which handles noise already and which you also cite in the appendix.
Mathematical formalities and exact language
l.087: "two-dimensional" -> It is unclear to which dimensions the authors refer. State space? Action space? Do you distinguish between state and observation space? Maybe you can formalize the problem model first.
The notation sections should be explicit from what vector space the vectors are taken.
l.103: ball --> -ball.
When you speak of a Lipschitz constant of a map then you need to specify the metric of both, and .
On l.115: Why is only assumed to be Lipschitz continuous in the state?
You use two notations for a function as argument: and , e.g., l.129, l.137. Stick to one
The and in l.159 is not defined.
l.160: "Our approach generalizes" --> in what sense?
l.165: "conservative value function" --> could be possibly read as a conservative function in the mathematical sense, i.e., where closed integrals are zero.
Experiments
Seeding.
Experiments involving randomness shall be run a couple of times (5 or more) with different seeds, and then mean and stddev should be reported on the numbers. The paper sometimes uses only one seed, or only three seeds, mostly does not report mean and stddev, sometimes only max.
Tuning effort.
Can you comment in what terms you invested equal tuning effort to all competitors in the experimental evaluation?
Somehow related, but beyond tuning: Offline RL on lunar lander would benefit from expert data, but you only provided random data. This limits the insight from your comparison between different methods.
More baselines.
In l.272 you note that your learning process can be applied to (Choi et al, 2025). So you could design experiment and compare to (Choi et al, 2025) and hence demonstrate how you improved the prior state of the art?
Minor comments.
The experiments on the DDH variants are often called "ablation studies" in the machine learning domain.
Discussion
Limitations are insufficiently discussed.
Questions for authors
In addition to the points addressed above:
- Out of curiosity, why use -loss instead of -loss in l.237?