acceptodds
Under review as a conference paper at ICLR 2027

Hamilton-Jacobi-Bellman Constrained Flow-based Distributional Q-Learning

Abstract

Flow-based distributional critics learn expressive return distributions from transition-level Bellman supervision, but this supervision constrains the local derivatives of the induced value function only indirectly. We introduce HJB-Constrained Value Flows, which augments this learning process with a stationary Hamilton–Jacobi–Bellman (HJB)-inspired differential regularizer. The noise-averaged initial velocity provides an expected-return estimate, motivated by the population flow-matching regression, without integrating the flow to its terminal time. We use this interface to motivate a local prior from an auxiliary unit-cost control problem and implement it through a single-noise stochastic surrogate. The return-flow architecture, distributional targets, and policy-extraction procedure remain unchanged. Experiments on 25 OGBench manipulation tasks and 12 D4RL Adroit tasks show improvements over the base Value Flows critic, with task-dependent gains. Five offline-to-online evaluations compare the complete training pipelines. Value-landscape and execution analyses illustrate accompanying changes in the critic and policy, while return-spread diagnostics show that the learned distributions retain nontrivial variability. These comparisons support the utility of the complete regularizer for flow-based value learning; they do not isolate its HJB-specific mechanism.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, flow matching PINN

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kRWeOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds on Value Flows, which uses a flow model to learn the distribution of future returns. It uses the flow’s initial velocity to estimate expected return, then adds a regularization term inspired by an HJB equation from a separate minimum-time control problem. The regularizer is meant to shape how the estimated value changes across nearby states and actions. The paper evaluates it on offline OGBench and D4RL tasks, plus five offline-to-online tasks. Results improve on several domain-level scores, though gains vary by task.

Critical strengths

1、The paper adds a local value-shaping signal to a distributional critic without adding a separate critic or dynamics model.

2、Evaluation covers 25 OGBench tasks, 12 D4RL Adroit tasks, and five offline-to-online tasks.

3、The paper is careful to distinguish its auxiliary HJB prior from the benchmark’s actual discounted return.

Critical weaknesses

1、The HJB connection is indirect. The prior comes from a separate minimum-time control problem, not the task’s return equation. The results show that the full regularizer can help, but do not show that the HJB structure itself is what helps.

2、The paper’s mean-return identity holds for the population flow-matching solution at the initial flow time. The training objective samples flow time uniformly, so that endpoint receives no direct supervision. The paper notes this caveat, but gives no quantitative check that the trained readout matches the actual mean return.

3、The implemented loss does more than regularize the mean value. It uses one noise sample per transition, and the paper shows that this also penalizes variation across noise-conditioned derivatives. There is no ablation showing which part drives the gains.

4、Optimizing derivatives with respect to state and action requires mixed derivatives during critic training. The paper does not report the resulting runtime or memory overhead.

Questions for authors

1、Does the training procedure directly sample the initial flow time, or rely on the network’s behavior near that endpoint? Is there a quantitative comparison between the readout and the actual mean return?

2、Are there existing ablations or sensitivity results for the individual regularizer terms and the final hyperparameters?

3、Can the authors report the training-time or memory overhead from the added state and action derivatives?

Reviewer_xth4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adds an HJB-inspired regularizer to Value Flows [1]. It uses the critic's initial velocity to form a noise-conditioned scalar readout, then constrains its state-directional derivative and action-gradient norm. The regularizer is motivated by an auxiliary minimum-time control problem; the original distributional losses and policy extraction remain unchanged. Experiments cover offline manipulation and Adroit tasks, offline-to-online fine-tuning, and analyses of the learned critic and policy.

[1] Perry Dong, Chongyi Zheng, Chelsea Finn, Dorsa Sadigh, and Benjamin Eysenbach. Value Flows. In ICLR 2026. See Proposition 3.

Critical strengths
  1. The method is simple to integrate. It uses the existing critic and offline transitions without adding a scalar critic or dynamics model. The auxiliary readout needs no flow integration, although other training computations still do. I also appreciate the distinction in Appendix A.3 between a constraint on the mean value and the noise-conditioned loss actually optimized.

  2. The results are encouraging across tasks. The reported offline peak scores improve over Value Flows on multiple OGBench manipulation and Adroit tasks. The improvements extend beyond the largest individual gain. Although this does not establish statistical significance on each task, it gives the approach empirical value.

Critical weaknesses
  1. The methodological novelty seems limited. The initial-velocity readout comes from [1], and HJB-inspired critic regularization has been studied in [2, 3]. In particular, the HJB-HIQL baseline in [3] uses the same state-directional residual form as Eq. (16) with M = 0. The main addition is applying a coupled state/action residual to a noise-conditioned flow readout. This is a reasonable extension, but I am not yet convinced that it offers an advantage over simpler alternatives. A comparison with M = 0 on the same VF backbone, keeping the readout and weighting fixed, would help clarify this. The paper should also discuss [2].

  2. The reported gains are difficult to assess from peak scores alone. On scene-play task 4, Table 4 reports an improvement from 7 ± 17 to 70 ± 12, but the regularized method's mean curve in Figure 6 falls back to around 10% by the end of training (visual estimate). I would like to understand how checkpoints are selected and whether the improvement remains at a common training endpoint. The puzzle-3x3 statistics also need clarification: Table 1 reports 93 ± 14, while Table 4 gives four scores of 100 ± 0 and one of 65 ± 0. The retained task-5 curve in Figure 7 approaches 100%, but two runs without retained logs are excluded from the plot. These summaries are difficult to reconcile without knowing the aggregation procedure and which runs enter the tables.

[2] Yun-Hsuan Lien, Ping-Chun Hsieh, Tzu-Mao Li, and Yu-Shuen Wang. Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning. In ICML 2024.

[3] Vittorio Giammarino, Ruiqi Ni, and Ahmed Qureshi. Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning. In NeurIPS 2025.

Questions for authors
  1. How does the method differ from [2, 3], and do you have a comparison with M = 0 or a related regularizer on the same VF backbone?

  2. How are peak checkpoints selected and scores aggregated across seeds? Is the protocol shared with VF, and does the scene-play task 4 gain remain at the final checkpoint or over the last evaluation window?

  3. What does ± denote, and how are tasks and seeds aggregated? How do the puzzle-3x3 tables relate to the retained curves, and are the runs without retained logs included in the tables?

Reviewer_4A4hOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HJB constrained distributional RL framework, based on Value Flow, which combines Value flow (for distributional RL) with a HJB inspired regularization term. The method use tweedie's formula at t=0 to infer the scalar expected return (Q) and impose the HJB constraints on Q function. In practice, this has been greatly improved Value Flow critics accuracy in both offline RL and offline-to-online learning.

The method preserves the distributional Bellman losses, flow architecture, and policy extraction mechanism of Value Flows, while integrating the HJB regularization. The Experiments conducted on both OGBench and D4RL Adroit tasks, and five offline-to-online tasks report improvements over Value Flows on several benchmarks.

Critical strengths
  1. The strengths are really the improvement over Value Flow in the performance level. It is clear that the HJB regularization is doing the works and vastly improve he accuracy in many OGBench and D4RL tasks. This implies that the differential regularization by HJB is really beneficial for Offline RL.

  2. Another benefit of the work is the simple integration with any flow critics methods. For example, we can add HJB regularization on existing methods from distributional critic. The HJB-inspired loss can be incorporated into the existing Value Flows architecture or even other broader flow critics methods without modifying its distributional Bellman targets or policy extraction procedure.

Critical weaknesses
  1. The novelty of the methodology is quite limited from my view. More specifically, I think the major contribution is the combinations of prior HJB-based Q learning and flow critics with tweedie's formula approximation. However, my major concerns are following:
  1. Using HJB equations to characterize Q-functions and guide Q-learning is well established. Kim and Yang (L4DC 2020) and Kim et al. (JMLR 2021) developed HJB formulations for continuous-time Q-learning, including action-dependent HJB equations with bounded action rates. 2) Related offline RL and HJB equation combinations: Lien et al. (ICML 2024), Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning, already proposed combining bellman equation with HJB-inspired first-order state–action consistency constraints to improve offline value estimation. This work is particularly close in motivation and methodology but is not discussed in the paper. Although the specific residuals differ, the general idea of supplementing offline Bellman learning with HJB-derived value-gradient supervision has already been explored.
  1. The initial-velocity estimator of the expected return is inherited from Value Flows (Dong et al., ICLR 2026). The submitted method then applies the HJB-inspired residual to the state and action derivatives of this scalar readout. Despite being presented as flow-based distributional Q-learning, the proposed HJB equation is defined on the scalar expected-return function, not the return distribution. Basically, it applied on the first order projected mean or expectation from the flow model but it does not utilize the distributional information of the value flow flow critics. Overall, I find the methodological novelty insufficiently distinguished from established Q-learning and first-order value regularization methods that have been developed in the literature. The flow matching component is not as strong as well due to the first order flow approximation of expectation or Tweedie's formula is a very standard techniques for mean extraction. It fails to integrate the HJB constraints in the distributional level.

  2. Insufficient theoretical justification and experimental evidence for the proposed HJB mechanism. There is also a discrepancy between the analytical HJB constraint and its implementation. The analytical formulation is imposed on the noise-averaged expected return, whereas the implemented objective penalizes a nonlinear HJB residual separately for individual noise-conditioned initial-velocity readouts. As acknowledged in Appendix A.3, this introduces additional effects involving residual variance and a Jensen gap. These effects are not equivalent to enforcing HJB on the mean Q-function and lack a clear distributional-control interpretation.

  3. I think another limitation is the lack of comparison with the HJB regularized methods, so it is unclear whether the advantage comes from the HJB or value flow. There are no comparisons against generic state/action gradient regularization, mean-Q HJB regularization, or prior first-order consistency methods such as Lien et al. (2024). The reported improvements may therefore arise from generic derivative regularization or stabilization of the initial-velocity readout rather than from the proposed HJB structure. Furthermore, no direct evidence is provided that the return distributions are estimated more accurately or calibrated better by applying HJB on value flow.

Reference

  • Lien et al., ICML 2024 — Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning
  • Kim et al., JMLR 2021 — Hamilton-Jacobi Deep Q-Learning
  • Kim and Yang, L4DC 2020 — HJB Equations for Q-Learning in Continuous Time
  • Wiltzer et al., ICML 2022 — Distributional HJB Equations for Continuous-Time RL
Questions for authors
  1. Novelty relative to existing HJB regularization for offline RL. How does the proposed approach fundamentally differ from the first-order state–action consistency framework of Lien et al. (ICML 2024)? Both methods incorporate HJB-inspired differential supervision into offline Bellman learning. Can the authors clarify the methodological distinction? An experiment comparison may not be feasible on the full OGBench but some illustration could be nice.

  2. Distributional advantage. The proposed analytical HJB constraint depends only on the scalar expected-return Q function. What advantage does the distributional return-flow representation provide for this constraint compared with an ordinary scalar Q-network? In particular, can the authors demonstrate that the single-noise formulation exploits distributional information beyond mean-value regularization?

  3. Justification and ablation of the HJB constraint. The auxiliary HJB equation assumes unit running cost, no discounting, and bounded action-change rates, which generally differ from the benchmark control problems. Why should this equation hold approximately for the learned discounted Q-function? Can the authors clarify the proposed objective against generic derivative regularization and an HJB penalty applied directly to the noise-averaged Q-function? Again, full OGBench and D4RL run may not be feasible during the time so some clarifications or toy examples could be nice to have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.