acceptodds
Under review as a conference paper at ICLR 2027

Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow

Abstract

Many real-world control problems, ranging from finance to robotics, evolve in continuous time with non-uniform, event-driven decisions. Standard discrete-time reinforcement learning (RL), based on fixed-step Bellman updates, struggles in this setting: as time gaps shrink, the -function collapses to the value function , eliminating action ranking. Existing continuous-time methods reintroduce action information via an advantage-rate function . However, they enforce optimality through complicated martingale losses or orthogonality constraints, which are sensitive to the choice of test processes. These approaches entangle and into a large, complex optimization problem that is difficult to train reliably. To address these limitations, we propose a novel decoupled continuous-time actor-critic algorithm with alternating updates: is learned from diffusion generators on , and is updated via a Hamiltonian-based value flow that remains informative under infinitesimal time steps, where standard max/softmax backups fail. Theoretically, we prove rigorous convergence via new probabilistic arguments, sidestepping the challenge that generator-based Hamiltonians lack Bellman-style contraction under the sup-norm. Empirically, our method outperforms prior continuous-time and leading discrete-time baselines across challenging continuous-control benchmarks and a real-world trading task, achieving 21% profit over a single quarternearly doubling the second-best method.

Primary area:
reinforcement learning
Keywords:
Irregular decision times, Continuous-time reinforcement learning, Hamiltonian flow, Trading, Control theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_u5hzOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines continuous-time reinforcement learning with irregular decision intervals. As the decision interval shrinks to zero, standard discrete-time Q-functions collapse into state-value functions, losing the ability to distinguish actions. To fix this, the authors propose a decoupled continuous-time actor-critic built on Hamiltonian flow.

Critical strengths

Learning with irregular decision times and infinitesimal intervals is a real problem in continuous-time RL. Splitting instantaneous advantage-rate estimation from value-function updates is a neat idea, and may simplify learning compared with coupled martingale-based formulations.

Critical weaknesses

which is exactly the soft Hamiltonian aggregation used in Equation (19). So Equation (18) does not actually follow from the stated infinitesimal limit on Equation (16). The upper bound in (17) does not give the required equality. If the action-space integral uses an unnormalized measure, the normalization-dependent term also needs handling.

Theorem 4.2 gives nonzero error bounds for finite holding intervals, while Theorem 4.3 claims the single-critic algorithm converges to the optimum without spelling out the joint limiting conditions. How the two results fit together needs serious clarification. These are problems with the math itself, not just the writing.

In continuous control, irregular decision intervals come from artificial sampling schemes. It's unclear whether the advantages hold up on truly event-driven decision processes.

The financial experiments are historical backtests over a single out-of-sample quarter. There's no evidence across different market regimes, and no risk-adjusted returns, maximum drawdown, or transaction-cost sensitivity. The reported ~21% quarterly return alone doesn't show reliable real-world trading performance.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer u5hz

We address all of the concerns below.

1. Theoretical Soundness and Equations (16)–(18)

The review's mathematical criticism begins with an incomplete sentence (“which is exactly the soft Hamiltonian aggregation used in Equation (19)”), without presenting the preceding expression. More importantly, the subsequent objection fundamentally conflates algorithmic motivation with mathematical proof.

Section 3.2 explains the thought process motivating our algorithm. Equation (17) is explicitly presented as an overestimation, and Equation (18) explicitly uses an approximation, not an exact infinitesimal-limit identity. The purpose is to motivate the proposed Hamiltonian-driven value update, not to establish its convergence.

The objection that an upper bound does not imply equality therefore misses the purpose of the construction. Such an equality is not required to motivate a new update operator. The inequality motivates the choice of an alternative update direction, while the actual iteration is defined in Equations (21)–(22) and analyzed independently.

Indeed, the conventional soft backup yields a vanishing increment as u→0u\to0, while its rescaled derivative gives only a fixed-reference average, corresponding to fixed-policy evaluation rather than entropy-regularized action optimization. Our proposed flow instead uses the soft Hamiltonian to define the non-degenerate iterative update Vk+1=Vk+τH(α)(Vk)V_{k+1}=V_k+\tau\mathcal H^{(\alpha)}(V_k).

The objection therefore reiterates the limitation of the original backup, reinforcing precisely why an alternative value-update mechanism like our proposed approach is needed. It does not demonstrate any failure of the proposed Hamiltonian-flow algorithm.

Crucially, Section 4 and Appendices D–F establish convergence independently of this motivational approximation:

  • Theorem 4.1: explicit convergence bounds for the ideal Picard-Hamiltonian iteration.
  • Theorem 4.2: finite-holding-time error bounds and convergence, including Richardson extrapolation.
  • Theorem 4.3 and Corollary 4.4: the corresponding single-critic and finite-sample guarantees.

These proofs directly analyze the proposed Hamiltonian iteration through comparison with the dynamic-programming semigroup. The review does not identify a failure in these independent convergence arguments. An objection to the motivational approximation does not establish a mathematical flaw in the proposed algorithm or its convergence guarantees.

2. Theorems 4.2 and 4.3

The claim that the joint limiting conditions are missing is contradicted by the manuscript. Theorem 4.2 explicitly states convergence as τ,u→0\tau,u \to 0 with τ/u→0\tau/u \to 0. Moreover, the proof of Theorem 4.3 in Appendix F invokes Theorem 4.2 and expressly concludes convergence under the corresponding joint refinement.

There is no contradiction between nonzero error bounds at finite resolution and convergence under refinement. Quantifying the finite-resolution error and showing when it vanishes is precisely the purpose of the layered analysis. Treating these statements as incompatible overlooks the stated conditions and the proof.

3. Experimental Scope

The paper evaluates a continuous-time RL algorithm under small and irregular decision intervals. It does not claim to solve all event-triggered control problems or to establish production-ready trading performance.

The irregular-time sampling protocol is explicitly specified and consistently applied across methods, with additional regular-time transfer evaluation in Appendix B. The trading experiment is an additional real-data benchmark using a chronological out-of-sample split, transaction costs, and identical market paths across methods. Broader market-regime and risk-adjusted analyses would be complementary extensions, not prerequisites for evaluating the algorithmic contribution or the comparisons presented.

Reviewer_MB8ROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a decoupled continuous-time reinforcement learning framework for stochastic control problems with irregular decision intervals. The paper provides theoretical convergence analyses for the proposed updates and evaluates the methods on four DeepMind Control Suite tasks and a historical financial trading environment.

Critical strengths
  1. The paper proposes a decoupled learning framework that avoids the complicated martingale-based objectives used in continuous-time RL.

  2. The proposed CT-SAC and CT-TD3 achieve better performance than several continuous-time and discrete-time baselines across multiple continuous-control benchmarks.

Critical weaknesses

1. Limited Novelty

The main contribution appears to be a straightforward decoupling of value-function and advantage-rate estimation, followed by a Hamiltonian-based value update. The advantage-rate formulation and its connection to the infinitesimal generator are already established in prior continuous-time RL literature [1]. Overall, this work appears to be highly incremental, as it is largely built upon the continuous-time RL framework established in [1], while the HJB residual-based value update is also not fundamentally new.

[1] Jia, Y., and Zhou, X. Y. (2023). q-Learning in Continuous Time. Journal of Machine Learning Research, 24(161), 1–61.

2. Theoretical Soundness

In Section 3.2, the authors introduce the following soft backup in Equation (16):

\exp\left(\frac{u q(x,a)}{\alpha}\right)da.$$ They then use Jensen's inequality in Equation (17) and claim that taking the infinitesimal limit yields the Hamiltonian flow in Equation (18): $$\lim_{u\to0}\frac{V_u(x)-V(x)}{u} \approx \alpha\log\int_{\mathcal A} \exp\left(\frac{q(x,a)}{\alpha}\right)da.$$ However, this derivation does not appear to be mathematically justified. To see this, suppose the action measure is normalized and denote it by $\mu$. A first-order Taylor expansion gives $$1+\frac{u q(x,a)}{\alpha}+O(u^2).$$ Consequently, $$\begin{aligned} \lim_{u\to0} \frac{\alpha}{u} \log\int_{\mathcal A} \exp\left(\frac{u q(x,a)}{\alpha}\right)d\mu(a) &= \lim_{u\to0} \frac{\alpha}{u} \log\left( 1+\frac{u}{\alpha}\mathbb E_\mu[q(x,a)]+O(u^2) \right)\ &= \mathbb E_\mu[q(x,a)]. \end{aligned}$$ This limit is generally not equal to the soft Hamiltonian $$\alpha\log\int_{\mathcal A} \exp\left(\frac{q(x,a)}{\alpha}\right)d\mu(a).$$ In other words, the infinitesimal limit of Equation (16) yields the average advantage rate under the reference action measure, rather than the log-sum-exp aggregation claimed in Equation (18). This is a critical concern because Equation (18) serves as the central derivation motivating the proposed Hamiltonian flow and the subsequent value updates. **3. Limited Experimental Evaluation** The experimental evaluation is relatively limited, consisting of only four continuous-control tasks and one trading environment. More importantly, the paper does not clearly explain how the trading environment is adapted to the continuous-time setting. In particular, the authors describe the action as buying or selling a bounded quantity within an industry group, but the exact action-space formulation is not specified. Even after examining the supplementary materials, it remains unclear whether the actions are discrete, continuous, or hybrid, how individual assets are selected, and how the agent's decisions affect the portfolio and subsequent transitions. Furthermore, holding times seem to be randomly sampled by the environment rather than determined by the agent or market events, making it unclear whether this setting adequately captures the event-driven continuous-time decision-making problems emphasized in the motivation. The paper does not sufficiently specify how rewards and evaluation returns are computed under irregular decision intervals. In standard DMC benchmarks, episode returns are typically reported over 1,000 control steps with per-step rewards bounded by 1. However, the proposed evaluation uses variable numbers of decision steps, and some reported returns, such as 1035.52 on Walker, exceed the conventional 1,000-point scale. **Minor Comments** Line 466: The phrase "statistical significant improvements" is grammatically incorrect and should be revised to "statistically significant improvements." Appendix C.6, Line 1053: The phrase "Here $\pi(da|x)$ is the measured induced by the policy" contains a grammatical error. "measured" should be replaced with "measure." Appendix C.6, Line 1085: "an useful lemma" should be corrected to "a useful lemma."
Questions for authors

Please refer to the major weaknesses discussed above. The reviewer would appreciate if the author could provide more details and results for experiments such as Point Mass, Reacher, Finger, or Manipulator in DMC.

1 comment
AuthorsOpenReview ↗

Response to Reviewer MB8R

We address all of the concerns below.

1. Novelty

We strongly disagree with the characterization of our algorithm as a straightforward application of existing results. This assessment overlooks the central contribution of the paper.

The HJB equation and advantage-rate definition are presented as established foundations in the Preliminaries (Section 2.3). Our novelty lies in the construction of a decoupled iterative learning procedure that replaces martingale-coupled optimization with alternating instantaneous-rate estimation and Hamiltonian-driven value updates.

This is not merely a conventional HJB residual method. Our formulation updates the value function through an iterative flow. Moreover, combining VV and qq as Q=V+qQ=V+q yields the distinctive, practical single-critic update in Equation (28). Neither that iterative construction nor the resulting update follows simply from stating the HJB equation.

Sections 3 and 4 develop and analyze these differences. The incremental characterization overlooks the actual algorithmic contribution.

2. Theoretical Soundness

The criticism fundamentally conflates algorithmic motivation with mathematical proof and overlooks the purpose of the approximation.

Section 3.2 explains the thought process motivating our algorithm. Equation (17) is explicitly identified as an overestimation, and Equation (18) explicitly uses an approximation, not an exact identity between the infinitesimal derivative and the soft Hamiltonian.

The reviewer's Taylor expansion computes the derivative of the original soft backup in Equation (16), precisely the construction whose limitations motivate our alternative approach. Under the reviewer's normalized reference measure μ\mu, this yields Eμ[qV]\mathbb E_\mu[q_V]. An iteration driven by this quantity would have fixed point Eμ[qV]=0\mathbb E_\mu[q_V]=0, corresponding to fixed-policy evaluation rather than the entropy-regularized optimality equation.

In contrast, Jensen's inequality gives Eμ[qV]≤αlog⁡Eμ[eqV/α]=sup⁡π≪μ{Eπ[qV]−αDKL(π∥μ)}\mathbb E_\mu[q_V]\leq\alpha\log\mathbb E_\mu[e^{q_V/\alpha}]=\sup_{\pi\ll\mu}\{\mathbb E_\pi[q_V]-\alpha D_{\mathrm{KL}}(\pi\|\mu)\}. The reviewer obtains a fixed-reference average, whereas the right-hand side performs entropy-regularized optimization over action distributions. Our construction deliberately uses the corresponding soft Hamiltonian to drive a non-degenerate iterative value update, Vk+1=Vk+τH(α)(Vk)V_{k+1}=V_k+\tau\mathcal H^{(\alpha)}(V_k).

In fact, the reviewer's calculation merely recovers the limitation that motivates our approach: the original soft backup produces a vanishing value increment as u→0u\to0, while its rescaled derivative yields only fixed-reference averaging, without entropy-regularized action optimization. That distinction motivates the explicitly stated overestimation in Equation (17) and approximation in Equation (18). The reviewer's calculation therefore reinforces precisely why an alternative value-update mechanism like our proposed approach is needed, rather than invalidating it.

More importantly, the convergence of our algorithm does not rely on the motivational calculation in Equation (18). Section 4 and the theoretical appendices establish, independently:

  • Theorem 4.1: convergence of the ideal Picard-Hamiltonian iteration with explicit error bounds.
  • Theorem 4.2: error and convergence guarantees for finite-holding-time approximations, including Richardson extrapolation.
  • Theorem 4.3 and Corollary 4.4: the single-critic formulation and finite-sample learning guarantees.

Treating the motivational approximation as the convergence proof is not a sound assessment of theoretical validity.

3. Experimental Evaluation

We disagree that the experiments are inadequately specified. Section 5 and Appendices B and J document physical integration steps, irregular holding-time distributions, episode lengths, training budgets, evaluation protocols, hyperparameter searches, ablations, and statistical testing over 12 seeds. Code and reproducibility materials are provided.

The selected DMC tasks test challenging locomotion. Omitting simpler tasks does not invalidate the comparisons. The trading setup uses minute-level observations, bounded trading actions, position limits, transaction costs, interval-based P&L, and a chronological out-of-sample evaluation on common market paths. Randomly varied holding intervals deliberately test robustness to irregular decision times, rather than attempting to model endogenous event-time generation.

Regarding Walker returns exceeding 1,000, the criticism applies the conventional fixed-step DMC scale to a different evaluation protocol. Table 1 explicitly specifies 1,200–2,000 decision steps per Walker episode. A return above 1,000 is therefore consistent with the reported setup.

Reviewer_SUoLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a decoupled continuous-time reinforcement learning framework that separates advantage-rate Q-learning from value-function updates, where specifically the value function is updated via a Hamiltonian-based value flow. Moreover, convergence guarantees are provided for the ideal iteration using a probabilistic analysis technique. Experiments are conducted on multiple benchmarks for evaluation.

Critical strengths

Strength

  1. The use of a Hamiltonian flow to generate the value function is an interesting design choice.
  2. Theoretical convergence analysis is provided for the value function, establishing explicit error bounds for the ideal Picard-Hamiltonian iteration.
  3. The proposed method is evaluated on both control tasks and real-world trading data.
Critical weaknesses

Weakness

  1. The comparison is insufficient, where the works mentioned in the related work such as Hua et al. (2025) [8] 和 Zhao et al. (2025) [29] should be compared in the experiments.
Questions for authors

Recent works such as Hua et al. (2025) [8] 和 Zhao et al. (2025) [29] should be compared in the experiments.

1 comment
AuthorsOpenReview ↗

Response to Reviewer SUoL

We thank the reviewer for the positive assessment of our method and theoretical contributions, as well as the suggestion regarding additional baselines.

As discussed in the Related Work (Appendix A), Hua et al. (2025) and Zhao et al. (2025) focus on diffusion-score-based extensions of existing continuous-time RL frameworks, rather than the decoupled value-learning procedure studied here. Their methodological objectives and application settings also differ from ours.

Our experiments already include the foundational continuous-time approaches most directly relevant to our contribution, namely qq-learning and CPPO, alongside strong discrete-time baselines and duration-aware adaptations. These comparisons are designed to evaluate our central methodological contribution.

We appreciate the suggestion and agree that comparisons with further application-specific extensions could provide complementary insights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.