- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper examines continuous-time reinforcement learning with irregular decision intervals. As the decision interval shrinks to zero, standard discrete-time Q-functions collapse into state-value functions, losing the ability to distinguish actions. To fix this, the authors propose a decoupled continuous-time actor-critic built on Hamiltonian flow.
Critical strengths
Learning with irregular decision times and infinitesimal intervals is a real problem in continuous-time RL. Splitting instantaneous advantage-rate estimation from value-function updates is a neat idea, and may simplify learning compared with coupled martingale-based formulations.
Critical weaknesses
which is exactly the soft Hamiltonian aggregation used in Equation (19). So Equation (18) does not actually follow from the stated infinitesimal limit on Equation (16). The upper bound in (17) does not give the required equality. If the action-space integral uses an unnormalized measure, the normalization-dependent term also needs handling.
Theorem 4.2 gives nonzero error bounds for finite holding intervals, while Theorem 4.3 claims the single-critic algorithm converges to the optimum without spelling out the joint limiting conditions. How the two results fit together needs serious clarification. These are problems with the math itself, not just the writing.
In continuous control, irregular decision intervals come from artificial sampling schemes. It's unclear whether the advantages hold up on truly event-driven decision processes.
The financial experiments are historical backtests over a single out-of-sample quarter. There's no evidence across different market regimes, and no risk-adjusted returns, maximum drawdown, or transaction-cost sensitivity. The reported ~21% quarterly return alone doesn't show reliable real-world trading performance.
Questions for authors
see weakness
Response to Reviewer u5hz
We address all of the concerns below.
1. Theoretical Soundness and Equations (16)–(18)
The review's mathematical criticism begins with an incomplete sentence (“which is exactly the soft Hamiltonian aggregation used in Equation (19)”), without presenting the preceding expression. More importantly, the subsequent objection fundamentally conflates algorithmic motivation with mathematical proof.
Section 3.2 explains the thought process motivating our algorithm. Equation (17) is explicitly presented as an overestimation, and Equation (18) explicitly uses an approximation, not an exact infinitesimal-limit identity. The purpose is to motivate the proposed Hamiltonian-driven value update, not to establish its convergence.
The objection that an upper bound does not imply equality therefore misses the purpose of the construction. Such an equality is not required to motivate a new update operator. The inequality motivates the choice of an alternative update direction, while the actual iteration is defined in Equations (21)–(22) and analyzed independently.
Indeed, the conventional soft backup yields a vanishing increment as u→0, while its rescaled derivative gives only a fixed-reference average, corresponding to fixed-policy evaluation rather than entropy-regularized action optimization. Our proposed flow instead uses the soft Hamiltonian to define the non-degenerate iterative update Vk+1=Vk+τH(α)(Vk).
The objection therefore reiterates the limitation of the original backup, reinforcing precisely why an alternative value-update mechanism like our proposed approach is needed. It does not demonstrate any failure of the proposed Hamiltonian-flow algorithm.
Crucially, Section 4 and Appendices D–F establish convergence independently of this motivational approximation:
These proofs directly analyze the proposed Hamiltonian iteration through comparison with the dynamic-programming semigroup. The review does not identify a failure in these independent convergence arguments. An objection to the motivational approximation does not establish a mathematical flaw in the proposed algorithm or its convergence guarantees.
2. Theorems 4.2 and 4.3
The claim that the joint limiting conditions are missing is contradicted by the manuscript. Theorem 4.2 explicitly states convergence as τ,u→0 with τ/u→0. Moreover, the proof of Theorem 4.3 in Appendix F invokes Theorem 4.2 and expressly concludes convergence under the corresponding joint refinement.
There is no contradiction between nonzero error bounds at finite resolution and convergence under refinement. Quantifying the finite-resolution error and showing when it vanishes is precisely the purpose of the layered analysis. Treating these statements as incompatible overlooks the stated conditions and the proof.
3. Experimental Scope
The paper evaluates a continuous-time RL algorithm under small and irregular decision intervals. It does not claim to solve all event-triggered control problems or to establish production-ready trading performance.
The irregular-time sampling protocol is explicitly specified and consistently applied across methods, with additional regular-time transfer evaluation in Appendix B. The trading experiment is an additional real-data benchmark using a chronological out-of-sample split, transaction costs, and identical market paths across methods. Broader market-regime and risk-adjusted analyses would be complementary extensions, not prerequisites for evaluating the algorithmic contribution or the comparisons presented.