- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies a Pontryagin-guided policy refinement framework for stochastic optimal control problems with state delays and control-dependent diffusion. The method has two parts: initially a history-dependent neural policy is trained using direct policy optimization, after which a the learned policy is refined by recovering the first- and second-order optimality information from simulated stochastic continuations. The key idea is to allow the required adjoint quantities to be estimated without explicitly solving the coupled delay-adjoint equations by differentiating through the delayed state dynamics while detaching future policy actions. Specifically, a current-injection duality that identifies the effective diffusion curvature with a current-state Hessian block is derived, which allows to avoid constructing the entire-history Hessian. The retrieved quantities are then used in a generalized-Hamiltonian optimization to refine the current control. From a theoretical perspective, results on discrete adjoint correspondence, consistency under mesh refinement, recovery-error bounds, and conditional objective improvement are presented. The paper applies the developed method on five benchmarks, demonstrating improvements in control accuracy and realized cost over the learned reference policy.
Critical strengths
Meaningful theoretical treatment of second-order stochastic delay-control sensitivities.
A strongest aspect of the paper is the identification of the effective diffusion curvature through current-injection duality. Despite extracting a current-state Hessian block being straightforward on a fixed discretization, establishing its correspondence with the effective curvature of the continuous-time coupled delay-adjoint family is more substantial. Furthermore, the paper analyzes consistency as the discretized history dimension grows, providing a theory-grounded way to avoid an expensive numerical adjoint computation while retaining physical delay dependencies.
A principled policy-refinement framework with useful numerical diagnostics.
The paper couples the recovered adjoint information with an anchored constrained control solve and the construction of this anchoring is particularly appealing because the diffusion-curvature correction preserves the first-order gradient at the reference action, which supports the recovery-error analysis. Such experiments span several stochastic delay-control settings and include checks of estimator accuracy, mesh dependence, curvature extraction, and diffusion-curvature ablation. Last, the reported improvements over the learned reference policy suggest that the approach can be useful in ptactical scenarios.
Critical weaknesses
Insufficient positioning and comparison with differentiable predictive-control approaches
The proposed method is closely related in spirit to differentiable predictive control (DPC) and its stochastic extensions, which exploit differentiable dynamics and trajectory rollouts for model-based policy optimization. However, the relationship to these approaches is not sufficiently discussed, particularly the distinction between direct differentiation of a predictive control objective and the proposed Pontryagin-guided recovery of adjoint quantities. Moreover, although the authors include a derivative-free Monte Carlo current-control search, a more relevant baseline would be a gradient-based one-step shooting method that directly optimizes the current control by differentiating the expected continuation cost through the same delayed stochastic simulator. Such a comparison would help establish whether the proposed adjoint-based recovery offers advantages in computational efficiency, control accuracy, or realized cost beyond standard automatic-differentiation-based predictive optimization.
Limited empirical evidence isolating the benefit of retaining delayed re-entry derivatives
The experiments suggest in a rather convincing manner that using the correct delayed continuation model can improve accuracy of the control. However, the re-entry-detached ablation in Table 15 does not show a statistically resolved improvement from retaining the delayed derivative paths in the two tested settings and this is despite the substantial differences in adjoint-estimation accuracy reported in the corresponding diagnostics. Consequently, the empirical contribution of the complete delay-sensitivity computation to deployed-policy performance remains unclear. Last, the method also requires substantial online simulation, limiting its demonstrated practical efficiency.
Questions for authors
-
Comparison with differentiable control optimization Could the authors clarify how the proposed framework relates to Differentiable Predictive Control and specifically its stochastic variant [1]? In particular, what is the conceptual and computational advantage of recovering the Pontryagin adjoints and effective diffusion curvature, rather than directly differentiating the expected predictive control objective through the stochastic delayed dynamics? A useful baseline could be for example a gradient-based one-step shooting optimizer (e.g., projected gradient descent or L-BFGS) that optimizes the current action while using the same learned reference policy for future continuation, with future control values held fixed as in the proposed sensitivity computation. Could the authors compare this approach against PGDPO under matched simulation or wall-clock budgets, reporting control accuracy, realized cost, and computation time? This would help clarify the incremental benefit of the proposed adjoint-based recovery relative to direct differentiable predictive optimization.
-
Contribution of delayed re-entry sensitivities
Table 15 reports no statistically resolved performance difference between full recovery and re-entry-detached recovery in the tested settings and this is despite the corresponding adjoint estimates differ substantially in accuracy. Can the authors explain investigate and explain this discrepancy? Are there regimes with longer delays or stronger delayed coupling in which preserving the derivative paths yields a clear improvement in deployed-control accuracy or objective value?
-
Applicability of the theoretical guarantees Can the authors clarify which localization, strong-concavity, and quadratic-growth assumptions are satisfied by the original benchmark problems? In particular, is it possible to verify the objective-improvement condition for any of the tested systems, or to obtain a guarantee for an unlocalized LQ subclass? This would strengthen the connection between the theoretical analysis and the practical algorithm.
-
Scalability to higher dimensional systems and longer histories The numerical experiments primarily consider relatively low-dimensional systems and short discretized histories, while the largest physical state dimension is (n=20). Given that a central motivation of the paper is to avoid the computational burden of full history Hessians, could the authors provide a larger-scale experiment involving higher-dimensional states and substantially longer history buffers?In particular, it would be useful to examine how computational time, memory consumption, estimation accuracy, and control performance scale with both state dimension and delay length. Such an experiment would substantiate the claimed practical scalability of the proposed current-state curvature extraction methodology beyond the (relatively) small-scale benchmarks considered.
[1] Drgoňa, Ján, et al. "Learning stochastic parametric diferentiable predictive control policies." IFAC-PapersOnLine 55.25 (2022): 121-126.
6 comments
Response to Reviewer WXRt (2/2): Applicable guarantees and scaling evidence
This part addresses the applicability and scaling questions Q3–Q4.
Unless stated otherwise, table and appendix references are to the revised manuscript.
Q3. Which guarantees apply to the benchmarks?
Several relevant properties follow directly from the models; they should be separated from the additional hypotheses for deployment guarantees. Theorem 5.1 identifies fixed-reference population sensitivities under its own assumptions. Its exact fixed-grid correspondence does not require Appendix D's localized recovery/improvement hypotheses.
Local recovery. In Appendix A's cost convention, all five supplied projected recovery quadratics are strongly convex: the effort weights are positive, drift and diffusion are affine in the current action, and the supplied curvature is nonnegative. This gives a well-defined linear/box solve. Separately, the Full/Detached/Instantaneous comparisons of revised Table 15 demonstrate the re-entry accuracy gains with finite-range adjoint-input projections disabled (Appendix B.3). This excludes calibrated clipping as the source of those observed gains; optimal-input containment remains a separate analytical requirement.
Unlocalized LQ subclass. For P1 and P2, including revised coupled P2, the stated linear controlled dynamics and , give global quadratic growth for square-integrable admissible controls. With the same initial history and Brownian motion, the state perturbation is linear in the control perturbation. Expanding about the unconstrained optimizer cancels the first variation and leaves nonnegative state-cost terms plus the effort term:
Here P2 is in total-cost units. This is a structural consequence of the original untruncated LQ model (not a separate theorem claimed in the paper or an inference from sampled trajectories). It does not by itself certify the recursively deployed finite-sample PGDPO update.
Objective improvement. Theorem 5.3 uses its own margin and defect conditions, not quadratic growth. Under its recovery and comparison assumptions, Remark D.22 provides a sufficient localized regime: with effort coefficient and Lipschitz constant of the state-objective gradient, , so supplies a positive margin, together with the stated defect bound. A numerical certificate of this condition for the benchmark runs is not supplied. Compact localization, optimal-input containment, and recursive stability (D.7) also remain separate from the LQ growth argument; Remark D.8 distinguishes transfer to the untruncated model. These qualifications concern the scope of the deployment guarantees, not the validity of the sensitivity identification or the empirical refinement results.
Q4. Higher dimension and longer histories
The completed revision provides both coupled-control and computational evidence. The P2 instances use full-matrix Riccati references and show 26.0%/24.3% control-error reductions, with paired-cost gains at both dimensions (; General Response §1). In Table 2, increasing state/control dimension fourfold from 5 to 20 increases total Stage-II evaluation time from 49.9 to 97.0 minutes, about 1.94-fold, at the same horizon, history length, and branch budgets. This is favorable measured scaling over the tested range, not an extrapolated complexity law.
The extraction saving is also measured directly. For P2 at , Table 18 reports an 8.38-fold full-buffer/current-block time ratio for continuation re-simulation and differentiation, together with the 121-fold dense-Hessian output-entry ratio and 11-fold query-direction ratio. These are component-level measurements; they are not proportional savings in total memory or online deployment time.
Remark C.3 explains the structural saving: stored curvature and current-direction queries, instead of dense curvature on buffer coordinates. The delay sweep separately reaches at , while the main run uses . The requested joint sweep at substantially larger state and history dimensions, including total memory and estimator accuracy, remains beyond the completed experiments. Propagation, graph storage, regression, and deployment retain history/horizon dependence. The present evidence therefore establishes the extraction mechanism and refinement at the tested coupled dimensions, while distinguishing them from broader end-to-end scaling.
Thank you for the detailed response and additional experiments. The stronger-coupling ablations is now demonstrating a statistically significant benefit from retaining delayed re-entry derivatives in deployed-control accuracy, and the coupled P2 results make the empirical evaluation stronger.
However, my main comparison request remains unresolved: the existing Monte Carlo search does not replace gradient-based shooting under matched continuation conditions and computational budgets. Furthermore, the objective-improvement conditions remain uncertified for the benchmark deployments, and the requested larger joint state/history scaling study is still incomplete. These gaps leave the practical advantage over direct differentiable optimization insufficiently established.
I appreciate the clearer distinction between the sensitivity-identification results and the conditional deployment guarantees. I therefore maintain my original recommendation.
Response to Reviewer WXRt (1/2): Direct optimization and re-entry evidence
Thank you for the theoretical assessment and the concrete comparison requests. Part 1 addresses Q1–Q2; part 2 addresses Q3–Q4.
Unless stated otherwise, table and appendix references are to the revised manuscript.
Q1. DPC and direct control optimization
PGDPO adds history-specific action refinement after policy learning. Stochastic DPC (Drgoňa et al., 2022) differentiates a sampled predictive objective to train policy parameters, closely related to our Stage I. Stage II freezes the actor, extracts reference-indexed sensitivities from its continuations, and uses them to refine the current action.
The completed direct-search comparison shows a control-accuracy benefit at matched forward work. Table 19 of the revised manuscript (Table 18 in the original submission) gives P1 control nRMSE 19.27% for MC search versus 5.34% for PGDPO (p<0.001); the paired cost difference is unresolved (p=0.10). The P2 rows retain the original separable configuration, separately from the new coupled-P2 experiments. This finite-candidate search tests one alternative use of the online simulations; it is not the gradient-based shooting comparison you requested.
Relative to shooting, the distinction is the organization and reuse of continuation sensitivities. PGDPO estimates first-order adjoint inputs and an n×n physical-current-state curvature block, then reuses them in an anchored quadratic linear/box solve without reevaluating the continuation for each candidate action. Current-injection duality identifies this compact curvature with the coupled delay system's effective diffusion curvature.
In the LQ tests, holding future action values fixed makes the exact continuation quadratic in the current action. PGDPO constructs a Newton-type local update from the current-state block and associated first-order inputs. With exact inputs, its anchor gradient agrees with that of the same-grid continuation objective; Table 12 shows O(h) control and curvature-coefficient discrepancies on the tested meshes. Thus the finite-mesh difference is quantified rather than left unspecified. This is an exact-input local-solve diagnostic, separate from learning and estimation errors.
Direct quadratic/Newton shooting is also available in LQ, and autograd through the physical delay graph can retain re-entry paths. The meaningful comparison is therefore control accuracy, realized cost, and total computation under matched continuation conditions, including fixed future action values. The completed MC result does not predetermine the outcome of that comparison.
Q2. Re-entry sensitivities and deployed performance
The expanded experiments resolve the control-accuracy question in stronger-delay regimes. At κ=2, Full reduces control nRMSE versus Detached by 24–25% in P1 and 12–16% in coupled P2 (p<0.001 in all four problem–delay settings; General Response §2). These findings survive Bonferroni adjustment across the eight Full–Detached control-error tests. The Full/Detached/Instantaneous comparisons in revised Table 15 use no finite-range adjoint-input projection (Appendix B.3), so this benefit does not depend on calibrated clipping ranges. Additional Full–Detached paired cost gains remain unresolved.
The earlier input/deployment discrepancy compares different objects: input diagnostics use exact-reference controls, while deployment refines learned actors and combines the input estimates jointly. Table 17 reports descriptive shared-initial-history diagnostics for P1 and the original separable P2. Full's corrections are better aligned with the oracle correction but smaller in magnitude than Detached's; both quantities affect the remaining action error. This helps distinguish input fidelity from the geometry of an action update without assigning a causal explanation to full-rollout differences.
Table 15 also gives, for coupled P2 at κ=2 and δ=0.2,0.4, mean control nRMSE (%) of 8.87/8.77 for Full, 9.25/9.83 for Instantaneous, and 10.12/10.41 for Detached. Instantaneous changes the continuation model and its derivatives together. Full–Detached is the comparison that retains the delayed forward model and isolates the re-entry derivative paths. The observed ordering is descriptive; the reported paired Full–Detached tests establish the delay-derivative benefit.