acceptodds
Under review as a conference paper at ICLR 2027

Differentiating Through Delay: Stochastic Control Without Full History Hessians

Abstract

In stochastic delay control, the current state re-enters future dynamics and costs through the history. The stochastic maximum principle represents these effects through anticipated first-order adjoints and, with control-dependent diffusion, a coupled second-order family carrying current–history cross-curvature. We recover the information needed for constrained control refinement directly from a learned history policy's simulated continuations. Future control values are detached under state differentiation, while all physical delay paths remain active. Current-injection duality identifies the required diffusion curvature with an fixed-control Hessian block, eliminating full history-Hessian assembly and a separate numerical solution of the coupled adjoint family. The stored curvature is ; trajectory propagation retains its history dependence. We establish exact discrete adjoint correspondence and convergence of the population recovery inputs under the stated refinement assumptions. Building on Pontryagin-guided direct policy optimization (PGDPO), an anchored local solve then refines the control, with localized error bounds and sufficient conditions for objective improvement. Across five benchmarks, refinement reduces its learned reference policy's control error by 24–91 percent where optimizer references exist and significantly reduces paired cost on two constrained applications. Delay-model and mesh diagnostics test the computation separately from these performance comparisons.

Primary area:
optimization
Keywords:
stochastic delay control, stochastic maximum principle, control-dependent diffusion, Pontryagin-guided policy optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WXRtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies a Pontryagin-guided policy refinement framework for stochastic optimal control problems with state delays and control-dependent diffusion. The method has two parts: initially a history-dependent neural policy is trained using direct policy optimization, after which a the learned policy is refined by recovering the first- and second-order optimality information from simulated stochastic continuations. The key idea is to allow the required adjoint quantities to be estimated without explicitly solving the coupled delay-adjoint equations by differentiating through the delayed state dynamics while detaching future policy actions. Specifically, a current-injection duality that identifies the effective diffusion curvature with a current-state Hessian block is derived, which allows to avoid constructing the entire-history Hessian. The retrieved quantities are then used in a generalized-Hamiltonian optimization to refine the current control. From a theoretical perspective, results on discrete adjoint correspondence, consistency under mesh refinement, recovery-error bounds, and conditional objective improvement are presented. The paper applies the developed method on five benchmarks, demonstrating improvements in control accuracy and realized cost over the learned reference policy.

Critical strengths

Meaningful theoretical treatment of second-order stochastic delay-control sensitivities.

A strongest aspect of the paper is the identification of the effective diffusion curvature through current-injection duality. Despite extracting a current-state Hessian block being straightforward on a fixed discretization, establishing its correspondence with the effective curvature of the continuous-time coupled delay-adjoint family is more substantial. Furthermore, the paper analyzes consistency as the discretized history dimension grows, providing a theory-grounded way to avoid an expensive numerical adjoint computation while retaining physical delay dependencies.

A principled policy-refinement framework with useful numerical diagnostics.

The paper couples the recovered adjoint information with an anchored constrained control solve and the construction of this anchoring is particularly appealing because the diffusion-curvature correction preserves the first-order gradient at the reference action, which supports the recovery-error analysis. Such experiments span several stochastic delay-control settings and include checks of estimator accuracy, mesh dependence, curvature extraction, and diffusion-curvature ablation. Last, the reported improvements over the learned reference policy suggest that the approach can be useful in ptactical scenarios.

Critical weaknesses

Insufficient positioning and comparison with differentiable predictive-control approaches

The proposed method is closely related in spirit to differentiable predictive control (DPC) and its stochastic extensions, which exploit differentiable dynamics and trajectory rollouts for model-based policy optimization. However, the relationship to these approaches is not sufficiently discussed, particularly the distinction between direct differentiation of a predictive control objective and the proposed Pontryagin-guided recovery of adjoint quantities. Moreover, although the authors include a derivative-free Monte Carlo current-control search, a more relevant baseline would be a gradient-based one-step shooting method that directly optimizes the current control by differentiating the expected continuation cost through the same delayed stochastic simulator. Such a comparison would help establish whether the proposed adjoint-based recovery offers advantages in computational efficiency, control accuracy, or realized cost beyond standard automatic-differentiation-based predictive optimization.

Limited empirical evidence isolating the benefit of retaining delayed re-entry derivatives

The experiments suggest in a rather convincing manner that using the correct delayed continuation model can improve accuracy of the control. However, the re-entry-detached ablation in Table 15 does not show a statistically resolved improvement from retaining the delayed derivative paths in the two tested settings and this is despite the substantial differences in adjoint-estimation accuracy reported in the corresponding diagnostics. Consequently, the empirical contribution of the complete delay-sensitivity computation to deployed-policy performance remains unclear. Last, the method also requires substantial online simulation, limiting its demonstrated practical efficiency.

Questions for authors
  1. Comparison with differentiable control optimization Could the authors clarify how the proposed framework relates to Differentiable Predictive Control and specifically its stochastic variant [1]? In particular, what is the conceptual and computational advantage of recovering the Pontryagin adjoints and effective diffusion curvature, rather than directly differentiating the expected predictive control objective through the stochastic delayed dynamics? A useful baseline could be for example a gradient-based one-step shooting optimizer (e.g., projected gradient descent or L-BFGS) that optimizes the current action while using the same learned reference policy for future continuation, with future control values held fixed as in the proposed sensitivity computation. Could the authors compare this approach against PGDPO under matched simulation or wall-clock budgets, reporting control accuracy, realized cost, and computation time? This would help clarify the incremental benefit of the proposed adjoint-based recovery relative to direct differentiable predictive optimization.

  2. Contribution of delayed re-entry sensitivities

Table 15 reports no statistically resolved performance difference between full recovery and re-entry-detached recovery in the tested settings and this is despite the corresponding adjoint estimates differ substantially in accuracy. Can the authors explain investigate and explain this discrepancy? Are there regimes with longer delays or stronger delayed coupling in which preserving the derivative paths yields a clear improvement in deployed-control accuracy or objective value?

  1. Applicability of the theoretical guarantees Can the authors clarify which localization, strong-concavity, and quadratic-growth assumptions are satisfied by the original benchmark problems? In particular, is it possible to verify the objective-improvement condition for any of the tested systems, or to obtain a guarantee for an unlocalized LQ subclass? This would strengthen the connection between the theoretical analysis and the practical algorithm.

  2. Scalability to higher dimensional systems and longer histories The numerical experiments primarily consider relatively low-dimensional systems and short discretized histories, while the largest physical state dimension is (n=20). Given that a central motivation of the paper is to avoid the computational burden of full history Hessians, could the authors provide a larger-scale experiment involving higher-dimensional states and substantially longer history buffers?In particular, it would be useful to examine how computational time, memory consumption, estimation accuracy, and control performance scale with both state dimension and delay length. Such an experiment would substantiate the claimed practical scalability of the proposed current-state curvature extraction methodology beyond the (relatively) small-scale benchmarks considered.

[1] Drgoňa, Ján, et al. "Learning stochastic parametric diferentiable predictive control policies." IFAC-PapersOnLine 55.25 (2022): 121-126.

6 comments
AuthorsOpenReview ↗

Response to Reviewer WXRt (1/2): Direct optimization and re-entry evidence

Thank you for the theoretical assessment and the concrete comparison requests. Part 1 addresses Q1–Q2; part 2 addresses Q3–Q4.

Unless stated otherwise, table and appendix references are to the revised manuscript.

Q1. DPC and direct control optimization

PGDPO adds history-specific action refinement after policy learning. Stochastic DPC (Drgoňa et al., 2022) differentiates a sampled predictive objective to train policy parameters, closely related to our Stage I. Stage II freezes the actor, extracts reference-indexed sensitivities from its continuations, and uses them to refine the current action.

The completed direct-search comparison shows a control-accuracy benefit at matched forward work. Table 19 of the revised manuscript (Table 18 in the original submission) gives P1 control nRMSE 19.27% for MC search versus 5.34% for PGDPO (p<0.001p<0.001); the paired cost difference is unresolved (p=0.10p=0.10). The P2 rows retain the original separable configuration, separately from the new coupled-P2 experiments. This finite-candidate search tests one alternative use of the online simulations; it is not the gradient-based shooting comparison you requested.

Relative to shooting, the distinction is the organization and reuse of continuation sensitivities. PGDPO estimates first-order adjoint inputs and an n×nn\times n physical-current-state curvature block, then reuses them in an anchored quadratic linear/box solve without reevaluating the continuation for each candidate action. Current-injection duality identifies this compact curvature with the coupled delay system's effective diffusion curvature.

In the LQ tests, holding future action values fixed makes the exact continuation quadratic in the current action. PGDPO constructs a Newton-type local update from the current-state block and associated first-order inputs. With exact inputs, its anchor gradient agrees with that of the same-grid continuation objective; Table 12 shows O(h)O(h) control and curvature-coefficient discrepancies on the tested meshes. Thus the finite-mesh difference is quantified rather than left unspecified. This is an exact-input local-solve diagnostic, separate from learning and estimation errors.

Direct quadratic/Newton shooting is also available in LQ, and autograd through the physical delay graph can retain re-entry paths. The meaningful comparison is therefore control accuracy, realized cost, and total computation under matched continuation conditions, including fixed future action values. The completed MC result does not predetermine the outcome of that comparison.

Q2. Re-entry sensitivities and deployed performance

The expanded experiments resolve the control-accuracy question in stronger-delay regimes. At κ=2\kappa=2, Full reduces control nRMSE versus Detached by 24–25% in P1 and 12–16% in coupled P2 (p<0.001p<0.001 in all four problem–delay settings; General Response §2). These findings survive Bonferroni adjustment across the eight Full–Detached control-error tests. The Full/Detached/Instantaneous comparisons in revised Table 15 use no finite-range adjoint-input projection (Appendix B.3), so this benefit does not depend on calibrated clipping ranges. Additional Full–Detached paired cost gains remain unresolved.

The earlier input/deployment discrepancy compares different objects: input diagnostics use exact-reference controls, while deployment refines learned actors and combines the input estimates jointly. Table 17 reports descriptive shared-initial-history diagnostics for P1 and the original separable P2. Full's corrections are better aligned with the oracle correction but smaller in magnitude than Detached's; both quantities affect the remaining action error. This helps distinguish input fidelity from the geometry of an action update without assigning a causal explanation to full-rollout differences.

Table 15 also gives, for coupled P2 at κ=2\kappa=2 and δ=0.2,0.4\delta=0.2,0.4, mean control nRMSE (%) of 8.87/8.77 for Full, 9.25/9.83 for Instantaneous, and 10.12/10.41 for Detached. Instantaneous changes the continuation model and its derivatives together. Full–Detached is the comparison that retains the delayed forward model and isolates the re-entry derivative paths. The observed ordering is descriptive; the reported paired Full–Detached tests establish the delay-derivative benefit.

AuthorsOpenReview ↗

Response to Reviewer WXRt (2/2): Applicable guarantees and scaling evidence

This part addresses the applicability and scaling questions Q3–Q4.

Unless stated otherwise, table and appendix references are to the revised manuscript.

Q3. Which guarantees apply to the benchmarks?

Several relevant properties follow directly from the models; they should be separated from the additional hypotheses for deployment guarantees. Theorem 5.1 identifies fixed-reference population sensitivities under its own assumptions. Its exact fixed-grid correspondence does not require Appendix D's localized recovery/improvement hypotheses.

Local recovery. In Appendix A's cost convention, all five supplied projected recovery quadratics are strongly convex: the effort weights are positive, drift and diffusion are affine in the current action, and the supplied curvature is nonnegative. This gives a well-defined linear/box solve. Separately, the Full/Detached/Instantaneous comparisons of revised Table 15 demonstrate the re-entry accuracy gains with finite-range adjoint-input projections disabled (Appendix B.3). This excludes calibrated clipping as the source of those observed gains; optimal-input containment remains a separate analytical requirement.

Unlocalized LQ subclass. For P1 and P2, including revised coupled P2, the stated linear controlled dynamics and Q,QT⪰0Q,Q_T\succeq0, R≻0R\succ0 give global quadratic growth for square-integrable admissible controls. With the same initial history and Brownian motion, the state perturbation is linear in the control perturbation. Expanding about the unconstrained optimizer cancels the first variation and leaves nonnegative state-cost terms plus the effort term:

J(u)−J(u⋆)≥λmin⁡(R)2∥u−u⋆∥H22.J(u)-J(u^{\star})\ge\frac{\lambda_{\min}(R)}{2}\left\lVert u-u^{\star}\right\rVert_{\mathcal{H}^2}^2.

Here P2 is in total-cost units. This is a structural consequence of the original untruncated LQ model (not a separate theorem claimed in the paper or an inference from sampled trajectories). It does not by itself certify the recursively deployed finite-sample PGDPO update.

Objective improvement. Theorem 5.3 uses its own margin and defect conditions, not quadratic growth. Under its recovery and comparison assumptions, Remark D.22 provides a sufficient localized regime: with effort coefficient rr and Lipschitz constant LstateL_{\mathrm{state}} of the state-objective gradient, μ−CJ≥(r−Lstate)/2\mu-C_J\ge(r-L_{\mathrm{state}})/2, so r>Lstater>L_{\mathrm{state}} supplies a positive margin, together with the stated defect bound. A numerical certificate of this condition for the benchmark runs is not supplied. Compact localization, optimal-input containment, and recursive stability (D.7) also remain separate from the LQ growth argument; Remark D.8 distinguishes transfer to the untruncated model. These qualifications concern the scope of the deployment guarantees, not the validity of the sensitivity identification or the empirical refinement results.

Q4. Higher dimension and longer histories

The completed revision provides both coupled-control and computational evidence. The n=5,20n=5,20 P2 instances use full-matrix Riccati references and show 26.0%/24.3% control-error reductions, with paired-cost gains at both dimensions (p<0.001p<0.001; General Response §1). In Table 2, increasing state/control dimension fourfold from 5 to 20 increases total Stage-II evaluation time from 49.9 to 97.0 minutes, about 1.94-fold, at the same horizon, history length, and branch budgets. This is favorable measured scaling over the tested range, not an extrapolated complexity law.

The extraction saving is also measured directly. For P2 at n=5n=5, Table 18 reports an 8.38-fold full-buffer/current-block time ratio for continuation re-simulation and differentiation, together with the 121-fold dense-Hessian output-entry ratio and 11-fold query-direction ratio. These are component-level measurements; they are not proportional savings in total memory or online deployment time.

Remark C.3 explains the structural saving: O(n2)O(n^2) stored curvature and nn current-direction queries, instead of dense curvature on n(Hh+1)n(H_h+1) buffer coordinates. The delay sweep separately reaches Hh=20H_h=20 at h=0.02h=0.02, while the n=20n=20 main run uses Hh=10H_h=10. The requested joint sweep at substantially larger state and history dimensions, including total memory and estimator accuracy, remains beyond the completed experiments. Propagation, graph storage, regression, and deployment retain history/horizon dependence. The present evidence therefore establishes the extraction mechanism and refinement at the tested coupled dimensions, while distinguishing them from broader end-to-end scaling.

Thank you for the detailed response and additional experiments. The stronger-coupling ablations is now demonstrating a statistically significant benefit from retaining delayed re-entry derivatives in deployed-control accuracy, and the coupled P2 results make the empirical evaluation stronger.

However, my main comparison request remains unresolved: the existing Monte Carlo search does not replace gradient-based shooting under matched continuation conditions and computational budgets. Furthermore, the objective-improvement conditions remain uncertified for the benchmark deployments, and the requested larger joint state/history scaling study is still incomplete. These gaps leave the practical advantage over direct differentiable optimization insufficiently established.

I appreciate the clearer distinction between the sensitivity-identification results and the conditional deployment guarantees. I therefore maintain my original recommendation.

Reviewer_z32fOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends Pontryagin-guided direct policy optimization (PGDPO) to stochastic control with state delay and control-dependent diffusion. Stage I trains an LSTM history policy by backpropagation through time. Stage II refines the control at each decision time:

  1. Simulate many future branches from the current history buffer with the frozen actor, holding future controls fixed under differentiation.
  2. Differentiate the continuation payoff with respect to the current state only, giving the first-order adjoint p and an n×n curvature block Π.
  3. Estimate the shifted input ζ by nested regression.
  4. Maximize a generalized Hamiltonian, here a quadratic or box-constrained problem, to obtain the refined control.
Critical strengths
  1. Rigorous connection between automatic differentiation and delay SMP theory. Identifying the effective diffusion curvature of the coupled second-order delay adjoint family with a current-state Hessian block is a non-trivial and potentially useful result, and the appendix proofs appear careful.
  2. Careful and honest evaluation. The paper uses paired evaluation paths, five seeds, joint seed/path bootstrap tests, exact references where available, and several ablations (instantaneous memory, re-entry-detached, diffusion-curvature off, Monte Carlo search). Non-significant results are reported openly.
Critical weaknesses
  1. Incremental over prior PGDPO work. The two-stage learn-then-recover pipeline, the anchored generalized Hamiltonian, and the nested regression for ζ all come from earlier PGDPO papers (Huh et al., 2025; Jeon et al., 2025; 2026). The paper itself acknowledges that the fixed-mesh current-block identity is just the chain rule. The new part is mainly the continuous-time identification, which is a technical extension rather than a new method.

  2. The delay-specific contribution is not supported empirically.

    • The re-entry-detached variant, which drops the delayed re-entry derivatives that motivate the paper, is statistically indistinguishable from full PGDPO in both control error (p = 0.55, 0.13) and cost (p = 0.57, 0.34). Its mean cost is even slightly lower (Table 15).
    • Against the instantaneous-memory variant, cost differences are never resolved.
    • The headline claim, that retaining delayed effects is needed, therefore rests only on input-accuracy diagnostics, not on deployed performance.
  3. Small practical gains. Cost improvements over the Stage-I actor are not significant on P1 (p = 0.06) or P3 (p = 0.30), are 0.24% on P4, and are small on P2. The large control-error reductions mostly do not translate into meaningful objective improvements.

  4. High online cost. Every decision requires 2,048 branch rollouts to the horizon plus a nested regression. Per-trajectory cost scales as O(N_h²). Stage II time often exceeds Stage I training time (e.g., P5: 91.6 vs 19.1 min), and batch-one latency (up to ~1.8 s) far exceeds the 0.02 s control step. The obvious compute-matched baseline, spending the same budget on longer or larger actor training, is missing. The Monte Carlo search comparison covers only a one-round candidate search.

  5. Toy-scale benchmarks.

    • Most problems have state dimension 1–2, 50 time steps, and a history length of 10.
    • The only higher-dimensional case (P2, n = 20) decouples exactly into independent scalar modes.
    • The method requires known differentiable dynamics.
    • Scalability claims about growing history dimension are therefore untested in practice.
  6. Heavily conditional theory. The guarantees rely on:

    • well-posedness of the delay adjoint families;
    • uniform strong concavity over compact projection sets that contain the optimal inputs (in practice these sets were calibrated empirically from preliminary runs);
    • quadratic growth, and the margin µ > C_J for improvement, which cannot be checked;
    • an extra unverified assumption (D.7) for recursive deployment.

    Consistency holds only in sequential limits. The bounds are local and offer little practical guidance.

  7. Weak baselines. The equation-based baselines (PDGM, DeepSMP) are often worse than the Stage-I actor alone. There is no comparison to standard history-based RL or to trajectory optimization / MPC, a natural competitor when the dynamics are known and online simulation is already used.

  8. Presentation and venue fit. The main text is extremely dense, with heavy notation and many hedged statements, and depends on a 60-page appendix. The contribution reads more like a stochastic-control paper (e.g., for SICON) than one accessible to a general ICLR audience.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer z32f (1/2): Contribution and completed evidence

Thank you for recognizing the connection between automatic differentiation and delay SMP. The headings group the concerns in your review; part 1 addresses W1–W4 and part 2 W5–W8.

Unless stated otherwise, table and appendix references are to the revised manuscript.

W1. Contribution beyond prior PGDPO

The new result identifies which compact continuation sensitivities supply the coupled delay system's effective diffusion curvature. Current-injection duality connects that curvature to an n×nn\times n fixed-control current-state Hessian block. Theorem 5.1 establishes exact discrete-adjoint correspondence and, under its refinement assumptions, consistency as the history dimension grows. The fixed-grid extraction is a chain-rule identity; the substantive identification is why its limit supplies the delay optimality system's required input. This enables refinement without a separate coupled-adjoint solver or full history-Hessian assembly.

Table 18 reports a 121-fold full/current dense-Hessian entry ratio, an 11-fold query-direction ratio, and an 8.38-fold time ratio for P2's measured continuation re-simulation/differentiation component. These quantify the extraction benefit, not total-pipeline speed or memory savings. We retain the earlier learning-and-refinement architecture while resolving its additional delay-specific sensitivity problem.

W2. Benefit of delayed re-entry derivatives

The new evidence concerns deployed actions, not only adjoint inputs. At κ=2\kappa=2, Full lowers control nRMSE versus Detached by 24–25% in P1 and 12–16% in coupled P2 (p<0.001p<0.001 in all four settings; General Response §2). The forward continuation model is unchanged, and both variants use the same actors and branch budgets. The four findings survive Bonferroni adjustment across the eight Full–Detached control-error tests. The additional cost differences remain unresolved: the supported delay-specific benefit is control accuracy in these stronger-coupling regimes.

W3. Control accuracy and objective performance

The revised coupled P2 improves both control accuracy and paired cost. The control-error reductions are 26.0% and 24.3% at n=5,20n=5,20, and both direct PGDPO–Stage I cost comparisons have p<0.001p<0.001. More generally, control fidelity and objective value measure different properties. At n=20n=20, Stage I has 15.74% control nRMSE despite an unresolved cost gap to the exact reference (p=0.58p=0.58); nonsignificance neither establishes cost equivalence nor certifies accurate actions.

For unconstrained LQ control, cost varies quadratically around its optimum, so substantial relative action-error reductions can coexist with smaller cost changes. The reported nRMSE evaluates reference actions at each method's visited histories; it is not identified with the objective gap. Constantinides (1986) and Mysore et al. (2021) also motivate evaluating decision behavior alongside aggregate performance. Our measured claim is improved reference-action fidelity, not their unmeasured application-specific outcomes.

W4. Online cost and alternative compute allocation

The completed comparisons isolate the contribution of online refinement and of its derivative paths. They answer a different question from allocating a fixed total budget between more actor training and test-time computation. Table 19 of the revised manuscript (Table 18 in the original submission) additionally compares one-round MC search at matched forward work: P1 control nRMSE is 19.27% for search and 5.34% for PGDPO (p<0.001p<0.001), with unresolved paired cost difference (p=0.10p=0.10).

Tables 2 and 10 report the online burden explicitly. P5 batch-one calls exceed the 0.02 s model step, so the present implementation demonstrates simulated closed-loop refinement. The total Stage-II time covers 256 evaluation paths and depends on deployment volume; the rollout work retains its quadratic horizon dependence. A compute-matched longer/larger-actor study is still needed to rank the alternative total-budget allocations. That open comparison does not negate the demonstrated same-actor refinement or the isolated re-entry benefit.

References: Constantinides, G. M. (1986). Capital Market Equilibrium with Transaction Costs. Journal of Political Economy 94(4), 842–862. Mysore, S., et al. (2021). Regularizing Action Policies for Smooth Control with Reinforcement Learning. ICRA.

AuthorsOpenReview ↗

Response to Reviewer z32f (2/2): Theory, scaling, and learning relevance

This part addresses W5–W8.

Unless stated otherwise, table and appendix references are to the revised manuscript.

W5. Nonseparable P2 and scaling evidence

The revised n=5,20n=5,20 problems remove the common eigenbasis and use full-matrix Riccati references. Control nRMSE falls by 26.0% and 24.3%, with paired-cost gains at both dimensions (p<0.001p<0.001; General Response §1). Table 2 also shows that a fourfold increase in state/control dimension raises total Stage-II evaluation time from 49.9 to 97.0 minutes, about 1.94-fold, at the common horizon, history length, and branch budgets. This measures the tested range, not asymptotic complexity.

The delay sweep separately reaches Hh=20H_h=20, while the n=20n=20 main case has Hh=10H_h=10. A larger joint state/history sweep remains beyond the completed experiments. This supports extraction and refinement on the tested coupled systems; known differentiable dynamics and history-dependent propagation remain required.

W6. Theoretical assumptions, projection, and sequential limits

The core sensitivity result and the recovery guarantees have distinct hypotheses. Theorem 5.1's fixed-grid correspondence does not require the localized recovery or improvement assumptions in Appendix D. Its population mesh-consistency statement has its own regularity and integrability conditions. Appendix D then adds conditions for policy-error transfer and objective improvement; these are not prerequisites for the discrete adjoint identification.

The Table 15 Full/Detached/Instantaneous comparisons do not rely on calibrated finite-range adjoint projections. For those comparisons, Appendix B.3 explicitly disables the finite-range input projections, yet reports the stronger-coupling control-accuracy gains. In the projected implementation, positive effort weights and nonnegative supplied curvature make the local quadratics strongly convex. This addresses clipping-driven improvement; optimal-input containment remains a separate theoretical condition.

For P1–P2, linear dynamics and positive quadratic control cost also yield global quadratic growth of the unlocalized LQ objective, as detailed in our WXRt Q3 response. Theorem 5.3 instead uses its margin and gradient-defect conditions; Remark D.22 gives the sufficient structural regime r>Lstater>L_{\mathrm{state}}. A numerical certificate of that margin/defect inequality and a global verification of localization, optimal-input containment, and recursive stability (D.7) are not supplied for the benchmark runs.

Remark C.28 separates fixed-mesh sampling limits from population mesh refinement. The error budget separates estimation/deployment, discretization, reference mismatch, and local-solve errors; it is not a joint mesh-uniform finite-budget guarantee.

W7. Baseline comparisons

Our principal evidence is independent of the equation-based baselines' ranking. Same-actor refinement and matched derivative ablations control the reference policy and distinguish what Stage II contributes. Appendix A documents PDGM/DeepSMP variants and method-specific validation-checkpoint selection under the common encoder size, learning rate, and update budget. This supports a reproducible comparison at the stated settings, although not an exhaustive per-method tuning study.

DeepSMP learns its first-order adjoints and deploys their stationarity-based control; PGDPO constructs an explicit curvature-guided action refinement from fixed-reference continuations without training an additional adjoint network. Table 19 evaluates the stated finite-candidate search, while standard history-based RL and general MPC remain broader comparison classes not benchmarked here. The completed evidence supports the refinement mechanism rather than a universal method ranking.

W8. Presentation and relevance to learning

The optimization problem is how to improve a trained history-dependent policy using computation at decision time. Stage I optimizes shared network parameters; a small parameter gradient does not itself certify pointwise action optimality. Stage II uses a differentiable stochastic model to construct a history-specific action update. The control-theoretic identification specifies which sensitivities this learning-and-refinement procedure needs and how to obtain them without another value or adjoint network.

This directly connects the work to model-based policy learning and test-time optimization. Figure 1 and Algorithm 1 give the operational construction, Theorem 5.1 justifies its inputs, and the appendix proof guide separates the technical dependencies. The completed coupled-P2 and re-entry experiments test the resulting algorithmic mechanism.

AuthorsOpenReview ↗

Follow-up to Reviewer z32f (1/2): Online computation and direct optimization

Thank you again for your assessment. We have completed additional experiments addressing two connected concerns in your review: whether recovery is a useful allocation of online computation compared with direct optimization, and whether its objective benefits persist beyond the previously tested scale.

All references below are to the latest revised manuscript.

1. Testing an alternative use of the online budget — W4 and W7

An improvement over the unrefined actor alone does not establish that recovery is a useful use of online computation. Your concerns about the one-round search and the absence of a direct-optimization baseline therefore motivated an additional comparison: spending the online budget on iterative gradient optimization of the current action, rather than on estimating recovery inputs and solving the curvature-guided update.

Unlike one-round candidate selection, gradient shooting repeatedly adjusts the action using gradients of the sampled continuation cost. It uses the same trained actors and delayed simulator, with future action values and noise fixed across iterations and delayed-state derivative paths retained. Step sizes and iteration counts are selected by independent validation cost.

The two budget allocations test different computational constraints. GFG_F matches total forward simulation work, including all shooting iterations and PGDPO's direct and nested branches. GTG_T calibrates decision time, including differentiation and updates, at three validation queries per seed within a 25% tolerance. These distinguish matching forward simulation work from calibrating measured decision time at selected queries (Tables 19–20).

Mean control errors are shown below. Cost differences are PGDPO minus shooting; negative values favor PGDPO. Both pp-value columns use the paired joint seed–path bootstrap, unadjusted for multiplicity.

ProblemAllocationControl nRMSE: shooting → PGDPO (%)pcontrolp_{\mathrm{control}}Cost difference (×10−3\times10^{-3})pcostp_{\mathrm{cost}}
P1Forward-work GFG_F7.73 → 5.34<0.001<0.001−0.400.451
P1Time-calibrated GTG_T8.47 → 5.34<0.001<0.001−0.430.470
Coupled P2, n=5n=5Forward-work GFG_F8.21 → 7.25<0.001<0.001−1.220.055
Coupled P2, n=5n=5Time-calibrated GTG_T9.12 → 7.25<0.001<0.001−1.790.0024

PGDPO reduces control error relative to shooting by 30.9%/37.0% in P1 and 11.6%/20.4% in P2 under GF/GTG_F/G_T. The P2 time-calibrated cost difference has a 95% CI of [−2.57,−0.82]×10−3[-2.57,-0.82]\times10^{-3}; the other cost contrasts remain statistically unresolved.

Importantly, shooting itself improves on the Stage-I actor's control accuracy in P2, with statistically resolved differences. PGDPO therefore improves on an online optimizer that already helps the actor in this case, rather than benefiting only from comparison with methods weaker than Stage I.

The computational interpretation is consequently more concrete than “recovery is cheap”: under the stated simulation and timing allocations, using continuation sensitivities for curvature-guided recovery yields more accurate actions than spending the budget on the tested iterative optimizer, with an additional cost benefit in the P2 time-calibrated comparison. This provides direct evidence for the practical value of the recovery procedure as an alternative use of online computation.

Reviewer_Cjo5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors extended Pontryagin-guided direct policy optimization (PGDPO), also denoted PG-DPO in prior work.

The proposed method learns a control policy in Stage I, then applies corrections at each state in Stage II. In Stage I: an LSTM policy is trained by backprop through simulated trajectories.
In Stage II: rollouts are simulated from the observed history, Stage-I policy chooses future controls.

The method estimates reward sensitivity to current state and noise. These estimates are used to correct control. The theory shows that these estimates supply what the optimality conditions for delay problems require.

The method is evaluated on five problems (linear-quadratic, epidemic, trade execution, servo speed tracking); most have exact or numerical optimal controls.

Stage I frozen policy, PDGM, DeepSMP serve as baselines.

Results: the proposed refinement brings Stage-I policy controls closer to optimal (when an optimum is known), and lowers cost significantly on 3/5 problems.

Critical strengths
  • Stochastic control problems usually require solving coupled SDEs (Meng et al., 2025). The paper provides theory to show that estimates of derivatives obtained from simulated rollouts can provide the necessary information (Theorem 5.1) without the need to solve coupled SDEs.

  • The authors provide tests showing how history parameters and the resulting estimates impact performance

  • For problems with known best control: the proposed method beats the frozen Stage-I policy; also beats simpler trial-and-error search.

Critical weaknesses
  • Appendix B.3, Table 15 shows the only experiment that isolates gains from differentiating through the history (delta=0.2) - no significant gains.

  • Weak results of PDGM and DeepSMP baselines can be due to lack of tuning their parameters; the learning rate and numberse of updates do not match the parameters used for the proposed method.

Questions for authors

From table 2 and 10, it seems that a decision can take close to a second even on a high-end GPU. Hence, real-time use doesn't seem feasible. Would the extra compute bring other methods stronger benefits, e.g. SHAC (Xu et al., 2022)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Cjo5

Thank you for your careful assessment and for suggesting stronger-delay tests.

Unless stated otherwise, table and appendix references are to the revised manuscript.

The new contribution is the identification that makes delay-aware refinement computable. Current-injection duality identifies the effective diffusion curvature of the coupled delay-adjoint family with an n×nn\times n fixed-control current-state Hessian block. Theorem 5.1 connects the discrete computation to that continuous-time object as history resolution grows. This makes the required inputs available from a learned policy's simulated continuations without a separate coupled-adjoint solver or full history-Hessian assembly. The earlier PGDPO architecture is retained; this identification resolves the additional delay-specific computational problem.

1. Contribution of delayed re-entry derivatives

The expanded tests now demonstrate an additional benefit in deployed control accuracy. At κ=2\kappa=2, Full reduces control nRMSE versus Detached by 24–25% in P1 and 12–16% in coupled P2 (p<0.001p<0.001 in all four settings; General Response §2). Full and Detached retain the same delayed forward model. Moreover, the Full/Detached/Instantaneous recovery comparisons in Table 15 of the revised manuscript use no finite-range adjoint-input projection, so these gains do not rely on the calibrated clipping ranges. The additional Full–Detached cost differences remain statistically unresolved.

2. Online recovery cost and allocation of computation

The same-actor comparisons establish what online refinement adds to an already trained policy. Choosing the best allocation between additional offline training and online recovery is a separate total-budget question. Longer actor training and SHAC concern learning the reference policy; Stage II uses its continuation to correct the current action. SHAC uses truncated differentiable rollouts and a critic during training (Xu et al., 2022), so these are potentially complementary roles. Our experiments isolate refinement; they do not yet rank it against longer/larger actor training or SHAC under a matched total budget.

Tables 2 and 10 distinguish total evaluation time from batch-one latency. In P5, latency is 1.73–1.79 s at the first decision, 0.030–0.031 s at the last, and 0.82–0.89 s overall median. All recorded batch-one calls exceed the 0.020.02 s model step. These are simulated closed-loop refinement results, not a real-time hardware demonstration. Table 2's cumulative Stage-II cost covers all 256 evaluation paths, so a total-budget comparison must also specify the deployment volume.

3. Completed comparison with current-action search

Table 19 in the revised manuscript (Table 18 in the original submission) tests a direct use of online simulation. With matched forward work in P1, finite-candidate MC search gives control nRMSE 19.27% versus 5.34% for PGDPO (p<0.001p<0.001); the paired cost difference is unresolved (p=0.10p=0.10). The P2 rows retain the original separable configuration and are distinct from the completed coupled-P2 results. This provides evidence against the stated one-round search baseline; gradient-based shooting is a distinct comparison.

4. Baseline tuning and interpretation

The central conclusions do not depend on PDGM or DeepSMP performing poorly. They follow from paired refinement of the same trained actors and controlled Stage-II ablations. Appendix A documents the baselines' architectures and problem-specific variants; each retains the checkpoint with its own method-specific validation loss, rather than simply the final iterate. They share the stated encoder size, learning rate, and update budget. This is not exhaustive per-method hyperparameter optimization, so the cross-method ranking is specific to that protocol. The within-actor and within-recovery evidence remains independent of which external baseline ranks best.

Reference: Xu, J., et al. (2022). Accelerated Policy Learning with Parallel Differentiable Simulation. ICLR.

AuthorsOpenReview ↗

Follow-up to Reviewer Cjo5: Online Recovery and Longer-Delay Results

Thank you for your assessment. Since our initial response, we have completed additional experiments addressing the practical value of online correction and the effect of maintaining delayed continuations over longer histories.

Unless stated otherwise, references are to the latest revised manuscript.

1. Comparing recovery with an online optimizer that improves the actor

Your concern about baselines performing worse than Stage I motivates testing recovery against an alternative that can itself improve the trained policy. We added iterative gradient shooting: starting from the same actor, it repeatedly updates the current action using gradients of the simulated continuation cost. This compares direct action optimization with curvature-guided recovery while keeping policy training fixed.

Shooting uses the same delayed simulator, retains delayed-state derivative paths, and holds future action values and noise fixed across iterations. Its step size and iteration count are selected by independent validation cost. We evaluate two allocations: GFG_F matches total forward simulation work, including all shooting iterations and PGDPO's direct and nested branches; GTG_T calibrates synchronized decision time, including differentiation and updates, at three validation queries per seed within a 25% tolerance (Appendix B.5, Tables 19–20).

The table reports five-seed mean control errors. Cost differences are PGDPO minus shooting; negative values favor PGDPO. Paired pp-values use the joint seed–path bootstrap, unadjusted for multiplicity.

ProblemShooting allocationControl nRMSE: shooting → PGDPO (%)pcontrolp_{\mathrm{control}}Cost difference (×10−3)(\times10^{-3})pcostp_{\mathrm{cost}}
P1Forward-work GFG_F7.73 → 5.34<0.001<0.001−0.400.451
P1Time-calibrated GTG_T8.47 → 5.34<0.001<0.001−0.430.470
Coupled P2, n=5n=5Forward-work GFG_F8.21 → 7.25<0.001<0.001−1.220.055
Coupled P2, n=5n=5Time-calibrated GTG_T9.12 → 7.25<0.001<0.001−1.790.0024

PGDPO achieves lower control error in all four comparisons. The P2 comparison is particularly informative for your baseline concern: both shooting allocations significantly improve the Stage-I control error of 9.81%, and PGDPO improves further to 7.25%. Its cost advantage over time-calibrated shooting is also statistically resolved, with a 95% CI of [−2.57,−0.82]×10−3[-2.57,-0.82]\times10^{-3}. The other shooting cost contrasts remain statistically unresolved.

The additional evidence therefore concerns the return from online computation: under the specified allocations, recovery obtains more accurate actions than a validation-selected iterative optimizer, including one that already helps the actor in P2. The resolved P2 cost difference also shows an objective benefit in the time-calibrated comparison.

2. Longer-delay tests show benefits in realized cost

We extended the continuation-model ablation to δ=0.8\delta=0.8, giving 40 lag steps at h=0.02h=0.02. This builds on the stronger-coupling Full–Detached evidence reported in our initial response by examining whether retaining the delayed continuation also leads to improvements in realized cost under substantially longer memory. The new comparison uses PGDPO and instantaneous-memory recovery, evaluated during deployment (Table 14).

ProblemControl nRMSE: Instantaneous → PGDPO (%)pcontrolp_{\mathrm{control}}Cost difference (×10−3)(\times10^{-3})pcostp_{\mathrm{cost}}
P125.12 → 4.49<0.001<0.001−8.34<0.001<0.001
Coupled P2, n=5n=510.20 → 7.16<0.001<0.001−1.730.018

Control errors are five-seed means, and cost differences are PGDPO minus Instantaneous. Paired tests use the joint seed–path bootstrap, unadjusted for multiplicity.

The P1 result illustrates why the continuation used for correction matters: instantaneous-memory recovery increases mean cost above Stage I by 7.41×10−37.41\times10^{-3}, whereas PGDPO decreases it by 0.93×10−30.93\times10^{-3}. In P2, both variants reduce mean cost relative to the actor, but retaining the delayed continuation provides a further statistically resolved reduction. At this longer delay, accounting for memory improves both reference-action accuracy and the objective attained by the deployed controller.

Together, these additions strengthen the practical case for refinement in two complementary ways: its accuracy advantage persists against an online alternative that improves the actor, and preserving the delayed continuation produces statistically resolved cost benefits in the longer-delay tests. We would appreciate your consideration of these additional results.

W1. Thank you for performing several experiments to address this point. The experiments with strengthened delayed-state contributions showed lower control error, and costs were improved in longer-delay experiments. Since the authors reported this clearly in Section 6, this resolves the main point raised in W1.

W2. It is still unclear why the different learning rates (5e-5 vs 1e-4) and numbers of updates (5000 vs 3000) were chosen. In my view, the relevance of the proposed method depends on how well it performs relative to the baselines.

Regarding the question about compute costs for correction, it seems that real-time performance is not currently feasible. Given that the method could in principle offer higher precision, I would encourage the authors to mention domains where this aspect would be highly beneficial. It would also be useful to conduct compute-matched experiments for training PDGM and DeepSMP longer, or to compare with methods such as SHAC (Xu et al., 2022).

A small comment to improve presentation clarity: the abstract mentions PGDPO as an existing method. In other parts of the submission PGDPO denotes the proposed method. To avoid confusion, it would be good to consistently refer to the method from prior work as, for example, delay-free PGDPO.

AuthorsOpenReview ↗

Response to Reviewer Cjo5: Training Settings, Practical Applications, and Terminology

Thank you for your follow-up and for confirming that the additional experiments and their presentation in Section 6 resolve W1. We appreciate your further comments on baseline training, practical use, and terminology.

1. Training settings and baseline comparisons

We appreciate your emphasis on the baseline comparisons. The learning rates and update budgets for Stage I, PDGM, and DeepSMP were practical, method-specific choices informed by our implementation and training experience with each method. We agree that broader hyperparameter tuning and compute-matched training would provide a useful additional assessment of their relative performance.

2. Where additional control precision can be useful

We agree that the reported recovery times do not demonstrate real-time readiness. The method is presently better suited to offline policy analysis and simulation-based controller design, where additional computation can be allocated to improving the accuracy of the computed actions.

One example is the analysis of investment and trading policies under market frictions. Constantinides (1986) shows that utility can be relatively insensitive to deviations from optimal portfolio proportions, while transaction costs substantially affect trading frequency and volume. This motivates examining the prescribed decisions alongside aggregate objective values, particularly when the policy itself is the object of study.

Another example is simulation-based controller development. Mysore et al. (2021) demonstrate the practical importance of action-signal smoothness, motivating attention to control behavior alongside aggregate performance. Although smoothness and reference-control accuracy are distinct properties, both highlight the importance of examining the actions produced by a controller. Our optimal-execution and delayed-feedback tracking benchmarks illustrate how refinement can change control profiles, constraint activation, and switching behavior while improving realized cost.

3. SHAC and terminology

We appreciate the suggestion to consider SHAC (Xu et al., 2022). We have added a brief discussion in Related Work, noting that SHAC uses short-horizon differentiable simulation and a learned critic for policy training, whereas our Stage-II recovery refines the current control while keeping the learned history policy fixed.

We have also clarified the terminology by referring to the earlier formulation as Markovian PGDPO and using PGDPO for the stochastic state-delay extension developed in this paper.

Thank you again for these constructive suggestions.

References

Constantinides, G. M. (1986). Capital Market Equilibrium with Transaction Costs. Journal of Political Economy, 94(4), 842–862.

Mysore, S., et al. (2021). Regularizing Action Policies for Smooth Control with Reinforcement Learning. ICRA.

Xu, J., et al. (2022). Accelerated Policy Learning with Parallel Differentiable Simulation. ICLR.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/3): Overview

We thank the reviewers for their careful assessments and constructive suggestions. We appreciate their recognition of the connection between automatic differentiation and stochastic delay optimality conditions, together with the numerical diagnostics. Our contribution extends Pontryagin-guided policy refinement to stochastic state-delay systems with control-dependent diffusion, combining a computational construction with theoretical justification.

We evaluate this extension through both objective performance and, where optimizer references are available, the accuracy of the recovered controls at the evaluated histories. Obtaining a good objective value and accurately recovering the model-implied decision rule are related but distinct numerical goals, particularly when the decision rule itself is of scientific or operational interest.

In response to the reviews, we have strengthened the empirical evidence in two ways:

  • Nonseparable P2 — z32f W5; related to WXRt Q4. We removed modal separability from the multidimensional P2 benchmark and reran the evaluation. In the revised 5- and 20-dimensional instances, PGDPO reduces its Stage-I policy’s control nRMSE by 26.0% and 24.3%, respectively. Direct paired comparisons also show lower realized cost at both dimensions (p<0.001p < 0.001 for the cost differences). These results demonstrate that the refinement gains are not restricted to a benchmark that decomposes into independent scalar control problems.

  • Delayed re-entry derivatives — Cjo5 W1; z32f W2; WXRt Q2. We expanded the delay-length and delay-strength ablation on P1 and the revised, coupled P2. When the delayed-state coefficients in drift and diffusion are doubled, preserving re-entry derivative paths reduces control nRMSE by approximately 12–25% relative to recovery that retains the delayed forward model but removes those derivative paths (p<0.001p < 0.001 in all four problem–delay combinations). The corresponding paired cost differences remain statistically unresolved. The new evidence therefore establishes an additional benefit in deployed control accuracy, beyond the previously reported differences in adjoint-input accuracy.

Together, these completed experiments establish refinement beyond separable scalar modes and a deployed-control benefit from the delay-specific derivative paths. Below we present the results and their interpretation; the individual responses address the remaining questions.

Unless stated otherwise, table and appendix references are to the revised manuscript.

AuthorsOpenReview ↗

General Response to All Reviewers (2/3): Nonseparable P2

1. Nonseparable P2: revised benchmark and updated evaluation

Addressing z32f W5 and W3; related to WXRt Q4.

Unless stated otherwise, table and appendix references are to the revised manuscript.

The refinement gains persist after removing modal separability from P2. We revised the n=5n=5 and n=20n=20 instances to remove the original simultaneous-diagonalization structure, introducing node-specific heterogeneity into the memory drift. Here nn denotes both the physical state and control dimension. At the baseline memory strength,

\qquad \eta=0.25,$$ where $E_n$ is diagonal with equally spaced entries from $-1$ to $1$, and $\eta$ controls the memory-drift heterogeneity. The modified $A_M$ no longer commutes with the current-state drift matrix $A$, preventing simultaneous diagonalization and the original decomposition into scalar control subproblems. We retain the linear-quadratic structure, distributed-delay kernel, and control-dependent diffusion; the $n=1$ instance remains the original scalar benchmark. For the coupled instances, we recomputed the exact same-grid reference using a **full-matrix generalized Riccati recursion on the augmented history state**, retaining cross-coordinate terms rather than solving separate scalar modes. We reran the main comparisons on the revised coupled dynamics, including LSTM-DPO, PGDPO, PDGM, and DeepSMP. The table below summarizes the refinement of the Stage-I policy by PGDPO. Control nRMSE is measured against the exact same-grid optimizer, and entries are means ± standard deviations over five training seeds. |$n$|LSTM-DPO control nRMSE (%)|PGDPO control nRMSE (%)|Relative reduction| |:---:|:---:|:---:|:---:| |5|$9.81\pm0.37$|$7.25\pm0.33$|$26.0\\%$| |20|$15.74\pm0.22$|$11.91\pm0.13$|$24.3\\%$| For comparison, PDGM and DeepSMP achieve control nRMSEs of $31.78\\%$ and $53.61\\%$ at $n=5$, and $108.0\\%$ and $59.75\\%$ at $n=20$, respectively. PGDPO also lowers realized total cost relative to Stage I. The paired differences are $-1.77\times10^{-3}$ at $n=5$ and $-1.74\times10^{-2}$ at $n=20$, with $p < 0.001$ in both cases; the latter corresponds to a **0.87% reduction in total cost**. These tests compare PGDPO directly with Stage I, not with the optimizer reference. **The updated evaluation therefore shows improvements in both control accuracy and paired cost after removing the separable structure.** This directly addresses z32f W5 with exact-reference evidence on coupled 5- and 20-dimensional problems. **Why the control-accuracy improvement matters (z32f W3).** Our numerical goal includes recovering model-prescribed controls, not only a low expected cost. For example, at $n=20$ the Stage-I control nRMSE is $15.74\\%$ even though its cost difference from the optimizer reference is not statistically resolved ($p=0.58$). This does not prove equivalent costs or quantify test power; it shows that a nonsignificant cost comparison alone does not certify action fidelity. Similar objective values need not imply equally accurate agreement with the model’s optimal decision rule. When the prescribed actions themselves are of scientific or operational interest, accurately computing those actions is a relevant numerical goal in its own right. Here, control nRMSE measures agreement with the optimizer’s action at the histories visited by each method. The **24–26% reductions therefore demonstrate improved numerical fidelity in recovering the model-implied controls**, even though the relative change in total cost is smaller. The LQ structure also makes substantial relative improvements in control accuracy compatible with smaller objective changes, because cost varies quadratically with admissible control perturbations around an optimum. This supports distinguishing action-level accuracy from aggregate objective performance; it does not identify the reported nRMSE with the objective gap. These comparisons evaluate Stage-II refinement as a whole. We next examine the distinct contribution of delayed re-entry differentiation, extending the experiments within the existing P1 formulation and using the revised, coupled P2.
AuthorsOpenReview ↗

General Response to All Reviewers (3/3): Re-entry and direct-optimization results

2. Delayed re-entry derivatives: evidence across delay length and strength

Addressing Cjo5 W1, z32f W2, and WXRt Q2.

Unless stated otherwise, table and appendix references are to the revised manuscript.

The expanded experiments show an additional control-accuracy benefit from retaining delayed re-entry derivatives under stronger delayed coupling. To examine whether the benefit extends beyond adjoint-input accuracy to deployed controls, we expanded the ablation on P1 (n=1n=1) and the revised, coupled P2 (n=5n=5) across delay lengths δ∈0.2,0.4\delta\in\\{0.2,0.4\\} and delay-strength multipliers κ∈1,2\kappa\in\\{1,2\\}, giving eight settings. Here, κ=2\kappa=2 doubles the delayed-state coefficients in both drift and diffusion relative to the baseline κ=1\kappa=1.

We compare full PGDPO recovery (“Full”) with re-entry-detached recovery (“Detached”), which retains the delayed forward model but blocks differentiation through strictly past delayed-state reads. This isolates the derivative paths without replacing the continuation model. The Full/Detached/Instantaneous recovery comparisons in Table 15 of the revised manuscript use no finite-range adjoint-input projection (Appendix B.3); the reported gains therefore do not depend on the calibrated clipping ranges.

The table reports control nRMSE (%) as mean ± standard deviation over five training seeds. The 95% confidence intervals concern Full-minus-Detached differences in percentage points, with negative values favoring Full; pp-values are from the joint bootstrap test.

Problem(δ,κ)(\delta,\kappa)Full nRMSE (%)Detached nRMSE (%)95% CI, Full − Detachedpp
P1(0.2,1)(0.2,1)5.34±0.175.34\pm0.175.46±0.215.46\pm0.21[−0.33,+0.19][-0.33,+0.19]0.560.56
P1(0.2,2)(0.2,2)5.67±0.365.67\pm0.367.48±0.387.48\pm0.38[−2.46,−1.00][-2.46,-1.00]<0.001< 0.001
P1(0.4,1)(0.4,1)4.43±0.184.43\pm0.184.84±0.334.84\pm0.33[−0.62,−0.09][-0.62,-0.09]0.0080.008
P1(0.4,2)(0.4,2)4.14±0.304.14\pm0.305.49±0.315.49\pm0.31[−1.77,−0.89][-1.77,-0.89]<0.001< 0.001
Coupled P2(0.2,1)(0.2,1)7.25±0.337.25\pm0.337.17±0.247.17\pm0.24[−0.16,+0.30][-0.16,+0.30]0.490.49
Coupled P2(0.2,2)(0.2,2)8.87±0.048.87\pm0.0410.12±0.0910.12\pm0.09[−1.61,−0.88][-1.61,-0.88]<0.001< 0.001
Coupled P2(0.4,1)(0.4,1)7.30±0.197.30\pm0.197.39±0.167.39\pm0.16[−0.24,+0.08][-0.24,+0.08]0.290.29
Coupled P2(0.4,2)(0.4,2)8.77±0.228.77\pm0.2210.41±0.4110.41\pm0.41[−2.10,−1.17][-2.10,-1.17]<0.001< 0.001

At κ=2\kappa=2, Full has lower control error in all four problem–delay combinations (p<0.001p < 0.001 each). These four results also survive Bonferroni adjustment within the eight Full–Detached control-error comparisons. The relative reductions against Detached are approximately 2424\\% and 2525\\% in P1, and 1212\\% and 1616\\% in coupled P2, at δ=0.2\delta=0.2 and 0.40.4, respectively. This directly addresses the concern that more accurate adjoint inputs had not been shown to yield more accurate deployed controls.

The comparison with Stage I further illustrates the contribution in coupled P2. At κ=2\kappa=2, Full reduces the Stage-I mean control error by approximately 15% and 20%, compared with 3% and 5% for Detached, at δ=0.2\delta=0.2 and 0.40.4, respectively. Thus, removing the re-entry derivative paths substantially reduces the average refinement gain in these stronger-coupling settings.

At the baseline setting (δ,κ)=(0.2,1)(\delta,\kappa)=(0.2,1), the Full–Detached control-error difference remains statistically unresolved in both problems. Increasing the delay length alone resolves the difference in P1 (unadjusted p=0.008p=0.008), but not in coupled P2 (p=0.29p=0.29). The additional benefit is therefore regime-dependent: stronger delayed coupling yields a consistent accuracy advantage at both tested delay lengths, whereas increasing delay length alone does not do so in both problems.

The demonstrated additional benefit is more accurate recovery of model-prescribed controls under stronger delayed coupling, in both the scalar point-delay and nonseparable distributed-delay problems. Full–Detached paired cost differences remain statistically unresolved in all eight settings.

3. Existing Monte Carlo current-control search comparison

Addressing WXRt Q1; related to z32f W7.

Table 19 of the revised manuscript (Table 18 in the original submission) reports the Monte Carlo current-control search comparison for P1 and the original (separable) P2 configuration of the submitted version. In P1, forward-work-matched search gives control nRMSE 19.2719.27\\% versus 5.345.34\\% for PGDPO (p<0.001p<0.001); the paired cost difference is unresolved (p=0.10p=0.10). The original-P2 entries in Table 19 are separate from the revised coupled-P2 results above. Our WXRt Q1 response explains the relationship to direct optimization and DPC.

Additional experiments

We present the completed experiments first. Additional work is actively underway: gradient-based one-step shooting and scalability experiments involving larger state dimensions and longer delay histories. These experiments are ongoing, and their results are not included in the completed comparisons above.

Area_Chair_nNbcOpenReview ↗

please engage

Dear reviewers,

Please join the discussion with the authors.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.