acceptodds
Under review as a conference paper at ICLR 2027

When Policy Improvement Breaks Future Learning: Self-Validating Policy Optimization

Abstract

Online reinforcement learning can invalidate its own next update. In continuing tasks with unbounded states, an approximate policy update can make the induced on-policy process heavy-tailed or poorly mixed. Subsequent value estimates and policy gradients may then become unreliable, even when the initial policy is stable and a stable optimum exists. To address this failure, we introduce a graded certificate for policy updates that identifies whether a raw policy preserves the statistical conditions needed by subsequent learning. We then develop self-validating policy optimization (SVPO), an average-cost actor–critic method that couples policy improvement to the validity of its own future updates. By retaining the physical task cost and preserving the certificate in the learned policy itself, SVPO keeps task performance aligned with the original objective and makes that policy valid for continued data collection and learning. The same construction yields Tilt, a certificate-preserving policy correction for existing policy optimizers. We prove that the certificate ensures that SVPO preserves the moment and mixing guarantees needed by future learning uniformly along its adaptive policy sequence. This establishes almost-sure convergence of the coupled actor–critic recursion to projected stationarity under standard assumptions. Experiments on switched LTI, input-saturated LQR, and unbounded queueing systems show that certificate-margin loss precedes downstream degradation, while SVPO maintains positive raw-policy margins with favorable physical task performance.

Primary area:
reinforcement learning
Keywords:
average-cost reinforcement learning, policy optimization, actor-critic, Lyapunov stability, stochastic approximation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vNVPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper deals in a relevant problem in online RL, where policy approximation/function approximations during the course of the trajectory is shown to cause instability in the system performance. The paper develops the necessary theory to characterise such an condition (CPPI), and then formulates an algorithm which in reasonable time and cost performs the self correction leading to provable guarantees in terms of average cost. Further the experimental evaluation done in the paper also seem sufficient to validate all the central claims in the paper.

Critical strengths
  1. I would say that the problem explored in this work is quite important, especially as is seen through the experiments where it is quite evident that without any self-correcting strategies systems with potentially unbounded state space and approximate policy evaluations can grow indefinitely.

  2. The algorithm proposed keeps the original task cost in check while improving the overall stability. Further this paper is firmly grounded theoretically.

  3. Experimental evaluations serve all the three important claims in the paper: whether the mentioned certificate for stability is important practically, whether their proposed algorithm SVPO maintains the certificate, and whether the extra time taken is within margins.

Critical weaknesses

Although the experiments done serve the primary purpose of validating the major claims made in the paper, it is quite evident that the experiments were done on structured control and queuing problems where the dynamics are already quite well established. The paper does not sufficiently explain how the proposed approach would scale to more complex environments.

Questions for authors
  1. Experimentations on some real and more challenging scenarios would be much useful.

  2. As is seen in "Related Works" section, there appears to be quite a number of works done for stable online RL scenarios. Why have these methods not been explicitly compared with the SVPO approach. The experimental section only present comparison with Tilt and PPO only.

  3. Alg 1 SVPO requires \phi_H and \psi_K, which are quantities which are difficult to calculate in real world scenarios, an explanation about how to get around such calculations in some real world settings would be great.

3 comments

Thank you for recognizing the importance of preserving the conditions required by future learning and the connection between our theory and experiments. We have strengthened the revision in two directions: a concrete account of feature construction and transfer across applications, and a direct comparison with Lyapunov cost shaping in queueing.

Q1. Experimentations on some real and more challenging scenarios would be much useful.

Q1. Evaluation in more complex environments

We agree that evaluation in larger-scale and real-world environments would further strengthen the empirical evidence. Our theory is formulated for general controlled Markov systems admitting an action-dependent Lyapunov comparison and a stabilizing reference policy, together with the stated moment and action-coverage conditions. For example, post-service workload in interacting queueing networks or post-action energy in nonlinear physical systems can provide candidate dissipativity features. The system-dependent task is to establish the relevant certificate ingredients; once they hold, CPPI and SVPO guarantee uniform drift, moments, and mixing across adaptive raw-policy updates, rather than requiring these properties to be checked after every update.

Our experiments target learning challenges that can also arise in more complex systems. Switched LTI examines the effects of actor-realization error, while zero-bias evaluations test subsequent sampling without injected actor bias. Input-saturated LQR deliberately restricts the critic representation to study learning under value-approximation error and continuous actions. Unbounded queueing tests SVPO with stochastic workloads and unbounded states and demonstrates how Tilt can be applied to an existing learner. Together, these results suggest that SVPO has the potential to address similar learning challenges beyond the structured environments studied here. Evaluation in larger-scale and real-world systems remains future work.

Manuscript changes: Appendices D.2 (p30–33), D.3 (p33–36), and D.4 (p36–39); Section 5.1 (p7–8) and the zero-bias evaluation in Appendix D.2 (p32).

Q2. As is seen in "Related Works" section, there appears to be quite a number of works done for stable online RL scenarios. Why have these methods not been explicitly compared with the SVPO approach. The experimental section only present comparison with Tilt and PPO only.

Q2. Comparisons with existing stable online-RL methods

We appreciate the request for broader baselines. Section 5.2 compares SVPO with Lagrangian AC and Filter AC, in addition to Vanilla AC, Entropy AC, and Vanilla+Tilt. Under the same input-saturated LQR task, SVPO obtains a median physical cost of 0.1383, compared with 0.2713 for Lagrangian AC and 0.1755 for Filter AC (Table 4). These baselines represent certificate penalties in learning and action correction at execution, respectively.

We added a PPO baseline that adapts the Lyapunov cost shaping and symlog state transformation of Pavse et al. (2024) to the queueing task, using constant learning rates. The comparison uses the same adaptive SVPO runs as the matched fixed/adaptive ablation (Table 7). Costs are medians across ten seeds of the cumulative average physical training cost over 200,000 steps; raw-policy margins are medians at the final checkpoint.

MethodPhysical costFinal raw-policy marginPersistent crossingsEscapes
SVPO7.81380.00990/100/10
PPO+Shaping7.93790.000610/100/10

Our comparisons distinguish how stability is incorporated into learning through objective penalties, executed-action correction, and SVPO's certificate-preserving raw-policy updates, while evaluating performance under the original physical task cost.

Manuscript changes: Sections 5.2–5.3 (p8–9); Table 4 (p35); Figure 3 (p9); Appendix D.4 (p36–39), Table 6 (p38), Figure 7 (p37), and the matched ablation in Table 7 and Figure 8 (p39).

Q3. Alg 1 SVPO requires \phi_H and \psi_K, which are quantities which are difficult to calculate in real world scenarios, an explanation about how to get around such calculations in some real world settings would be great.

Q3. Obtaining the features required by Algorithm 1

The two features require different information. ψK\psi_K is a standard fixed state-action task-critic representation, implementable through a finite kernel dictionary (Section 3.1) or other feature design satisfying the stated conditions. Beyond standard critic regularity, its certificate-related requirement is that the task component gθ=⟨θtask,ψK⟩g_\theta=\langle\theta_{\mathrm{task}},\psi_K\rangle has lower-order growth O(1+Lq/k)O(1+L^{q/k}), with q<kq<k (Assumption 5). ϕH\phi_H captures action-dependent state growth and can be constructed from an action-conditioned post-action state and an energy- or workload-like state-growth function. The three experiments in the paper use post-action representations.

For nonlinear stochastic dynamics xt+1=f(xt,at)+wt+1x_{t+1}=f(x_t,a_t)+w_{t+1}, the post-action growth L(f(x,a))−1L(f(x,a))-1 supplies a candidate ϕH(x,a)\phi_H(x,a), with disturbance effects included in Assumption 1's drift comparison. For data-driven extensions, transition samples can be used to fit this representation or its growth, with feature-error bounds incorporated into the dissipativity comparison. This provides a natural Lyapunov-dissipativity framework for general systems, in which the system-specific requirements concern constructing and validating the certificate feature, while the task critic retains the flexibility of standard feature design.

Manuscript changes: Section 3.1 (p3–4); Appendix D, post-action construction (p29); Appendices D.2 (p30–33), D.3 (p33–36), and D.4 (p36–39).

Reviewer_vNVPOpenReview ↗

Response to authors

Thank you for your comments. I will keep my score to 3.

Reviewer_XytEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how approximate policy updates in unbounded, continuing MDPs can destroy the moment and mixing properties needed for subsequent learning. It defines graded CPPI certificates on excess dissipativity load and constructs SVPO, an average-cost actor–critic with a positive certificate term shared by the critic and sampling policy. A related exponential reweighting, Tilt, corrects other policies. Under explicit drift, coverage, regularity, and stochastic-approximation assumptions, the paper establishes uniform stability properties and critic convergence, with actor stationarity requiring an additional gradient-field condition. Experiments on switched LTI control, input-saturated LQR, and a two-queue system examine certificate diagnostics, learning variability, and physical cost.

Critical strengths
  1. The graded excess-load condition, positive certificate channel, and uniform policy-family analysis give a coherent account of how to preserve the moments and Markov-noise regularity needed by an adaptive learner. The distinction between stability of an executed controller and regularity of the policy generating learning data is useful, and the certificate construction provides a concrete way to enforce the latter.

  2. The CriticOnly comparison without actor bias in Table 2 tests the importance of including the certificate in the sampling policy. The matched-policy update-direction test in Table 3 shows how critic approximation can reverse the margin response. Together with ten matched seeds and separate reporting of margins and physical escape, these controls support the proposed failure mechanism more directly than return comparisons alone.

Critical weaknesses
  1. Theorem 3 requires the equilibrium actor field to equal the gradient of some function F; Eq. (8) is explicitly a statewise semi-gradient, and the paper does not establish this identity for the implemented actor. Appendix C.11 additionally assumes that policy-induced TD perturbations are smaller than the frozen-policy TD decrease. These are meaningful restrictions, not consequences of the certificate. Keeping physical cost in the TD target also does not itself show that F is the original average-cost objective; Appendix C.13 assumes that equality separately. The stability result is valuable, but the abstract’s convergence framing should distinguish these additional requirements more prominently.

  2. All environments have analytically specified certificate features, and the main stress tests deliberately introduce actor bias or restrict critic features. Appendix D.3 notes that the saturated-LQR example violates Assumptions 1 and 3, so its finite-grid margins are empirical diagnostics rather than verification of the theoretical certificate. The comparisons also do not isolate the value of learning the shared coefficient against a fixed coefficient with the same initial raw policy, or compare with the cited Lyapunov cost-shaping approach. This leaves uncertainty about the practical gain from the full coupled method beyond supplying an effective stabilizing policy bias.

Questions for authors
  1. Can F be identified for the actor actually used, and can the gradient-field identity and the TD perturbation condition be verified in a nontrivial experimental instance? If not, please distinguish certificate preservation under its own assumptions from the additional conditions for critic convergence and actor stationarity in the headline claims.
  2. How does SVPO compare with a fixed-coefficient certificate policy initialized to the same raw policy, and with the cited Lyapunov cost-shaping method under the same physical-cost evaluation? Such controls would clarify what is gained by learning and sharing the certificate channel.
4 comments

Thank you for recognizing the contribution of the graded certificate, the uniform policy-family analysis, and the controlled mechanism experiments. Your questions help sharpen the connection between these contributions and the practical algorithm. We have clarified the theoretical guarantees and added the requested fixed-η\eta and cost-shaping comparisons in the queueing environment.

Q1. Can F be identified for the actor actually used, and can the gradient-field identity and the TD perturbation condition be verified in a nontrivial experimental instance? If not, please distinguish certificate preservation under its own assumptions from the additional conditions for critic convergence and actor stationarity in the headline claims.

W1/Q1. Constructive learning regularity and convergence of the coupled updates

For the implemented multidimensional practical actor, an explicit FF satisfying hˉa(u,θ∗(u))=∇F(u)\bar h_a(u,\theta^*(u))=\nabla F(u) has not been established. In Theorem 3, FF is an assumed potential of the equilibrium averaged actor field. Likewise, the uniform cTD<λTDc_{\mathrm{TD}}<\lambda_{\mathrm{TD}} condition has not been verified for a complete experimental learner. However, these conditions limit the application of Theorem 3 to the experimental learners, but are not required for the certificate-preservation and learning-regularity guarantees established under the stated system and model assumptions.

We clarify their verification criteria in Appendices C.11–C.12. A continuous scalar actor field admits a potential by integration, while a continuously differentiable multidimensional field on an open convex domain admits a potential when its Jacobian is symmetric. This criterion accounts for the dependence of the equilibrium critic and invariant distribution on the actor parameter. The TD condition requires the frozen-policy TD decrease to dominate the perturbation caused by critic-dependent policy changes.

We further examined these conditions at 240 checkpoints from the existing switched-LTI experiments (Appendix D.2). All 240 checkpoints exhibit positive frozen-policy TD decrease, and the computed projected fast Jacobians have negative largest real parts, with the equilibria lying on the projection boundary. The maximum relative asymmetry of the full actor Jacobian is 4.54%, 6.47%, and 28.17% at pressure levels 00, 0.20.2, and 0.50.5, respectively. These results provide local numerical evidence for critic stability and show smaller deviations from a gradient field at lower pressure. However, these local diagnostics do not establish all assumptions of Theorem 3. Full verification requires controlling policy-induced TD perturbations across the admissible policy family and establishing the gradient-field identity for the practical actor.

The additional conditions of Theorem 3 concern learning-algorithm convergence, rather than certificate preservation. Our central contribution is to constructively preserve the learning regularity that classical actor–critic analyses typically assume along the adaptive policy sequence. The results and their respective conditions are summarized below. Corollary 3 provides a physical average-cost comparator bound without the additional convergence conditions of Theorem 3, whereas Appendix C.13 establishes global optimality under stronger conditions.

ResultGuaranteeConditions
Theorems 1–2 and Corollary 2Certificate preservation and uniform drift, moments, and mixing across raw-policy updatesSystem and model conditions
Lemmas 1–2Mean-field, Poisson-equation, and Markovian-noise regularitySystem and model conditions
Theorem 3Critic-residual convergence and practical-actor projected stationaritySystem and model + extra learning conditions
Corollary 3Physical average-cost comparator boundSystem and model conditions
Appendix C.13Global optimality for physical average costSystem and model + extra learning conditions

We have clarified these distinctions in the Abstract and Introduction and added verification criteria in Appendices C.11–C.12. We also extend the convergence analysis to βtρ=rβt\beta_t^\rho=r\beta_t and βtθ=βt\beta_t^\theta=\beta_t for any fixed r>0r>0, allowing different update scales for the average-cost estimate and critic. The corresponding weighted Lyapunov analysis is provided in Appendix C.12.

Manuscript changes: Abstract/Introduction (p1–2); Algorithm 1 (p4); Section 4.2 (p5–7), Theorem 3 (p6); Assumption 6 (p13); Appendices C.11–C.12 (p24–28); Appendix D.2 (p30–33; numerical check p32–33).

Q2. How does SVPO compare with a fixed-coefficient certificate policy initialized to the same raw policy, and with the cited Lyapunov cost-shaping method under the same physical-cost evaluation? Such controls would clarify what is gained by learning and sharing the certificate channel.

W2/Q2. Matched coefficient ablation and Lyapunov cost shaping

Fixed versus adaptive η\eta. To isolate the benefit of learning the shared certificate coefficient, we added a matched queueing ablation (Appendix D.4, Figure 8). Both variants start from the same raw policy and coefficient η0\eta_0, use matched seeds, and train for 200,000 steps. The fixed variant holds η=η0\eta=\eta_0 in both the critic and sampling policy, whereas adaptive SVPO learns the shared coefficient. Physical cost is the cumulative average training cost; tail occupancy and raw-policy margin are measured at the final checkpoint. Scalar entries are medians over ten seeds.

MethodPhysical costFinal tail occupancyFinal raw-policy marginSeeds with persistent crossingsSeeds with physical escape
Fixed-η\eta SVPO7.98100.43330.003240/100/10
Adaptive-η\eta SVPO7.81380.08120.009890/100/10

Adaptive SVPO lowers physical cost and final tail occupancy in all ten pairs relative to fixed-η\eta SVPO, showing a performance benefit even when the fixed initialization preserves positive margins.

Lyapunov cost-shaping comparison. We also added PPO+Shaping, adapting the Lyapunov cost shaping and symlog state transformation of Pavse et al. (2024) to the queueing task. The baseline uses constant learning rates, with its implementation detailed in Appendix D.4. All methods are evaluated using the original physical cost.

MethodPhysical costFinal tail occupancyFinal raw-policy marginSeeds with persistent crossingsSeeds with physical escape
SVPO7.81380.08120.009890/100/10
PPO+Shaping7.93790.21190.000610/100/10

SVPO has lower physical cost and retains positive margins at every checkpoint, whereas PPO+Shaping has persistent crossings despite its positive final median margin.

Controlled stress tests and zero-bias evaluation. The switched-LTI actor bias and restricted LQR critic are deliberate choices to isolate actor-realization error and insufficient critic representation, respectively, both of which can arise in more complex learning systems. To further examine the mechanism without injected actor bias, we evaluated frozen policies in switched LTI with λ=0\lambda=0 (Appendix D.2). In the added checkpoint-matched frozen-policy evaluations, Vanilla AC incurs 18/960 subsequent physical escapes, all from nonpositive-margin checkpoints, versus 0/960 for SVPO.

Frozen-policy evaluationPhysical escapes / rollouts
Vanilla AC, all checkpoints18/960
Vanilla AC, nonpositive-margin checkpoints18/544
Vanilla AC, positive-margin checkpoints0/416
SVPO, all checkpoints0/960

This complements the existing zero-bias CriticOnly ablation (Table 2), where margin crossings occur in 10/10 seeds versus 0/10 for SVPO, linking certificate deterioration to downstream sampling risk under zero actor bias.

Applicability. Switched LTI analytically verifies the system-level conditions (Appendix D.2); queueing supplies explicit reference-policy contraction and action coverage (D.4). Input-saturated LQR tests the raw-policy mechanism with continuous actions and critic approximation beyond the sufficient geometric-drift and coverage conditions. Across all three, ϕH\phi_H is constructed from an action-dependent post-action state and a state-growth function.

Manuscript changes: Sections 5.1 (p7–8) and 5.3 (p9, Figure 3); Appendix D.2 (p30–33; zero-bias evaluation p32, Table 2 p31); Appendix D.3 (p33–36, Table 3 p34); Appendix D.4 (p36–39, Table 6 p38, Table 7 and Figure 8 p39).

Thanks for the response. It addresses most of my concerns, so I'm raising my score to 3.

Thank you for your encouraging feedback. We appreciate your careful consideration and constructive suggestions, which have helped us improve the clarity and strengthen the paper!

Reviewer_Z23vOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a failure mode in online reinforcement learning: policy updates can destroy the recurrence, moment, and mixing properties required for subsequent learning. It introduces graded certificate-preserving policy improvement, develops Self-Validating Policy Optimization (SVPO), and proposes an exponential-tilting correction for other optimizers. The theoretical analysis establishes stability regularity and conditional actor–critic convergence. Experiments consider switched linear systems, input-saturated LQR, and queueing systems.

Critical strengths
  1. The paper addresses a consequential issue in continuing RL: each policy update changes the process generating future learning data. The graded certificate connects policy construction to higher-order drift, moment bounds, and mixing regularity. This provides a useful perspective on learning in unbounded state spaces.

  2. The certificate enters the learned policy directly, providing a constructive mechanism for preserving regularity during data collection. The analysis of uniform moments along the adaptive policy sequence strengthens the contribution beyond a stability statement for individual fixed policies.

Critical weaknesses
  1. The policy-optimization guarantee remains substantially conditional. Equation (8) estimates a statewise free-energy gradient with the critic held fixed; its invariant-state average is a semi-gradient. Theorem 3 additionally assumes hˉa(u,θ∗(u))=∇F(u)\bar h_a(u,\theta^*(u))=\nabla F(u), but this property is not established for the general practical actor, nor is FF generally identified with the physical average-cost objective. Appendix C.11 also imposes the substantive condition cTD<λTDc_{\mathrm{TD}}<\lambda_{\mathrm{TD}} for the coupled critic dynamics. These assumptions are not consequences of certificate preservation. Consequently, the paper establishes the stability mechanism more convincingly than its broader optimization interpretation, which is central to the proposed method.

  2. The applicability of the assumptions and their connection to the experiments are insufficiently established. The construction requires a suitable Lyapunov function, dissipativity feature, reference drift, uniform action coverage, and common minorization. Appendix D.3 explicitly states that the LQR experiment violates the geometric reference-drift and uniform action-coverage conditions. In the switched-LTI experiments, the clearest physical failures involve an injected actor bias, while the zero-bias experiments show certificate crossings without physical escape over the reported horizon. The evidence therefore leaves uncertainty about when the complete theoretical guarantees apply and how reliably certificate loss predicts a failure of subsequent learning.

Questions for authors
  1. For which stated practical actor family is hˉa(u,θ∗(u))=∇F(u)\bar h_a(u,\theta^*(u))=\nabla F(u) established? Please clarify whether FF is the physical average cost, a regularized objective, or an assumed potential.

  2. Which existing experimental instance satisfies the complete assumptions of Theorem 3, including cTD<λTDc_{\mathrm{TD}}<\lambda_{\mathrm{TD}}? Please distinguish conditions verified in the manuscript from conditions imposed as assumptions.

  3. What prior system information is required to construct LL and ϕH\phi_H in the reported examples? Please clarify whether the method requires known dynamics or an independently available stabilizing controller.

  4. What evidence in the existing zero-bias experiments connects certificate crossings to degraded estimation or learning, beyond the certificate diagnostic itself? This clarification would help assess the claimed training-path failure mechanism.

6 comments

Thank you for highlighting the importance of connecting certificate preservation, policy optimization, and subsequent learning. Our revision strengthens this connection through a clearer organization of the guarantees and a zero-bias evaluation of the sampling and estimation behavior of policies encountered during training.

Q1. For which stated practical actor family is hˉa(u,θ∗(u))=∇F(u)\bar h_a(u,\theta^*(u))=\nabla F(u) established? Please clarify whether FF is the physical average cost, a regularized objective, or an assumed potential.

W1/Q1. Actor potential and optimization guarantees

For the implemented multidimensional practical actor, FF in Theorem 3 is an assumed potential of the equilibrium averaged actor field hˉa(u,θ∗(u))\bar h_a(u,\theta^*(u)), rather than an identified physical average-cost or regularized objective.

We clarify when the potential condition can hold in Appendix C.12. A continuous scalar-parameter actor field admits a potential by integration. For a multidimensional actor, a continuously differentiable equilibrium averaged field on an open convex neighborhood admits a potential when its Jacobian is symmetric. This criterion accounts for the dependence of both the equilibrium critic and invariant distribution on the actor parameter. A potential can therefore exist even when these quantities vary with the actor, provided the resulting derivative terms are included in the actor update or cancel in the averaged field.

We numerically examine the full actor Jacobian at 240 switched-LTI checkpoints (Appendix D.2). The maximum relative asymmetries are 4.54%, 6.47%, and 28.17% at bias strengths 0, 0.2, and 0.5, respectively. These results show smaller deviations from gradient-field structure at lower bias strengths, although an exact potential has not been established for the implemented multidimensional actor.

This additional condition is needed for practical-actor stationarity, but not for our central certificate-preservation and uniform learning-regularity guarantees. CPPI constructively preserves this regularity across the adaptive raw-policy sequence, turning the corresponding regularity conditions typically assumed in classical actor–critic analyses into properties maintained by policy construction. The resulting guarantees, their additional conditions, and their roles in connecting policy construction to physical task performance are distinct:

ResultGuaranteeConditions
Eq. (5) and Proposition 2Statewise free-energy minimization; Tilt is the minimum-KL correction at its achieved certificate loadSystem and model conditions
Theorems 1–2 and Corollary 2Uniform drift, moments, and mixingSystem and model conditions
Lemmas 1–2Mean-field, Poisson-equation, and Markovian-noise regularitySystem and model conditions
Theorem 3Critic-residual convergence; projected practical-actor stationaritySystem and model + extra learning conditions
Corollary 3Physical average-cost gap bounded by critic approximation, actor realization, and regularization termsSystem and model conditions
Appendix C.13Global optimalitySystem and model + extra learning conditions

We have revised the abstract and introduction to distinguish the certificate-preservation and learning-regularity guarantees from the additional conditions required for algorithmic convergence and optimality. Separately, Corollary 3 provides a physical average-cost comparator bound, while Appendix C.13 establishes global optimality under stronger conditions.

Manuscript changes: Abstract/Introduction (p1–2); Section 4.2 (p5–7), Theorem 3 (p6); Appendices C.12 (p25–28) and C.13 (p28); Appendix D.2, numerical check (p32–33).

Q2. Which existing experimental instance satisfies the complete assumptions of Theorem 3, including cTD<λTDc_{\text{TD}}<\lambda_{\text{TD}}? Please distinguish conditions verified in the manuscript from conditions imposed as assumptions.

W1/Q2. Verified system conditions and learning-convergence assumptions

We have not verified all assumptions of Theorem 3 for any reported experimental instance. This limits the instance-level verification of the learning-algorithm convergence guarantee, but does not affect the certificate-preservation and uniform learning-regularity results established under their stated system and model assumptions. At the system level, Appendix D.2 analytically verifies reference drift, higher-order moments, action coverage, and irreducibility for switched LTI. Appendix D.4 gives an explicit queueing reference-policy contraction, 12(0.82+1.152)=0.98125<1\tfrac12(0.8^2+1.15^2)=0.98125<1, and uniform low-excess-load action coverage. Input-saturated LQR extends the empirical evaluation to continuous actions and critic approximation beyond the geometric-reference-drift and uniform-coverage conditions, as analyzed in Appendix D.3.

The critic condition cTD<λTDc_{\mathrm{TD}}<\lambda_{\mathrm{TD}} has the same small-Lipschitz role as conditions in classical SARSA-control analysis. It requires the Lipschitz bound on policy-induced TD perturbations to be smaller than the frozen-policy TD decrease, yielding a strict critic Lyapunov decrease. The numerical checks summarized in Appendix D.2 provide local evidence for the projected fast dynamics. Establishing this comparison uniformly requires bounds on policy-dependent stationary distributions over the entire admissible family, which makes verification demanding in unbounded state spaces. The actor-potential condition is addressed in Q1.

CPPI supplies uniform learning regularity for the evolving raw policies; the small-Lipschitz condition controls the remaining critic-policy feedback. We have clarified the distinction between certificate preservation and conditional learning-algorithm convergence in the Abstract and Introduction, and added verification criteria in Appendices C.11–C.12. We also allow βtρ=rβt\beta_t^\rho=r\beta_t and βtθ=βt\beta_t^\theta=\beta_t for a fixed r>0r>0 throughout their common decay schedule. The scalar average-cost and feature-weighted TD updates can thus use different initial gains. The corresponding Lyapunov weight in Appendix C.12 preserves the critic decrease argument.

Manuscript changes: Section 4 (p4–7); Algorithm 1 (p4); Appendix A.2 (p13–14); Appendices C.11–C.12 (p24–28); Appendices D.2 (p30–33) and D.4 (p36–39).

Q3. What prior system information is required to construct LL and ϕH\phi_H in the reported examples? Please clarify whether the method requires known dynamics or an independently available stabilizing controller.

W2/Q3. System information and dissipativity-feature construction

Prior knowledge and assumptions of the system. In our experiments, ϕH\phi_H is constructed from the action-dependent post-action state and a state-growth function. This yields a unified construction across switched LTI, LQR, and queueing, with post-action states represented by faxf_ax, Ax+BuAx+Bu, and FaqF_aq, respectively. The Lyapunov function LL is supplied, so ϕH\phi_H is evaluated analytically. The theoretical guarantees require a reference policy satisfying Assumption 1 as a stabilizability witness, but Algorithm 1 does not execute that policy as a backup controller.

The online policy evaluates the supplied ϕH\phi_H for candidate actions before sampling. For a general dynamical system xt+1=f(xt,at)+wt+1x_{t+1}=f(x_t,a_t)+w_{t+1}, the post-action growth L(f(x,a))−1L(f(x,a))-1 provides a candidate feature, with the disturbance contribution controlled by Assumption 1.

This action-dependent Lyapunov comparison also provides the construction principle for general systems. A data-driven extension could fit post-action growth from transition samples, provided that the resulting feature-error bounds preserve the dissipativity comparison.

Manuscript changes: Section 3.1 (p3–4); Appendix D, post-action construction (p29); Appendices D.2 (p30–33), D.3 (p33–36), and D.4 (p36–39).

Q4. What evidence in the existing zero-bias experiments connects certificate crossings to degraded estimation or learning, beyond the certificate diagnostic itself? This clarification would help assess the claimed training-path failure mechanism.

W2/Q4. Downstream evidence under zero actor bias

We evaluated frozen raw policies from zero-bias LTI training. For each method, the evaluation uses ten seeds, six checkpoints, and sixteen rollouts per checkpoint, each lasting up to 5,000 steps. This evaluates the subsequent behavior of the same policies whose margins are measured.

Across all checkpoints, the escape counts pool evaluation rollouts. The TD-error statistic is the median checkpoint variance, where each checkpoint variance averages the within-rollout sample variances over completed rollouts.

MethodCheckpoint marginPhysical escapes / rolloutsMedian checkpoint TD-error variance
Vanilla ACNonpositive18/5440.02243
Vanilla ACPositive0/4160.00611
SVPOAll checkpoints0/9600.00435

Vanilla AC's 18/960 escapes all occur at nonpositive-margin checkpoints. The online training trajectories in Table 1 record 0/10 escapes for both methods at zero bias; the additional rollouts reveal downstream sampling risks associated with the encountered policies.

At the final checkpoint, physical cost is averaged over completed 5,000-step rollouts within each seed. To test subsequent estimation, we freeze each policy and reinitialize the original task critic and average-cost estimate, using the same initialization and learning rates for both methods. Both columns below report medians across ten training seeds; the standard deviation is computed within each seed over completed rollouts.

MethodPhysical costWithin-seed standard deviation of average-cost estimates
Vanilla AC0.0299300.01159
SVPO0.0263860.00564

At the final checkpoint, Vanilla AC incurs 5/160 escapes, compared with 0/160 for SVPO; the cost and estimation statistics above use completed rollouts. SVPO has lower physical cost in all ten paired seeds. These measurements connect the raw-policy drift diagnostic to downstream sampling risk, physical cost, and estimation variability under zero injected bias.

Manuscript changes: Section 5.1 (p7–8); Appendix D.2, zero-bias evaluation (p32) and Table 1 (p31).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.