- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper deals in a relevant problem in online RL, where policy approximation/function approximations during the course of the trajectory is shown to cause instability in the system performance. The paper develops the necessary theory to characterise such an condition (CPPI), and then formulates an algorithm which in reasonable time and cost performs the self correction leading to provable guarantees in terms of average cost. Further the experimental evaluation done in the paper also seem sufficient to validate all the central claims in the paper.
Critical strengths
-
I would say that the problem explored in this work is quite important, especially as is seen through the experiments where it is quite evident that without any self-correcting strategies systems with potentially unbounded state space and approximate policy evaluations can grow indefinitely.
-
The algorithm proposed keeps the original task cost in check while improving the overall stability. Further this paper is firmly grounded theoretically.
-
Experimental evaluations serve all the three important claims in the paper: whether the mentioned certificate for stability is important practically, whether their proposed algorithm SVPO maintains the certificate, and whether the extra time taken is within margins.
Critical weaknesses
Although the experiments done serve the primary purpose of validating the major claims made in the paper, it is quite evident that the experiments were done on structured control and queuing problems where the dynamics are already quite well established. The paper does not sufficiently explain how the proposed approach would scale to more complex environments.
Questions for authors
-
Experimentations on some real and more challenging scenarios would be much useful.
-
As is seen in "Related Works" section, there appears to be quite a number of works done for stable online RL scenarios. Why have these methods not been explicitly compared with the SVPO approach. The experimental section only present comparison with Tilt and PPO only.
-
Alg 1 SVPO requires \phi_H and \psi_K, which are quantities which are difficult to calculate in real world scenarios, an explanation about how to get around such calculations in some real world settings would be great.
3 comments
Q3. Alg 1 SVPO requires \phi_H and \psi_K, which are quantities which are difficult to calculate in real world scenarios, an explanation about how to get around such calculations in some real world settings would be great.
Q3. Obtaining the features required by Algorithm 1
The two features require different information. is a standard fixed state-action task-critic representation, implementable through a finite kernel dictionary (Section 3.1) or other feature design satisfying the stated conditions. Beyond standard critic regularity, its certificate-related requirement is that the task component has lower-order growth , with (Assumption 5). captures action-dependent state growth and can be constructed from an action-conditioned post-action state and an energy- or workload-like state-growth function. The three experiments in the paper use post-action representations.
For nonlinear stochastic dynamics , the post-action growth supplies a candidate , with disturbance effects included in Assumption 1's drift comparison. For data-driven extensions, transition samples can be used to fit this representation or its growth, with feature-error bounds incorporated into the dissipativity comparison. This provides a natural Lyapunov-dissipativity framework for general systems, in which the system-specific requirements concern constructing and validating the certificate feature, while the task critic retains the flexibility of standard feature design.
Manuscript changes: Section 3.1 (p3–4); Appendix D, post-action construction (p29); Appendices D.2 (p30–33), D.3 (p33–36), and D.4 (p36–39).
Response to authors
Thank you for your comments. I will keep my score to 3.
Thank you for recognizing the importance of preserving the conditions required by future learning and the connection between our theory and experiments. We have strengthened the revision in two directions: a concrete account of feature construction and transfer across applications, and a direct comparison with Lyapunov cost shaping in queueing.
Q1. Evaluation in more complex environments
We agree that evaluation in larger-scale and real-world environments would further strengthen the empirical evidence. Our theory is formulated for general controlled Markov systems admitting an action-dependent Lyapunov comparison and a stabilizing reference policy, together with the stated moment and action-coverage conditions. For example, post-service workload in interacting queueing networks or post-action energy in nonlinear physical systems can provide candidate dissipativity features. The system-dependent task is to establish the relevant certificate ingredients; once they hold, CPPI and SVPO guarantee uniform drift, moments, and mixing across adaptive raw-policy updates, rather than requiring these properties to be checked after every update.
Our experiments target learning challenges that can also arise in more complex systems. Switched LTI examines the effects of actor-realization error, while zero-bias evaluations test subsequent sampling without injected actor bias. Input-saturated LQR deliberately restricts the critic representation to study learning under value-approximation error and continuous actions. Unbounded queueing tests SVPO with stochastic workloads and unbounded states and demonstrates how Tilt can be applied to an existing learner. Together, these results suggest that SVPO has the potential to address similar learning challenges beyond the structured environments studied here. Evaluation in larger-scale and real-world systems remains future work.
Manuscript changes: Appendices D.2 (p30–33), D.3 (p33–36), and D.4 (p36–39); Section 5.1 (p7–8) and the zero-bias evaluation in Appendix D.2 (p32).
Q2. Comparisons with existing stable online-RL methods
We appreciate the request for broader baselines. Section 5.2 compares SVPO with Lagrangian AC and Filter AC, in addition to Vanilla AC, Entropy AC, and Vanilla+Tilt. Under the same input-saturated LQR task, SVPO obtains a median physical cost of 0.1383, compared with 0.2713 for Lagrangian AC and 0.1755 for Filter AC (Table 4). These baselines represent certificate penalties in learning and action correction at execution, respectively.
We added a PPO baseline that adapts the Lyapunov cost shaping and symlog state transformation of Pavse et al. (2024) to the queueing task, using constant learning rates. The comparison uses the same adaptive SVPO runs as the matched fixed/adaptive ablation (Table 7). Costs are medians across ten seeds of the cumulative average physical training cost over 200,000 steps; raw-policy margins are medians at the final checkpoint.
Our comparisons distinguish how stability is incorporated into learning through objective penalties, executed-action correction, and SVPO's certificate-preserving raw-policy updates, while evaluating performance under the original physical task cost.
Manuscript changes: Sections 5.2–5.3 (p8–9); Table 4 (p35); Figure 3 (p9); Appendix D.4 (p36–39), Table 6 (p38), Figure 7 (p37), and the matched ablation in Table 7 and Figure 8 (p39).