acceptodds
Under review as a conference paper at ICLR 2027

Learning to Switch with Stability and Convergence Guarantees: Policy Gradient for State-Independent Controller Switching in Linear Systems

Abstract

The design of feedback controllers plays a central role in determining how dynamical systems behave under closed-loop operation. A single fixed controller, however, may not provide the desired behavior for every performance objective. Addressing different requirements by repeatedly redesigning the controller can be difficult, especially when the design space is continuous. When a finite set of pre-designed controllers is available, the problem can be reduced to deciding how these controllers should be selected over the operating time horizon, motivating controller switching. However, the key challenge of switching is to improve closed-loop performance while maintaining stability. This is especially important in online learning, where an unstable update can destabilize the system during training. To address this, we develop StableSwitch-PG, a policy- gradient framework that optimizes a state-independent controller-selection distribution while preserving mean-square stability throughout learning for linear time- invariant systems. For known dynamics, we derive the exact policy gradient and establish global convergence of the objective to the optimal randomized switching policy. For unknown dynamics, we estimate the gradient from sampled trajectories and establish asymptotic and finite-time convergence guarantees. Finally, we derive a Lyapunov-based update condition that preserves mean-square stability at every learning iteration.

Primary area:
learning theory
Keywords:
Policy Gradient, Controller Switching, Mean-Square Stability, Safe Reinforcement Learning, Switched Linear Systems, Model-Free Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_C7fPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper considers a linear time-invariant (LTI) dynamical system and assumes that a library of linear state feedback controller for this LTI system is available. The paper then aims at learning the distribution for switching among the available feedback gain to minimize a typical discounted quadratic cost. The authors propose a policy gradient framework to find learn the switching distribution in two regimes: known and unknown dynamics.

While the paper is well-written and theoretical my recommendation is reject. The problem in this paper is not relevant. To be more specific, at high level, this paper considers the simplest linear time varying system. Why would one study switching controllers for this system? There is no motivation for this. It is well known that a single controller is enough for controlling a linear system. The paper could have some motivation if at least linear parametric systems were considered but it is not the case. Again compare the simplest system of linear time invariant noise-free dynamical system with more general dynamical systems (nonlinear, uncertain, domain randomized, etc..) currently in the literature. There is no motivation for this setup, there is no gap and the theoretical results bring no value.

Critical strengths

Paper is well-written and easy to follow. One can start reading the paper and follow it.

Critical weaknesses

The setup is too simple: linear time-invariant dynamical system, without any noise and uncertainty. There is no research gap in the paper rather what is sold as research gap is a made up problem that brings no value to the literature.

The problem is not scientifically sound: there is no need to have switching for the linear time-invariant systems. Why would someone consider switching controllers for a linear system? There is no need for that. Design a single optimal controller and never switch! This can be the optimal switching. Then there is no need for all the theoretical developments here and caring about stability and this and that.

Questions for authors

The authors give the controller selection probability in (2). And they say that (2) is used to sample the switching signal and (2) is the distribution over controllers. You cannot use a distribution over controllers when controlling. What you are trying to say is that you use (2) as soft max.

There is a half-sentence after Lemma 1. Please take care of that.

After Lemma 2, the authors discuss regret with respect to the optimal switching. There is no optimal switching. The optimal switching is “no-switching”. I understand that you mean optimal with respect to a given set of controllers but I would argue that why would one consider such a set? You have a time invariant linear system without any uncertainty. You have quadratic costs. So just design the optimal controller for this system from theories developed almost a century ago.

1 comment
AuthorsOpenReview ↗

Response to the Reviewer's comments

We thank the reviewer for the thoughtful comments regarding the motivation for controller switching and its relationship to classical LQR. It appears there may be a misunderstanding: the reviewer notes that because the problem is an LQR formulation, a new controller can be designed directly by solving the Riccati equation, and thus interprets our work as an attempt to provide an alternative solution to this well-established result. However, this is not the case.

Motivation of the Work

The reviewer is absolutely correct that for an LTI system with LQR objective, the optimal gain can be found by solving the Algebraic Riccati Equation. In this setting, the objective is not to synthesize another feedback gain, but to determine how the available controllers should be selected to improve performance under the new objective.

For example, autonomous UAVs and NASA aircrafts may use onboard autopilots containing prevalidated flight-control laws. Adding new controllers or modifying the gains of the approved ones may require additional review. Similarly, aircraft electronic engine control systems (often known as FADEC) may include previously approved selectable control options, while introducing new control software may require further validation and approval. These examples motivate the use of existing controller libraries, where performance must be improved through supervisory controller selection without modifying the individual feedback gains. To the best of our knowledge our work appears to be the first one to address this challenge. We agree that the scope is limited due to the LTI and LQR nature of the problem. Nonetheless, the proposed method can still be adapted to nonlinear systems by leveraging iterative LQR (iLQR) framework.

We understand that the proposed idea goes against the traditional thinking for an LQR problem. We hope the above examples clarifies the need for the proposed study. We don't claim that our method to be implemented for every LQR systems, but we respectfully emphasize that certain systems would benefit from the from the results presented in this paper, and such systems are often most sensitive and safety critical.

Reviewer_ZNCDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates randomized switching among fixed linear feedback controllers, learning a state-independent softmax policy to minimize discounted quadratic cost. For known dynamics, it derives an exact policy gradient and proves global convergence under an advantage-positivity assumption. For unknown dynamics, it develops a REINFORCE estimator with convergence guarantees. It also proposes a model-dependent step-size rule to preserve mean-square stability. Experiments on a linearized CartPole compare exact, model-free, and stability-preserving policy gradient methods.

Critical strengths

1. The value and occupancy representations, exact gradient and performance-difference identity are specialized cleanly to i.i.d. switching, and the proofs are complete and easy to verify.

2. The paper separates discounted admissibility from mean-square stability, which matters because mixtures of individually stable modes can be unstable. The step-size rule is simple and applies to any update direction, exact or sampled.

Critical weaknesses
  1. Limited new insight The results mostly transfer standard tools to this special case: the MJLS Lyapunov value representation and mean-square stability criterion, the policy-gradient theorem and performance difference, softmax gradient domination (Mei et al. 2020), and Robbins–Siegmund. Global convergence rests on Assumption 1, which is imposed on every iterate and is neither characterized nor verified. The O(1/k) rate further assumes bounded logit spread along the trajectory, which excludes optima at simplex vertices, exactly where softmax PG is known to slow down (Li et al. 2021).
  2. The key mechanism is not analyzed: why and when randomized switching helps. For a scalar system, J(p) is linear-fractional in p, so the optimum lies at a vertex and randomization cannot help; any benefit must therefore come from non-commuting closed-loop modes. The paper characterizes neither when p* is interior nor when mixtures of stable modes destabilize, even though its own experiment shows mixing halving the cost.
Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Response to the Reviewer's comments

We thank the reviewer for the thoughtful comments.

This paper investigates a PG method to design controller switching. Our analysis borrows the major concepts from PG and hence it may appear that the results are `standard'. However, PG has not been studied in this context and our analysis required additional stability of LTI systems during online learning, which later resulted in the refinement of the learning rate. Thus, our main contribution is to enable performance optimization while preserving stability throughout the learning process.

We acknowledge that softmax-based policy-gradient methods can exhibit slow convergence as the optimal distribution approaches a simplex vertex. Alternative probability parameterizations or optimization methods may help mitigate this limitation. Exploring such alternatives in our switching framework is a worthwhile direction for future research.

  1. Note on Assumption 1.

In the original manuscript, Assumption 1 requires

Γθ=−∑m=1Mp⋆(m)Aθ,m⪰0.\Gamma_\theta=-\sum_{m=1}^{M}p^\star(m)A_{\theta,m}\succeq0.

This condition means that the optimal mixture must provide a nonnegative improvement in every state direction compared with the current policy. We acknowledge that this is a strong requirement, and determining the general conditions under which it holds is beyond the scope of this work. However, we observed that this condition can be relaxed. In our proof, we only need the following inequality:

Tr⁡[(κˉXθ−Xp⋆)Γθ]≥0.\operatorname{Tr}\left[(\bar\kappa X_\theta-X_{p^\star})\Gamma_\theta\right]\geq0.

The original assumption requires the optimal mixture to provide a nonnegative improvement in every state direction. The relaxed assumption does not require this. The improvement can be negative in some state directions, provided that the overall weighted contribution remains nonnegative. This relaxed condition is sufficient to obtain the inequality J(θ)−J⋆≤κˉTr⁡(XθΓθ)J(\theta)-J^\star\leq\bar\kappa\operatorname{Tr}(X_\theta\Gamma_\theta), which is needed for our gradient-dominance proof. Therefore, the subsequent convergence results remain valid under this weaker assumption, together with the other stated conditions. We emphasize that the relaxed condition is not guaranteed to hold for every system. However, it is less restrictive than the original assumption because Γθ\Gamma_\theta is no longer required to be positive semidefinite.

  1. Note on p∗p^*.

Switching between stable controllers can lead to destabilized behavior and that has been illustrated in Switched Systems: Stability Analysis and Control Synthesis by Daniel Liberzon (among many others) https://liberzon.csl.illinois.edu/teaching/Liberzon-LectureNotes.pdf. Existing results focus on finding switching laws to stabilize systems, but they do not focus on the optimality of such switching laws with regards to any control performance metric. In this work, the objective is to find switching that achieves the best performance but, more importantly, mean-square stability is enforced at every intermediate switching policy during the learning. In general the optimal p∗p^* would be in the interior. For instance consider two matrices F1=A−BK1=diagF_1 = A- BK_1 = \text{diag}{ a1,a2a_1, a_2} and F2=A−BK2=diagF_2 = A- BK_2 = \text{diag}{ a2,a1a_2, a_1}. Supposed the QQ and RR matrices have been chosen such that Ci=Q+Ki⊺RKi=cIC_i = Q + K_i^\intercal R K_i = cI. Then, one obtains J(p)=c1−γ[pa12+(1−p)a22]+c1−γ[pa22+(1−p)a12]J(p) = \frac{c}{1-\gamma[pa_1^2+(1-p)a_2^2]} + \frac{c}{1-\gamma[pa_2^2+(1-p)a_1^2]}, where pp is the probability of selecting the gain K1K_1 and 1−p1-p for the gain K2K_2. For such systems clearly, p=0.5p= 0.5 is the optimal solution, which is in the interior.

Characterizing when p∗p^* lies in the interior of the probability simplex is interesting and is a potential future direction.

  1. Switching between individually stable controllers can make the system unstable.

In the paper On Stability of Linear Switched Differential Algebraic Equations by Daniel Liberzon and Stephan Trenn, it is demonstrated that switching between two individually stable subsystems can destabilize the overall system (Figure 3).

Reviewer_mWipOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a state independent switching scheme for feedback control: given a finite set of feedback gain matrices and an LTI system, formulate an LQR problem and learn a switching policy which determines the best gain matrix to use at each time step. A policy gradient formulation is proposed to solve for the switching policy. The results are demonstrated on a simple cart-pole system, in which the goal is to stabilize the pole to the inverted pendulum position, if the pole starts at or very close to this equilibrium.

Critical strengths

The paper started out with a lot of promise, namely that of learning switched control schemes, which is a challenging yet very practical problem in control.

The literature review is very good – the authors clearly identified previous work and their contributions, and discussed interesting cases, such as the case in which even if all the feedback gain matrices result in asymptotically stable systems, switching between them can destabilize the system.

Critical weaknesses

Why focus on state independent switching? It seems to me that, from a practical perspective, state dependent switching is much more interesting.

My main concern is with the motivation for this work. It is well known that for an LTI system, the LQR solution is a constant gain matrix for the infinite time horizon, and a time varying gain matrix for the finite time horizon. Why then, set up this problem of learning how to switch between known feedback gain matrices, why not just compute the solution directly?

A second issue is the demonstration of this method on a real system. The cart pole example in the appendix is a very simple system, especially without the swing-up phase. Why develop all this machinery for such a simple example?

There are some places where awkward wording made it hard to understand: Introduction: “At every time step, one controller is sampled from the same probability distribution, independently of the current state and the previously active controller.”;

“The gains are fixed,” – sounds like the gains do not change, what you mean to say is there is a finite set of gains specified ahead of time to choose from.

After Lemma 1, “therefore summarizes the expected future quadratic cost under the current switching distribution.” is repeated.

Questions for authors

Please explain why you cannot use the Riccati equation solution for the problem considered in this paper. Even if A, B are unknown, this seems like the best route, by first identifying A, B from trajectory data.

The relative improvement for using this method seems quite mild, for the example in the appendix. Could you please comment on that?

1 comment
AuthorsOpenReview ↗

Response to the Reviewer's comments

We thank the reviewer for the thoughtful comments regarding the motivation for controller switching, the relationship to classical LQR, and the numerical evaluation.

Motivation of the Work

The reviewer is absolutely correct that for an LTI system with LQR objective, the optimal gain can be found by solving the Algebraic Riccati Equation. In this setting, the objective is not to synthesize another feedback gain, but to determine how the available controllers should be selected to improve performance under the new objective.

For example, autonomous UAVs and NASA aircrafts may use onboard autopilots containing prevalidated flight-control laws. Adding new controllers or modifying the gains of the approved ones may require additional review. Similarly, aircraft electronic engine control systems (often known as FADEC) may include previously approved selectable control options, while introducing new control software may require further validation and approval. These examples motivate the use of existing controller libraries, where performance must be improved through supervisory controller selection without modifying the individual feedback gains. To the best of our knowledge our work appears to be the first one to address this challenge. We agree that the scope is limited due to the LTI and LQR nature of the problem. Nonetheless, the proposed method can still be adapted to nonlinear systems by leveraging iterative LQR (iLQR) framework.

State Dependent Switching

We focus on state-independent switching because it simplifies both the learning problem and the theoretical analysis. In state-independent switching, we only need to learn a fixed probability distribution over the available controllers, whereas state-dependent switching requires learning how these probabilities change with the system state. This becomes more challenging as the state dimension increases. Moreover, state-independent switching allows us to calculate the value function exactly by solving a matrix equation, without requiring value-function approximation. It also allows us to obtain an exact mean-square stability condition and derive the policy gradient analytically. In state-dependent switching, the value function is generally no longer quadratic, and the stability analysis becomes more complicated. Therefore, we focus on state-independent switching as a simpler starting point for understanding how controller-selection probabilities affect performance and stability. Extending the analysis to state-dependent switching is an important direction for future work. It is also noteworthy that a state-dependent switching will require continuous observation of the state by the controller scheduler, thus making its implementation more challenging than the state-independent one.

Purpose of the CartPole Example

We use the CartPole example to demonstrate and evaluate our proposed method. Although CartPole is a simple system, it allows us to compare the performance of individual controllers, randomized switching, exact policy gradient, model-free policy gradient, and stability-preserving policy gradient under the same condition. Our main objective is to verify our theoretical derivations using a simple benchmark, which is why we chose the CartPole system.

Improvement from Switching

We respectfully clarify that the improvement relative to the individual-controller baselines is substantial. In Table 2, the best individual controller, K3K_3, achieves a mean cost of 99.12099.120, while Model-free PG achieves 49.73049.730, corresponding to an approximately 5050% reduction. StableSwitch-PG achieves a mean cost of 54.14054.140, representing an approximately 45.38%45.38\%% reduction relative to the best individual controller and a 42.31%42.31\%% reduction relative to equal-probability switching. These results demonstrate that optimizing the controller-selection distribution can substantially improve performance relative to selecting any individual controller in the prescribed library.

Summary

We sincerely thank the reviewer for their thoughtful comments and suggestions. We hope that we are able to convey under what scenarios the LQR controller is not practically feasible and thereby requiring the study presented this paper. We don't claim that our method to be implemented for every LQR systems, but we do emphasize that certain systems would benefit from the from the results presented in this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.