- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a unified framework for learning, stabilizing and optimizing for general nonlinear systems. The framework first learns a stabilizing linear feedback controller gain and certifies it. Then the framework continues with learning the dynamics and covariance of the noise. Finally the learned model is used to find an optimal policy. The algorithm is called Gibbs-Randomized Model-Based Adaptive Control. The paper is timely, targets an interesting topic and comes with theoretical proofs. They are however modeling assumptions that contradict the main motivation of this work and for this, my recommendation is reject. Please see my comments below.
Critical strengths
The paper has a holistic view on learning, stabilizing and optimization. This is unique in the current literature. The paper provides theoretical results in terms of regret analysis along with the simulation results. This is also important given that the majority of publications in ML venues stick to the algorithmic level and not beyond.
Critical weaknesses
The authors motivate by saying that the current SOTA puts assumptions on the dynamics of noise and we remove those. I do not agree with this claim. They specifically say several works consider the following structure but we do not. However on page 5 at the beginning of the theoretical results, they say that “Let ” denote the known component of and then they propose to model as . This is the same thing as previous works, right? You know a part of the dynamics and then you model the residuals with linear features. If you know nothing about the system so $ f^{pub}$ is zero and your model falls literally to the previous works.
The notation in the main body of the paper obscures the understanding. I’ve got lost after equation (4). I could not find the definition for . Similarly, I could not find the definition of . If these are used in the main text, they should be defined in the main text. In addition to that, it is difficult to understand why they have been defined and how one wants to use those.
While the paper claims about general nonlinear dynamics, the method is only for nonlinear systems that are stabilizable by a linear state-feedback which is not a big class. In addition, when the dynamics are not known, one cannot know if they are stabilizable by linear state-feedback or not. The assumption cannot be verified for the considered problem formulation.
Another point in the claim is the related to noise saying that SOTA has limiting assumption on noise. This paper considers a Gaussian noise with unknown covariance. There are many works that consider non-Gaussian noise, general noise or even adversary. I would say that the current assumption on the noise is among the most restrictive cases. Of course it is fine to make this assumption about the noise but please do not over-claim.
Questions for authors
What is a sieve estimation? It is good to discuss it in the introduction a bit as it is not common knowledge.
In discussing limitations, point ii, authors mentioned that several works consider the feature map to be uniformly bounded over all states and admissible actions and discussed that this is a strong restriction. Whether this is restrictive or not depends on the definition of admissibility. If this definition requires stability (which) is typically the case, this is not that restrictive anymore.
When one assumes that the dynamics are unknown and even the parametric structures are not known, how could one verify that the system is stabilizable with a linear state feedback?
Again when we know nothing about the dynamics, how can we make sure that this decomposition is valid and later there exists a correct to converge to in Theorem 1?
Theorem 3 gives the regret analysis. Authors have defined \Pi^{A} as the policy generated by the algorithm. Then the regret analysis is given with respect to .
Have you specified it? I cannot find it in the text. It is unclear to me how to define the baseline in the regret analysis for the general result in Theorem 3 and not simulation.
8 comments
Response to the Reviewer Auwc (2/7)
W1: The authors motivate by saying that the current SOTA puts assumptions on the dynamics of noise and we remove those. I do not agree with this claim. They specifically say several works consider the following structure but we do not. However on page 5 at the beginning of the theoretical results, they say that “Let ” denote the known component of and then they propose to model as . This is the same thing as previous works, right? You know a part of the dynamics and then you model the residuals with linear features. If you know nothing about the system so is zero and your model falls literally to the previous works.
R-W1 (2/2):
-
To better show the difference between our method and the existing works, in Appendix D.2 of this paper, we provide a proper example:
Here, and are unknown. In this example, the sieve space is
\begin{aligned} &\mathcal F_t =\Bigl\\{f: f= c_uu +\sum_{\ell=1}^{d_t-1}c_\ell\sin(\ell x),\\ (c_u,c_1,\ldots,c_{d_t-1})\in\mathbb R^{d_t} \Bigr\\}. \end{aligned}
Thus, the corresponding feature map is . In fact, without loss of generality, we can extend this example to the more general form. Let be any bounded interval with , and set . If with for some , then admits the following representation on :
�0�
+\sum_{\ell=1}^{\infty} \bigl
�1�
, \qquad x\in[a,b]. \end{aligned}$$
For the above , in the settings of some existing works, such as (Mania et al., 2022) and (Musavi et al., 2024), both cited in this paper, there must exist a known , such that for , . However, assuming that such a known exists is restrictive (for example, such an does not exist for a linear system). This issue is a main motivation for this paper and a problem that we aim to solve.
- In this paper, we allow that for any , . This cause a difficulty: there exists a nonzero error between and , which is called the approximation error in statistics (naturally, in the existing works shown in the second point, the approximation error is zero). Such an approximation error makes the proof methods in existing works unable to be applied to this paper. Specifically, approximation error can bias the parameter estimates in the existing works using fixed feature maps, thereby leaving a gap between the estimated model and the true dynamics that cannot be eliminated. For example, under and sufficient excitation, (Mania et al., 2022), cited in this paper, show that the estimations of parameters converge to : . However, if is a best uniform approximation with error , the above convergence result no longer holds. Specifically, the prediction-error bound in (9) of (Mania et al., 2022), cited in this paper, is no longer guaranteed because approximation error can introduce an additional bias term.
Response to the Reviewer Auwc (3/7)
W2: The notation in the main body of the paper obscures the understanding. I’ve got lost after equation (4). I could not find the definition for . Similarly, I could not find the definition of . If these are used in the main text, they should be defined in the main text. In addition to that, it is difficult to understand why they have been defined and how one wants to use those.
R-W2: We apologize for the unclear notation and provide formal definitions of the notation used in the main text. Here are the definitions of and : denotes the update period introduced in Algorithm 1 of this paper. We update our estimator of every steps. denotes the smallest -field generated by the random variables .
W3: While the paper claims about general nonlinear dynamics, the method is only for nonlinear systems that are stabilizable by a linear state-feedback which is not a big class. In addition, when the dynamics are not known, one cannot know if they are stabilizable by linear state-feedback or not. The assumption cannot be verified for the considered problem formulation.
R-W3 (1/2): We would like to thank the reviewer for this insightful observation. The reply is shown below.
-
We understand this comment to refer to Assumption 1(b), which requires the existence of a linear state-feedback gain that globally stabilizes the system. Indeed, this assumption restricts the application area of our results. However, we just want to find a control policy satisfying (38) and (39) in Lemma 1 of this paper:
We note that, under Assumption 1(b), the first stage of the proposed algorithm (lines 2–9) learns a linear feedback controller satisfying the above two conditions within steps, with probability at least ; see Proposition 2 and Lemma 1 of this paper. This is the motivation for proposing Assumption 1(b).
-
Thus, the key requirement for the stabilization stage is a feedback policy , possibly nonlinear, satisfying (38) and (39). If such a policy is either known a priori or can be learned by an algorithm in finitely many steps, Assumption 1(b) is no longer needed for this stage. However, learning a stabilizing controller for an unknown nonlinear system is itself a challenging problem. Given the page limit and this paper's focus on online identification and regret analysis, we leave this problem to future work and provide the following studies as potential starting points for relaxing Assumption 1(b):
[1] De Persis C, Rotulo M, Tesi P. Learning controllers from data via approximate nonlinearity cancellation[J]. IEEE Transactions on Automatic Control, 2023, 68(10): 6082-6097.
[2] Luppi A, De Persis C, Tesi P. On data-driven stabilization of systems with nonlinearities satisfying quadratic constraints[J]. Systems & Control Letters, 2022, 163: 105206.
[3] Strässer R, Schaller M, Worthmann K, et al. SafEDMD: A Koopman-based data-driven controller design framework for nonlinear dynamical systems[J]. Automatica, 2026, 185: 112732.
-
In fact, there are many existing works that have more restrictive assumptions and cannot be verified with unknown nonlinear systems. Concretely, some existing works directly impose stability conditions on the uncontrolled or zero-input dynamics. For example, Assumption 6.1 of Ziemann and Tu (2022), cited in this paper, requires that
indicating that ; [4] assumes:
indicating that ; [5] assumes:
indicating that . These stability conditions are special cases of Assumption 1(b) of this paper with and suitable choices of the constants and the Lyapunov matrix . Such stability assumptions generally appear in the literature (see [4-6]), and we also cannot verify them when the system is unknown.
Response to the Reviewer Auwc (1/7)
We sincerely appreciate your time reviewing the manuscript and for your thoughtful feedback. We reply to your comments in the below.
R-W1 (1/2):
We apologize for the misunderstanding regarding the contribution of this paper. From the author's perspective, the reason for generating this misunderstanding is that we did not clearly explain the term “sieve space”. This may arise from the differences in terminology between the statistics and control communities. For simplicity of explanation, we set fpub≡0 in this response.
Roughly speaking, many existing works, including (Mania et al., 2022), (Musavi et al., 2024), and (Wagenmaker et al., 2023), all cited in this paper, directly assume f=θ∗ϕ, where ϕ is a known, fixed-dimensional feature map. These works transform the estimation of f into a parametric estimation problem. This approach has been widely used in control theory.
In this paper, we do not have any a priori information about f. However, roughly speaking, according to approximation theory (see Section 3 of (Belloni et al., 2015), cited in this paper), if f is sufficiently smooth (e.g., Hölder smooth or C∞ smooth), we can construct sieve spaces: \mathcal F_t :=\left\\{ f_{\theta_t}:f_{\theta_t}(x,u) =\theta_t\phi^{(t)}(x,u),\\ \theta_t\in\mathbb R^{d_x\times d_t} \right\\}, t=1,2,\dots (a precise definition of a sieve space has been added to the preliminaries), where ϕ(t) can be chosen as a Chebyshev basis, or a Fourier basis, such that there exists fθt∗=θt∗ϕ(t)∈Ft satisfying
�0�
^{d_x+d_u})} \longrightarrow0 \qquad\text{as }t\to\infty. \end{aligned}$$
Here, Ct can increase slowly with t. More precisely, we establish the convergence rate in terms of the dimension dt of the sieve space Ft in (52) of this paper. This analysis is essentially different from the formulation of the existing works shown in the second point. Therefore, for sieve spaces Ft, we need to analyze the error between θt∗ and θt in Theorem 1 of this paper and combine it with the approximation error $\|f_{\theta^*_t}-f\|_{L^\infty(
−Ct,Ct^{d_x+d_u})}$ to obtain the convergence rate of $\|f-f_{\theta_t}\|_{L^\infty(
−Ct,Ct^{d_x+d_u})}$:
�3� ^{d_x+d_u})} \leq \\|f-f_{\theta_t^*}\\|_{L^\infty( �4� ^{d_x+d_u})}+ \sup_{(x,u)\in �5� ^{d_x+d_u}} \\|\phi^{(t)}(x,u)\\|_2 \\|\theta_t-\theta_t^*\\|_{\rm F}. \end{aligned}$$