acceptodds
Under review as a conference paper at ICLR 2027

A Unified Framework for Learning, Stabilization, and Optimization in Online Nonlinear Control

Abstract

We study online stochastic optimal control with unknown nonlinear dynamics. We address three limitations of existing approaches: prespecified parametric dynamics; reliance on a priori bounded trajectories or a known stabilizing controller; and restrictive noise assumptions, such as bounded disturbances or known covariance structures. Specifically, we propose Gibbs-Randomized Model-Based Adaptive Control (), a two-stage algorithm that first learns and certifies a stabilizing feedback gain and then improves control performance through sieve-based dynamics estimation, noise-covariance estimation, and residual-policy updates. Under mild assumptions, we derive finite-sample bounds for the dynamics and covariance estimation errors and establish regret guarantees relative to the optimal finite-horizon controller. We further specialize the theoretical analysis to linear quadratic regulator, nonlinear Fourier dynamics, and nonlinear transient price impact. Under the corresponding conditions, the proposed controller achieves a sublinear regret bound that scales as the square root of the time horizon and is asymptotically optimal.

Primary area:
learning on time series and dynamical systems
Keywords:
Online stochastic control, Data-driven stabilization, Sieve-based system identification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AuwCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a unified framework for learning, stabilizing and optimizing for general nonlinear systems. The framework first learns a stabilizing linear feedback controller gain and certifies it. Then the framework continues with learning the dynamics and covariance of the noise. Finally the learned model is used to find an optimal policy. The algorithm is called Gibbs-Randomized Model-Based Adaptive Control. The paper is timely, targets an interesting topic and comes with theoretical proofs. They are however modeling assumptions that contradict the main motivation of this work and for this, my recommendation is reject. Please see my comments below.

Critical strengths

The paper has a holistic view on learning, stabilizing and optimization. This is unique in the current literature. The paper provides theoretical results in terms of regret analysis along with the simulation results. This is also important given that the majority of publications in ML venues stick to the algorithmic level and not beyond.

Critical weaknesses

The authors motivate by saying that the current SOTA puts assumptions on the dynamics of noise and we remove those. I do not agree with this claim. They specifically say several works consider the following structure f(xt,ut)=θϕ(xt,ut)f(x_t,u_t)= \theta \phi(x_t,u_t) but we do not. However on page 5 at the beginning of the theoretical results, they say that “Let fpubf^{pub}” denote the known component of ff and then they propose to model as fθ(x,u)=fpub(x,u)+θϕ(x,u)f_{\theta}(x,u)=f^{pub}(x,u)+\theta \phi(x,u). This is the same thing as previous works, right? You know a part of the dynamics and then you model the residuals with linear features. If you know nothing about the system so $ f^{pub}$ is zero and your model falls literally to the previous works.

The notation in the main body of the paper obscures the understanding. I’ve got lost after equation (4). I could not find the definition for MM. Similarly, I could not find the definition of Gτ\mathcal{G}_{\tau}. If these are used in the main text, they should be defined in the main text. In addition to that, it is difficult to understand why they have been defined and how one wants to use those.

While the paper claims about general nonlinear dynamics, the method is only for nonlinear systems that are stabilizable by a linear state-feedback which is not a big class. In addition, when the dynamics are not known, one cannot know if they are stabilizable by linear state-feedback or not. The assumption cannot be verified for the considered problem formulation.

Another point in the claim is the related to noise saying that SOTA has limiting assumption on noise. This paper considers a Gaussian noise with unknown covariance. There are many works that consider non-Gaussian noise, general noise or even adversary. I would say that the current assumption on the noise is among the most restrictive cases. Of course it is fine to make this assumption about the noise but please do not over-claim.

Questions for authors

What is a sieve estimation? It is good to discuss it in the introduction a bit as it is not common knowledge.

In discussing limitations, point ii, authors mentioned that several works consider the feature map to be uniformly bounded over all states and admissible actions and discussed that this is a strong restriction. Whether this is restrictive or not depends on the definition of admissibility. If this definition requires stability (which) is typically the case, this is not that restrictive anymore.

When one assumes that the dynamics are unknown and even the parametric structures are not known, how could one verify that the system is stabilizable with a linear state feedback?

Again when we know nothing about the dynamics, how can we make sure that this decomposition fθ(x,u)=fpub(x,u)+θϕ(x,u)f_{\theta}(x,u)=f^{pub}(x,u)+\theta \phi(x,u) is valid and later there exists a correct θ∗\theta^* to converge to in Theorem 1?

Theorem 3 gives the regret analysis. Authors have defined \Pi^{A} as the policy generated by the algorithm. Then the regret analysis is given with respect to Π\Pi.

Have you specified it? I cannot find it in the text. It is unclear to me how to define the baseline in the regret analysis for the general result in Theorem 3 and not simulation.

8 comments
AuthorsOpenReview ↗

Response to the Reviewer Auwc (1/7)

We sincerely appreciate your time reviewing the manuscript and for your thoughtful feedback. We reply to your comments in the below.

W1: The authors motivate by saying that the current SOTA puts assumptions on the dynamics of noise and we remove those. I do not agree with this claim. They specifically say several works consider the following structure f(xt,ut)=θϕ(xt,ut)f(x_t,u_t)=\theta\phi(x_t,u_t) but we do not. However on page 5 at the beginning of the theoretical results, they say that “Let fpubf^{pub} ” denote the known component of ff and then they propose to model as fθ(x,u)=fpub(x,u)+θϕ(x,u)f_\theta(x,u)=f^{pub}(x,u)+\theta\phi(x,u) . This is the same thing as previous works, right? You know a part of the dynamics and then you model the residuals with linear features. If you know nothing about the system so fpubf^{pub} is zero and your model falls literally to the previous works.

R-W1 (1/2):

  • We apologize for the misunderstanding regarding the contribution of this paper. From the author's perspective, the reason for generating this misunderstanding is that we did not clearly explain the term “sieve space”. This may arise from the differences in terminology between the statistics and control communities. For simplicity of explanation, we set fpub≡0f^{\rm pub}\equiv0 in this response.

  • Roughly speaking, many existing works, including (Mania et al., 2022), (Musavi et al., 2024), and (Wagenmaker et al., 2023), all cited in this paper, directly assume f=θ∗ϕf=\theta^*\phi, where ϕ\phi is a known, fixed-dimensional feature map. These works transform the estimation of ff into a parametric estimation problem. This approach has been widely used in control theory.

  • In this paper, we do not have any a priori information about ff. However, roughly speaking, according to approximation theory (see Section 3 of (Belloni et al., 2015), cited in this paper), if ff is sufficiently smooth (e.g., Hölder smooth or C∞C^\infty smooth), we can construct sieve spaces: \mathcal F_t :=\left\\{ f_{\theta_t}:f_{\theta_t}(x,u) =\theta_t\phi^{(t)}(x,u),\\ \theta_t\in\mathbb R^{d_x\times d_t} \right\\}, t=1,2,\dots (a precise definition of a sieve space has been added to the preliminaries), where ϕ(t)\phi^{(t)} can be chosen as a Chebyshev basis, or a Fourier basis, such that there exists fθt∗=θt∗ϕ(t)∈Ftf_{\theta_t^*} =\theta_t^*\phi^{(t)}\in\mathcal F_t satisfying

�0�

^{d_x+d_u})} \longrightarrow0 \qquad\text{as }t\to\infty. \end{aligned}$$

Here, CtC_t can increase slowly with tt. More precisely, we establish the convergence rate in terms of the dimension dtd_t of the sieve space Ft\mathcal F_t in (52) of this paper. This analysis is essentially different from the formulation of the existing works shown in the second point. Therefore, for sieve spaces Ft\mathcal{F}_{t}, we need to analyze the error between θt∗\theta_t^* and θt\theta_t in Theorem 1 of this paper and combine it with the approximation error $\|f_{\theta^*_t}-f\|_{L^\infty(

−Ct,Ct-C_t,C_t

^{d_x+d_u})}$ to obtain the convergence rate of $\|f-f_{\theta_t}\|_{L^\infty(

−Ct,Ct-C_t,C_t

^{d_x+d_u})}$:

�3� ^{d_x+d_u})} \leq \\|f-f_{\theta_t^*}\\|_{L^\infty( �4� ^{d_x+d_u})}+ \sup_{(x,u)\in �5� ^{d_x+d_u}} \\|\phi^{(t)}(x,u)\\|_2 \\|\theta_t-\theta_t^*\\|_{\rm F}. \end{aligned}$$
AuthorsOpenReview ↗

Response to the Reviewer Auwc (2/7)

W1: The authors motivate by saying that the current SOTA puts assumptions on the dynamics of noise and we remove those. I do not agree with this claim. They specifically say several works consider the following structure f(xt,ut)=θϕ(xt,ut)f(x_t,u_t)=\theta\phi(x_t,u_t) but we do not. However on page 5 at the beginning of the theoretical results, they say that “Let fpubf^{pub} ” denote the known component of ff and then they propose to model as fθ(x,u)=fpub(x,u)+θϕ(x,u)f_\theta(x,u)=f^{pub}(x,u)+\theta\phi(x,u) . This is the same thing as previous works, right? You know a part of the dynamics and then you model the residuals with linear features. If you know nothing about the system so fpubf^{pub} is zero and your model falls literally to the previous works.

R-W1 (2/2):

  • To better show the difference between our method and the existing works, in Appendix D.2 of this paper, we provide a proper example:

    f(x,u)=b⋆u+∑ℓ=1∞αℓ,⋆sin⁡(ℓx).\begin{aligned} &f(x,u) =b_\star u +\sum_{\ell=1}^{\infty} \alpha_{\ell,\star}\sin(\ell x). \end{aligned}

    Here, b⋆b_\star and αℓ,⋆ℓ≥1\\{\alpha_{\ell,\star}\\}_{\ell\geq1} are unknown. In this example, the sieve space is

    \begin{aligned} &\mathcal F_t =\Bigl\\{f: f= c_uu +\sum_{\ell=1}^{d_t-1}c_\ell\sin(\ell x),\\ (c_u,c_1,\ldots,c_{d_t-1})\in\mathbb R^{d_t} \Bigr\\}. \end{aligned}

    Thus, the corresponding feature map is ϕ(t)(x,u)=(u,sin⁡x,…,sin⁡((dt−1)x))⊤\phi^{(t)}(x,u) =(u,\sin x,\ldots,\sin((d_t-1)x))^\top. In fact, without loss of generality, we can extend this example to the more general form. Let [a,b][a,b] be any bounded interval with a<ba < b, and set ξ(x):=2πb−a(x−a+b2)\xi(x):=\frac{2\pi}{b-a}\bigl(x-\frac{a+b}{2}\bigr). If f(x,u)=b⋆u+g(x)f(x,u)=b_\star u+g(x) with g∈C0,α([a,b])g\in C^{0,\alpha}([a,b]) for some 1/2<α≤11/2 < \alpha\leq1, then ff admits the following representation on [a,b][a,b]:

�0�

+\sum_{\ell=1}^{\infty} \bigl

�1�

, \qquad x\in[a,b]. \end{aligned}$$

For the above ff, in the settings of some existing works, such as (Mania et al., 2022) and (Musavi et al., 2024), both cited in this paper, there must exist a known n~\widetilde{n}, such that for ℓ n~\ell \> \widetilde{n}, αℓ,∗=βℓ,⋆=α~ℓ,⋆=β~ℓ,⋆=0\alpha_{\ell,*}=\beta_{\ell,\star}=\widetilde{\alpha}_{\ell,\star}=\widetilde{\beta}_{\ell,\star}=0. However, assuming that such a known n~\widetilde{n} exists is restrictive (for example, such an n~\widetilde{n} does not exist for a linear system). This issue is a main motivation for this paper and a problem that we aim to solve.

  • In this paper, we allow that for any t∈[N−1]t\in[N-1], f∉Ftf\notin \mathcal{F}_t. This cause a difficulty: there exists a nonzero error ∣f−fθt∗∣\\|f-f_{\theta_t^*}\\| between fθ∗_tf_{\theta^*\_t} and ff, which is called the approximation error in statistics (naturally, in the existing works shown in the second point, the approximation error is zero). Such an approximation error makes the proof methods in existing works unable to be applied to this paper. Specifically, approximation error can bias the parameter estimates in the existing works using fixed feature maps, thereby leaving a gap between the estimated model and the true dynamics that cannot be eliminated. For example, under f=A⋆ϕf=A_\star\phi and sufficient excitation, (Mania et al., 2022), cited in this paper, show that the estimations of parameters A^jj=1∞\\{\widehat{A}_j\\}_{j=1}^{\infty} converge to A\*A_\*: A^j→PA⋆\widehat A_j\xrightarrow{\mathbb P}A_\star. However, if A⋆ϕA_\star\phi is a best uniform approximation with error ∣A⋆ϕ−f∣L∞≥Δ\\|A_\star\phi-f\\|_{L^\infty}\ge\Delta, the above convergence result no longer holds. Specifically, the prediction-error bound in (9) of (Mania et al., 2022), cited in this paper, is no longer guaranteed because approximation error can introduce an additional bias term.
AuthorsOpenReview ↗

Response to the Reviewer Auwc (3/7)

W2: The notation in the main body of the paper obscures the understanding. I’ve got lost after equation (4). I could not find the definition for MM. Similarly, I could not find the definition of Gτ\mathcal{G}_\tau. If these are used in the main text, they should be defined in the main text. In addition to that, it is difficult to understand why they have been defined and how one wants to use those.

R-W2: We apologize for the unclear notation and provide formal definitions of the notation used in the main text. Here are the definitions of MM and Gτ\mathscr{G}_\tau: MM denotes the update period introduced in Algorithm 1 of this paper. We update our estimator of ff every 2M2M steps. Gτ\mathscr{G}_\tau denotes the smallest σ\sigma-field generated by the random variables (x~0,u~0,…,x~τ)(\widetilde{x}_0,\widetilde{u}_0,\dots,\widetilde{x}_{\tau}).

W3: While the paper claims about general nonlinear dynamics, the method is only for nonlinear systems that are stabilizable by a linear state-feedback which is not a big class. In addition, when the dynamics are not known, one cannot know if they are stabilizable by linear state-feedback or not. The assumption cannot be verified for the considered problem formulation.

R-W3 (1/2): We would like to thank the reviewer for this insightful observation. The reply is shown below.

  • We understand this comment to refer to Assumption 1(b), which requires the existence of a linear state-feedback gain that globally stabilizes the system. Indeed, this assumption restricts the application area of our results. However, we just want to find a control policy K(x)K(x) satisfying (38) and (39) in Lemma 1 of this paper:

    v‾∣x∣22≤Vsta(x)≤v‾∣x∣22,Vsta(f(x,K(x)+v)+w)≤ρstaVsta(x)+bu∣v∣22+bw∣w∣22.\begin{aligned} &\underline v\\|x\\|_2^2 \le V_{\rm sta}(x) \le\overline v\\|x\\|_2^2,\\\\ &V_{\rm sta}\bigl(f(x,K(x)+v)+w\bigr) \le\rho_{\rm sta}V_{\rm sta}(x) +b_u\\|v\\|_2^2+b_w\\|w\\|_2^2. \end{aligned}

    We note that, under Assumption 1(b), the first stage of the proposed algorithm (lines 2–9) learns a linear feedback controller K(x)=K^xK(x)=\widehat{\mathbf K}x satisfying the above two conditions within N~\widetilde N steps, with probability at least 1−δconf1-\delta_{\rm conf}; see Proposition 2 and Lemma 1 of this paper. This is the motivation for proposing Assumption 1(b).

  • Thus, the key requirement for the stabilization stage is a feedback policy K(x)K(x), possibly nonlinear, satisfying (38) and (39). If such a policy is either known a priori or can be learned by an algorithm in finitely many steps, Assumption 1(b) is no longer needed for this stage. However, learning a stabilizing controller for an unknown nonlinear system is itself a challenging problem. Given the page limit and this paper's focus on online identification and regret analysis, we leave this problem to future work and provide the following studies as potential starting points for relaxing Assumption 1(b):

    [1] De Persis C, Rotulo M, Tesi P. Learning controllers from data via approximate nonlinearity cancellation[J]. IEEE Transactions on Automatic Control, 2023, 68(10): 6082-6097.

    [2] Luppi A, De Persis C, Tesi P. On data-driven stabilization of systems with nonlinearities satisfying quadratic constraints[J]. Systems & Control Letters, 2022, 163: 105206.

    [3] Strässer R, Schaller M, Worthmann K, et al. SafEDMD: A Koopman-based data-driven controller design framework for nonlinear dynamical systems[J]. Automatica, 2026, 185: 112732.

  • In fact, there are many existing works that have more restrictive assumptions and cannot be verified with unknown nonlinear systems. Concretely, some existing works directly impose stability conditions on the uncontrolled or zero-input dynamics. For example, Assumption 6.1 of Ziemann and Tu (2022), cited in this paper, requires that

    f0(x)=ϕ(A⋆x),ϕ(0)=0,Lip⁡(ϕ)≤1,A⋆⊤P⋆A⋆⪯ρP⋆,P⋆⪰I,0<ρ<1\begin{aligned} & f_0(x)=\phi(A_\star x),\\ \phi(0)=0,\\ \operatorname{Lip}(\phi)\leq 1,\\ A_\star^\top P_\star A_\star\preceq\rho P_\star, \\ P_\star\succeq I,\qquad 0 < \rho < 1 \end{aligned}

    indicating that ∣f0(x)∣P⋆≤∣A⋆x∣P⋆≤ρ,∣x∣P⋆\\|f_0(x)\\|_{P_\star}\leq\\|A_\star x\\|_{P_\star} \leq\sqrt{\rho}\\,\\|x\\|_{P_\star}; [4] assumes:

    f0(x):=f(x,0)=ϕ(Ax),ϕ(0)=0,Lip⁡(ϕ)≤1,∣A∣op<1,\begin{aligned} &f_0(x):=f(x,0)=\phi(Ax),\qquad \phi(0)=0,\qquad \operatorname{Lip}(\phi)\leq1,\qquad \\|A\\|_{\rm op} < 1, \end{aligned}

    indicating that ∣f0(x)∣2≤∣A∣op∣x∣2\\|f_0(x)\\|_2 \leq\\|A\\|_{\rm op}\\|x\\|_2; [5] assumes:

    f0(x)⊤Pf0(x)≤ρx⊤Px,P≻0,0≤ρ<1,\begin{aligned} &f_0(x)^\top P f_0(x) \leq\rho x^\top Px,\qquad P\succ0,\qquad 0\leq\rho < 1, \end{aligned}

    indicating that ∣f0(x)∣P≤ρ∣x∣P\\|f_0(x)\\|_P\leq\sqrt{\rho}\\|x\\|_P. These stability conditions are special cases of Assumption 1(b) of this paper with K⋆=0du×dxK_\star=\mathbf{0}_{d_u\times d_x} and suitable choices of the constants q1,q2q_1,q_2 and the Lyapunov matrix P\*P_\*. Such stability assumptions generally appear in the literature (see [4-6]), and we also cannot verify them when the system is unknown.

Reviewer_YszwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies online stochastic optimal control for unknown nonlinear dynamical systems with Gaussian noise of unknown covariance. The authors propose Gibbs-Randomized Model-Based Adaptive Control (GR-MBAC), a two-stage algorithm that first learns and certifies a stabilizing feedback controller and then improves control performance through sieve-based dynamics estimation, noise covariance estimation, and policy optimization.

The main theoretical contributions include finite-sample error bounds for dynamics and covariance estimation, stability guarantees, and a finite-horizon regret bound. Under suitable assumptions, the authors establish an \widetilde{O}(H^{-1/2}) average regret bound for linear quadratic regulators, nonlinear Fourier dynamics, and nonlinear transient price-impact models.

The theoretical results are complemented by numerical experiments on linear quadratic control, an inverted pendulum, and dynamic pricing and inventory control. The experiments compare the proposed method with existing adaptive control algorithms and demonstrate competitive performance in terms of cumulative regret and control costs.

Critical strengths
  1. Unified framework with rigorous theoretical guarantees.

The paper proposes a unified framework that integrates stability certification, nonlinear dynamics estimation, noise covariance estimation, and policy optimization. Unlike many existing approaches, the proposed method does not require a known stabilizing controller or an exact finite-dimensional representation of the dynamics. The finite-sample estimation bounds and regret guarantees provide a substantial theoretical contribution to online nonlinear control.

  1. Broad applicability and experimental validation.

The authors establish an \widetilde{O}(H^{-1/2}) average regret bound for several control problems, including linear quadratic regulators, nonlinear Fourier dynamics, and nonlinear transient price impact. Numerical experiments on linear and nonlinear control tasks further demonstrate the effectiveness of the proposed framework. The combination of theoretical guarantees and empirical validation strengthens the significance of the paper.

Critical weaknesses
  1. Restrictive stability assumptions

Although the paper aims to develop a unified framework for general nonlinear control systems, Assumption 1(b) requires the existence of a linear feedback controller that ensures global contraction with respect to a quadratic Lyapunov function. This is a strong restriction that excludes many nonlinear systems requiring nonlinear feedback or admitting only local stability guarantees. Consequently, the applicability of the proposed framework to general nonlinear control problems remains limited.

  1. Limited evaluation of computational efficiency and scalability

The proposed algorithm involves several computationally demanding procedures, including stability certification, Gibbs sampling, and finite-horizon policy optimization. Although the paper provides theoretical guarantees and numerical experiments, the computational complexity and scalability of these procedures are not sufficiently investigated. In particular, it remains unclear whether the proposed framework can be efficiently implemented for higher-dimensional nonlinear systems.

Questions for authors
  1. Novelty compared with existing model-based reinforcement learning methods.

The proposed framework appears to follow a standard model-based reinforcement learning approach, combining system identification, stabilization, and policy optimization. Could the authors clarify the main methodological and theoretical novelties compared with existing model-based reinforcement learning and adaptive control methods? In particular, which components are fundamentally new, rather than extensions or combinations of existing techniques?

  1. Significance of the theoretical guarantees.

The paper establishes an \widetilde{O}(H^{-1/2}) average regret bound for several control problems. Since similar regret rates have already been established in the literature, could the authors clarify the main theoretical improvements over existing results? In particular, which assumptions are substantially relaxed, and what new technical challenges are resolved by the proposed analysis?

14 comments
AuthorsOpenReview ↗

Response to the Reviewer Yszw (1/9)

We sincerely appreciate your time reviewing the manuscript and for your thoughtful feedback. We reply to your comments in the below.

W1. 1. Restrictive stability assumptions

Although the paper aims to develop a unified framework for general nonlinear control systems, Assumption 1(b) requires the existence of a linear feedback controller that ensures global contraction with respect to a quadratic Lyapunov function. This is a strong restriction that excludes many nonlinear systems requiring nonlinear feedback or admitting only local stability guarantees. Consequently, the applicability of the proposed framework to general nonlinear control problems remains limited.

R-W1 (1/2): We would like to thank the reviewer for raising this important point. The reply is shown below.

  • We acknowledge that Assumption 1(b) does restrict the class of systems covered by our analysis, such as the discrete-time Brockett integrator [1]. However, there also exist nonlinear systems satisfying Assumption 1(b), such as suitable discretized flexible-link robot models (Nugroho et al., 2021), cited in this paper, and discrete-time uncertain nonlinear systems with sector-bounded nonlinearities [2].

  • Compared with some assumptions requiring stable zero-input dynamics, Assumption 1(b) has less restriction. Concretely, some existing works directly impose stability conditions on the uncontrolled or zero-input dynamics. For example, Assumption 6.1 of Ziemann and Tu (2022), cited in this paper, requires that

    f0(x)=ϕ(A⋆x),ϕ(0)=0,Lip⁡(ϕ)≤1,A⋆⊤P⋆A⋆⪯ρP⋆,P⋆⪰I,0<ρ<1\begin{aligned} & f_0(x)=\phi(A_\star x),\\ \phi(0)=0,\\ \operatorname{Lip}(\phi)\leq 1,\\ A_\star^\top P_\star A_\star\preceq\rho P_\star, \\ P_\star\succeq I,\qquad 0 < \rho < 1 \end{aligned}

    indicating that ∣f0(x)∣P⋆≤∣A⋆x∣P⋆≤ρ,∣x∣P⋆\\|f_0(x)\\|_{P_\star}\leq\\|A_\star x\\|_{P_\star} \leq\sqrt{\rho}\\,\\|x\\|_{P_\star}; [3] assumes:

    f0(x):=f(x,0)=ϕ(Ax),ϕ(0)=0,Lip⁡(ϕ)≤1,∣A∣op<1,\begin{aligned} &f_0(x):=f(x,0)=\phi(Ax),\qquad \phi(0)=0,\qquad \operatorname{Lip}(\phi)\leq1,\qquad \\|A\\|_{\rm op} < 1, \end{aligned}

    indicating that ∣f0(x)∣2≤∣A∣op∣x∣2\\|f_0(x)\\|_2 \leq\\|A\\|_{\rm op}\\|x\\|_2; [4] assumes:

    f0(x)⊤Pf0(x)≤ρx⊤Px,P≻0,0≤ρ<1,\begin{aligned} &f_0(x)^\top P f_0(x) \leq\rho x^\top Px,\qquad P\succ0,\qquad 0\leq\rho < 1, \end{aligned}

    indicating that ∣f0(x)∣P≤ρ∣x∣P\\|f_0(x)\\|_P\leq\sqrt{\rho}\\|x\\|_P. These stability conditions are special cases of Assumption 1(b) of this paper with K⋆=0du×dxK_\star=\mathbf{0}_{d_u\times d_x} and suitable choices of the constants q1,q2q_1,q_2 and the Lyapunov matrix P\*P_\*.

  • A priori boundedness of the state-action trajectories provides an alternative to the trajectory control established through Assumption 1(b). Specifically, existing works, such as (Self et al., 2022), (Brown et al., 2024) and (Vinod et al., 2022), cited in this paper, assume that the state-action trajectories remain in fixed compact sets. Under this condition, we can directly remove Assumption 1(b), the component of our algorithm for identifying a linear state-feedback policy, and K^xi\widehat{\mathbf K}x_i in (3) of this paper, since the purpose of using K^\widehat{\mathbf K} and Assumption 1(b) is to ensure that the state-action trajectories grow slowly with high probability (see Lemma 3), which is implied by the boundedness condition above. Therefore, under this boundedness condition, Theorems 1–3 and Proposition 1 of this paper still hold.

  • We do not assume uniform boundedness of the state-action trajectories because this property depends on the iterates generated by the control policy and the system noise. Knowledge of the dynamics alone is therefore insufficient to verify it. For example, Theorem 4 of (Lu and Mo, 2025), cited in this paper, establishes the following bound for open-loop stable LQ systems under their adaptive controller:

    P!(∣xk∣2≤Cxlog⁡(k/δ) for all k≥1)≥1−2δ,\begin{aligned} &\mathbb P\\!\left( \\|x_k\\|_2\leq C_x\log(k/\delta) \text{ for all }k\geq1 \right)\geq1-2\delta, \end{aligned}

    for any δ∈(0,1/2]\delta\in(0,1/2]. This is a logarithmic growth bound, rather than a fixed compact-set guarantee.

AuthorsOpenReview ↗

Response to the Reviewer Yszw (2/9)

W1. Restrictive stability assumptions

Although the paper aims to develop a unified framework for general nonlinear control systems, Assumption 1(b) requires the existence of a linear feedback controller that ensures global contraction with respect to a quadratic Lyapunov function. This is a strong restriction that excludes many nonlinear systems requiring nonlinear feedback or admitting only local stability guarantees. Consequently, the applicability of the proposed framework to general nonlinear control problems remains limited.

R-W1 (2/2):

  • In fact, Assumption 1(b) can also be relaxed. We just want to find a control policy K(x)K(x) satisfying (38) and (39) in Lemma 1 of this paper:

    v‾∣x∣22≤Vsta(x)≤v‾∣x∣22,Vsta(f(x,K(x)+v)+w)≤ρstaVsta(x)+bu∣v∣22+bw∣w∣22.\begin{aligned} &\underline v\\|x\\|_2^2 \le V_{\rm sta}(x) \le\overline v\\|x\\|_2^2,\\\\ &V_{\rm sta}\bigl(f(x,K(x)+v)+w\bigr) \le\rho_{\rm sta}V_{\rm sta}(x) +b_u\\|v\\|_2^2+b_w\\|w\\|_2^2. \end{aligned}

We note that, under Assumption 1(b), the first stage of the proposed algorithm (lines 2–9) learns a linear feedback controller K(x)=K^xK(x)=\widehat{\mathbf K}x satisfying the above two conditions within N~\widetilde N steps, with probability at least 1−δconf1-\delta_{\rm conf}; see Proposition 2 and Lemma 1 of this paper. This is the motivation for proposing Assumption 1(b).

  • Thus, the key requirement for the stabilization stage is a feedback policy K(x)K(x), possibly nonlinear, satisfying (38) and (39). If such a policy is either known a priori or can be learned by an algorithm in finitely many steps, Assumption 1(b) is no longer needed for this stage. However, learning a stabilizing controller for an unknown nonlinear system is itself a challenging problem. Given the page limit and this paper's focus on online identification and regret analysis, we leave this problem to future work and provide the following studies as potential starting points for relaxing Assumption 1(b):

    [5] De Persis C, Rotulo M, Tesi P. Learning controllers from data via approximate nonlinearity cancellation[J]. IEEE Transactions on Automatic Control, 2023, 68(10): 6082-6097.

    [6] Luppi A, De Persis C, Tesi P. On data-driven stabilization of systems with nonlinearities satisfying quadratic constraints[J]. Systems \\& Control Letters, 2022, 163: 105206.

    [7] Strässer R, Schaller M, Worthmann K, et al. SafEDMD: A Koopman-based data-driven controller design framework for nonlinear dynamical systems[J]. Automatica, 2026, 185: 112732.

[1] Altafini C. Involutive flows and discretization errors for nonlinear driftless control systems[J]. Systems & Control Letters, 2017, 110: 29-37.

[2] Zhang M, Fujisaki Y. Parametric Quadratic Stabilization of a Class of Discrete-Time Uncertain Nonlinear Systems[C]//Proceedings of the ISCIE International Symposium on Stochastic Systems Theory and its Applications. The ISCIE Symposium on Stochastic Systems Theory and Its Applications, 2025, 2026: 71-74.

[3] Oymak S. Stochastic gradient descent learns state equations with nonlinear activations[C]//conference on Learning Theory. PMLR, 2019: 2551-2579.

[4] Foster D, Sarkar T, Rakhlin A. Learning nonlinear dynamical systems from a single trajectory[C]//Learning for Dynamics and Control. PMLR, 2020: 851-861.

AuthorsOpenReview ↗

Response to the Reviewer Yszw (3/9)

W2. Limited evaluation of computational efficiency and scalability

The proposed algorithm involves several computationally demanding procedures, including stability certification, Gibbs sampling, and finite-horizon policy optimization. Although the paper provides theoretical guarantees and numerical experiments, the computational complexity and scalability of these procedures are not sufficiently investigated. In particular, it remains unclear whether the proposed framework can be efficiently implemented for higher-dimensional nonlinear systems.

R-W2 (1/2): We would like to thank the reviewer for this comment. The reply is shown below.

  • We first analyze the computational complexity of each component in our algorithm to show the scalability of these components.

    • As described in Appendix B.1.1 of this paper, Explore\mathtt{Explore} and LearnCertify\mathtt{LearnCertify} involve only basic arithmetic operations (addition, subtraction, multiplication, and division). Their computational complexity grows linearly with the dimensions of xx and uu, and these steps therefore do not suffer from the curse of dimensionality.

    • As described in Appendix B.1.2 of this paper, CertifyGain\mathtt{CertifyGain} can be implemented using interval branch-and-bound method. Its computational cost can grow rapidly with the search dimension. However, computational efficiency can be improved through heuristic methods, such as constraint propagation (Neumaier, 2004), cited in this paper, and the maximum smear rule [8].

    • As shown in Assumption 4 of this paper, dt≤td_t\leq t. With the feature vectors and fpubf^{\rm pub} precomputed, evaluating fθf_\theta in SieveFit\mathtt{SieveFit} requires O(dxdt)O(d_xd_t) operations. Meanwhile, st(θ)s_t(\theta) is convex and smooth with respect to θ\theta. If Θt\Theta_t is convex, a Nesterov-type accelerated projected-gradient method [9] can be applied to compute θ^t\widehat\theta_t with accuracy ε 0\varepsilon \> 0 in

      O!(1+2∣∑i=0nt−1ϕi(t)(ϕi(t))⊤∣opdiam⁡(Θt)2ε)\begin{aligned} &O\\!\left( 1+\sqrt{ \frac{2\\|\sum_{i=0}^{n_t-1} \phi_i^{(t)}(\phi_i^{(t)})^\top\\|_{\rm op}\operatorname{diam}(\Theta_t)^2} {\varepsilon} }\right) \end{aligned}

      iterations. Each iteration requires O(tdxdt)O(td_xd_t) operations for a direct gradient calculation. Therefore, SieveFit\mathtt{SieveFit} does not suffer from the curse of dimensionality.

    • As described in Definition 4 of this paper, GreedyCover\mathtt{GreedyCover} constructs a cover Ct\mathcal C_t satisfying

      ∣Ct∣≤(1+2diam⁡(Θt)εcov,t)dxdt.\begin{aligned} &|\mathcal{C}_t|\leq \left( 1+\frac{2\operatorname{diam}(\Theta_t)} {\varepsilon_{{\rm cov},t}} \right)^{d_xd_t}. \end{aligned}

      We need to solve O(∣Ct∣)O(|\mathcal{C}_t|) global optimization subproblems. Given this cover and precomputed features, directly evaluating all weights in GibbsSample\mathtt{GibbsSample} requires O(∣Ct∣ntdxdt)O(|\mathcal{C}_t|n_td_xd_t) operations. Hence the computational cost in this component can grow rapidly with the search dimension. However, we can reduce the computational cost in practice by using heuristic methods, such as Algorithm Approx of [10] and Metropolis's method [11].

    • CovarianceUpdate\mathtt{CovarianceUpdate} involve only basic arithmetic operations and its computational cost is O(dxdt+dx2)O(d_xd_t+d_x^2). Therefore, CovarianceUpdate\mathtt{CovarianceUpdate} does not suffer from the curse of dimensionality.

    • As described in Appendix B.1.3 of this paper, the implementation of FiniteHorizon\mathtt{FiniteHorizon} is reduced to solving a sequence of least-squares problems (see lines 1183-1184 and Appendix E of this paper for the formal definition and analysis). These problems can also be solved using Nesterov-type accelerated projected-gradient method [9], with computational complexity growing linearly with the dimensions of xx and uu.

    • In the subsequent part of Algorithm 1 of this paper, there are just some arithmetic calculations. Therefore, the computational complexity of the subsequent part grows linearly with the dimensions of xx and uu.

    • In conclusion, our algorithm is computationally feasible for high-dimensional nonlinear systems.

Reviewer_4bH6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies online learning and control problems in a general form to relax existing assumptions on the functional structure of the nonlinear dynamics, boundedness of trajectories, and noise models. The method learns a certified state feedback, which is then used to refine the estimate of the dynamics. Different stages of the proposed algorithm are presented, and a numerical example at the end is given for comparison to selected methods, including a case of known basis functions and data-driven policy optimization.

Critical strengths

The paper presents a clear framework with well-defined goals and stages. The literature is properly reviewed, and theoretical comparisons are extensively reported in the paper.

Critical weaknesses
  • The assumption of a linear state feedback gain for global stabilization is restrictive and undermines the claims about the nonlinear system control and estimation. Please discuss the necessity of this assumption and if there is hope to relax it.
  • The steps and individual functions in the algorithm are not clearly defined and explained in the main text, which reduces readability. Please consider reorganizing to present the main components of the method in the main text.
  • The consistency between the different dynamics representations is not clearly presented, and different sections make varying assumptions about the function classes and additional assumptions. The paper would greatly improve by streamlining the modeling assumptions and representations used in the proposed method and how they fit the proposed algorithm as a whole, not just specific parts.
Questions for authors
  • The Introduction and preliminaries have lots of overlap and repetition in explaining the problem and shortcomings of the existing literature. Consider streamlining these sections for better readability.
  • The definition of the sieve at the beginning of the paper is not clear. Can you please provide a clear definition of the sieve in the preliminaries section?
  • Please discuss how restrictive it is to assume a linear state feedback policy when compared to the existing literature, especially in light of the shortcomings identified in the introduction.
  • Do you require any PE assumptions on the initial trajectories for identifying the initial state feedback? How is safety ensured in this stage under a probabilistic policy? Please clarify.
  • Do you not need any assumption on the boundedness or stability of f? Is it possible that f has finite escape with the existing assumptions? Please discuss.
  • Your definition of the function structure with the known + feature map representation recovers other methods from the existing literature, which you cited as having shortcomings due to the structure prefixing. How does your framework differ in this sense?
  • What assumptions are there on the feature selection and the functional form of the features? Do you require Lipschitz continuity, boundedness, smoothness, or similar assumptions?
  • Please clarify how a time-invariant stabilizing gain exists and is found for the Fourier nonlinear dynamics you propose in the appendix.
  • Please discuss the verifiability of the constants in the assumptions and the dimension dependencies in Theorem 1 and other main results to give an indication about how well the framework scales, especially for the claimed class of nonlinear dynamics.
  • What algorithm do you use for the specialist algorithm? And how do you choose the "general sieve class" bases in general? Is the example representative of a class of basis functions, or do you have a general approximator class? How was this class chosen?
  • Can you extend the numerical example to nonlinear systems to show the utility of the method against a baseline? Can you parametrize the problem to show statistical results on the transient performance of the different algorithms? How does the method perform with different general sieve functions, or do you claim something about the choice of functions?
13 comments
AuthorsOpenReview ↗

Response to the Reviewer 4bH6(1/11)

We sincerely appreciate your time reviewing the manuscript and for your thoughtful feedback. We reply to your comments below.

W1: The assumption of a linear state feedback gain for global stabilization is restrictive and undermines the claims about the nonlinear system control and estimation. Please discuss the necessity of this assumption and if there is hope to relax it.

R-W1:

  • We would like to thank the reviewer for pointing out this. We understand this comment to refer to Assumption 1(b), which requires the existence of a linear state-feedback gain that globally stabilizes the system. In fact, we just want to find a control policy K(x)K(x) satisfying (38) and (39) in Lemma 1 of this paper:

    v‾∣x∣22≤Vsta(x)≤v‾∣x∣22,Vsta(f(x,K(x)+v)+w)≤ρstaVsta(x)+bu∣v∣22+bw∣w∣22.\begin{aligned} &\underline v\\|x\\|_2^2 \le V_{\rm sta}(x) \le\overline v\\|x\\|_2^2,\\\\ &V_{\rm sta}\bigl(f(x,K(x)+v)+w\bigr) \le\rho_{\rm sta}V_{\rm sta}(x) +b_u\\|v\\|_2^2+b_w\\|w\\|_2^2. \end{aligned}

    We note that, under Assumption 1(b), the first stage of the proposed algorithm (lines 2–9) learns a linear feedback controller K(x)=K^xK(x)=\widehat{\mathbf K}x satisfying the above two conditions within N~\widetilde N steps, with probability at least 1−δconf1-\delta_{\rm conf}; see Proposition 2 and Lemma 1 of this paper. This is the motivation for proposing Assumption 1(b).

  • Thus, the key requirement for the stabilization stage is a feedback policy K(x)K(x), possibly nonlinear, satisfying (38) and (39). If such a policy is either known a priori or can be learned by an algorithm in finitely many steps, Assumption 1(b) is no longer needed for this stage. However, learning a stabilizing controller for an unknown nonlinear system is itself a challenging problem. Given the page limit and this paper's focus on online identification and regret analysis, we leave this problem to future work and provide the following studies as potential starting points for relaxing Assumption 1(b):

    [1] De Persis C, Rotulo M, Tesi P. Learning controllers from data via approximate nonlinearity cancellation[J]. IEEE Transactions on Automatic Control, 2023, 68(10): 6082-6097.

    [2] Luppi A, De Persis C, Tesi P. On data-driven stabilization of systems with nonlinearities satisfying quadratic constraints[J]. Systems \\& Control Letters, 2022, 163: 105206.

    [3] Strässer R, Schaller M, Worthmann K, et al. SafEDMD: A Koopman-based data-driven controller design framework for nonlinear dynamical systems[J]. Automatica, 2026, 185: 112732.

W2: The steps and individual functions in the algorithm are not clearly defined and explained in the main text, which reduces readability. Please consider reorganizing to present the main components of the method in the main text.

R-W2: We apologize for the unclear definitions of the individual functions and steps in our algorithm. The formal description of the algorithm and the definitions of the individual functions have been added to the main text of this paper.

W3: The consistency between the different dynamics representations is not clearly presented, and different sections make varying assumptions about the function classes and additional assumptions. The paper would greatly improve by streamlining the modeling assumptions and representations used in the proposed method and how they fit the proposed algorithm as a whole, not just specific parts.

R-W3: We apologize for the chaotic assumptions for different dynamics. We have reorganized Appendix D and added tables in Appendix D to explicitly clarify how the dynamics and assumptions for each system specialization correspond to the general dynamics and assumptions in the main text.

Q1: The Introduction and preliminaries have lots of overlap and repetition in explaining the problem and shortcomings of the existing literature. Consider streamlining these sections for better readability.

R-Q1: We would like to thank the reviewer for this comment. We have removed the overlapping parts in the introduction and preliminaries to improve readability.

AuthorsOpenReview ↗

Response to the Reviewer 4bH6 (2/11)

Q2: The definition of the sieve at the beginning of the paper is not clear. Can you please provide a clear definition of the sieve in the preliminaries section?

R-Q2: We apologize for the unclear definition of sieve. We have added an accurate definition of the sieve space Ft\mathcal{F}_t in the preliminary part. The definition is shown as follows.

Sieve spaces. For each update time tt, define the sieve space

\begin{aligned} &\mathcal F_t :=\left\\{ f_\theta:\\; f_\theta(x,u)=f^{\rm pub}(x,u) +\theta\phi^{(t)}(x,u),\\; \theta\in\Theta_t \right\\}, \end{aligned}

where fpub:Rdx+du→Rdxf^{\rm pub}:\mathbb{R}^{d_x+d_u}\to\mathbb{R}^{d_x} is a known component of the dynamics, ϕ(t)=(ϕ1(t),…,ϕdt(t))⊤\phi^{(t)}=(\phi_1^{(t)},\ldots,\phi_{d_t}^{(t)})^\top is a prescribed feature map, and Θt⊆Rdx×dt\Theta_t\subseteq\mathbb R^{d_x\times d_t} is a compact coefficient set. We use the sequence {Ft}t=1N−1\{\mathcal F_t\}_{t=1}^{N-1} to approximate the unknown transition function ff. The number of features dtd_t can increase with tt.

AuthorsOpenReview ↗

Response to the Reviewer 4bH6 (3/11)

Q3: Please discuss how restrictive it is to assume a linear state feedback policy when compared to the existing literature, especially in light of the shortcomings identified in the introduction.

R-Q3: Thanks for this comment. In Assumption 1(b) of this paper, we require that the system can be stabilized by a linear state-feedback policy. We discuss this assumption below.

  • Assumption 1(b) restricts the class of systems covered by our analysis, such as the discrete-time Brockett integrator [4]. However, there also exist nonlinear systems satisfying Assumption 1(b), such as suitable discretized flexible-link robot models (Nugroho et al., 2021), cited in this paper, and discrete-time uncertain nonlinear systems with sector-bounded nonlinearities [5].

  • Compared with assumptions requiring stable zero-input dynamics, Assumption 1(b) also allows systems whose zero-input dynamics are unstable. Concretely, some existing works directly impose stability conditions on the uncontrolled or zero-input dynamics. For example, Assumption 6.1 of Ziemann and Tu (2022), cited in this paper, requires that

    f0(x)=ϕ(A⋆x),ϕ(0)=0,Lip⁡(ϕ)≤1,A⋆⊤P⋆A⋆⪯ρP⋆,P⋆⪰I,0<ρ<1\begin{aligned} & f_0(x)=\phi(A_\star x),\\ \phi(0)=0,\\ \operatorname{Lip}(\phi)\leq 1,\\ A_\star^\top P_\star A_\star\preceq\rho P_\star, \\ P_\star\succeq I,\qquad 0 < \rho < 1 \end{aligned}

    indicating that ∣f0(x)∣P⋆≤∣A⋆x∣P⋆≤ρ,∣x∣P⋆\\|f_0(x)\\|_{P_\star}\leq\\|A_\star x\\|_{P_\star} \leq\sqrt{\rho}\\,\\|x\\|_{P_\star}; [6] assumes:

    f0(x):=f(x,0)=ϕ(Ax),ϕ(0)=0,Lip⁡(ϕ)≤1,∣A∣op<1,\begin{aligned} &f_0(x):=f(x,0)=\phi(Ax),\qquad \phi(0)=0,\qquad \operatorname{Lip}(\phi)\leq1,\qquad \\|A\\|_{\rm op} < 1, \end{aligned}

    indicating that ∣f0(x)∣2≤∣A∣op∣x∣2\\|f_0(x)\\|_2 \leq\\|A\\|_{\rm op}\\|x\\|_2; [7] assumes:

    f0(x)⊤Pf0(x)≤ρx⊤Px,P≻0,0≤ρ<1,\begin{aligned} &f_0(x)^\top P f_0(x) \leq\rho x^\top Px,\qquad P\succ0,\qquad 0\leq\rho < 1, \end{aligned}

    indicating that ∣f0(x)∣P≤ρ∣x∣P\\|f_0(x)\\|_P\leq\sqrt{\rho}\\|x\\|_P. These stability conditions are special cases of Assumption 1(b) of this paper with K⋆=0du×dxK_\star=\mathbf{0}_{d_u\times d_x} and suitable choices of the constants q1,q2q_1,q_2 and the Lyapunov matrix P\*P_\*.

  • A priori boundedness of the state-action trajectories provides an alternative to the trajectory control established through Assumption 1(b). Specifically, existing works, such as (Self et al., 2022), (Brown et al., 2024) and (Vinod et al., 2022), cited in this paper, assume that the state-action trajectories remain in fixed compact sets. Under this condition, we can directly remove Assumption 1(b), the component of our algorithm for identifying a linear state-feedback policy, and K^xi\widehat{\mathbf K}x_i in (3) of this paper, since the purpose of using K^\widehat{\mathbf K} and Assumption 1(b) is to ensure that the state-action trajectories grow slowly with high probability (see Lemma 3), which is implied by the boundedness condition above. Therefore, under this boundedness condition, Theorems 1–3 and Proposition 1 of this paper still hold.

  • We do not assume uniform boundedness of the state-action trajectories because this property depends on the iterates generated by the control policy and the system noise. Knowledge of the dynamics alone is therefore insufficient to verify it. For example, Theorem 4 of (Lu and Mo, 2025), cited in this paper, establishes the following bound for open-loop stable LQ systems under their adaptive controller:

    P!(∣xk∣2≤Cxlog⁡(k/δ) for all k≥1)≥1−2δ,\begin{aligned} &\mathbb P\\!\left( \\|x_k\\|_2\leq C_x\log(k/\delta) \text{ for all }k\geq1 \right)\geq1-2\delta, \end{aligned}

    for any δ∈(0,1/2]\delta\in(0,1/2]. This is a logarithmic growth bound, rather than a fixed compact-set guarantee.

[4] Altafini C. Involutive flows and discretization errors for nonlinear driftless control systems[J]. Systems & Control Letters, 2017, 110: 29-37.

[5] Zhang M, Fujisaki Y. Parametric Quadratic Stabilization of a Class of Discrete-Time Uncertain Nonlinear Systems[C]//Proceedings of the ISCIE International Symposium on Stochastic Systems Theory and its Applications. The ISCIE Symposium on Stochastic Systems Theory and Its Applications, 2025, 2026: 71-74.

[6] Oymak S. Stochastic gradient descent learns state equations with nonlinear activations[C]//conference on Learning Theory. PMLR, 2019: 2551-2579.

[7] Foster D, Sarkar T, Rakhlin A. Learning nonlinear dynamical systems from a single trajectory[C]//Learning for Dynamics and Control. PMLR, 2020: 851-861.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.