acceptodds
Withdrawn from ICLR 2027

Teacher-Policy Value Identification from Control Demonstrations with Teacher-Relative Performance Guarantees

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

SDRE controllers are useful teachers for high-dimensional nonlinear systems, but their algebraic value surrogate need not coincide with the infinite-horizon cost-to-go realized by the resulting policy. We propose a control-supervised approach to learn this teacher-policy value from held-input transitions and control labels, without value labels or statewise drift evaluations during training. A single neural value function, trained by one-step policy-value consistency and control supervision, yields both an estimate of the teacher-policy value and a gradient-defined controller that avoids per-state Riccati solves at deployment. We prove that a uniform one-step residual identifies the realized cost-to-go; Through a completion-of-the-square identity, value consistency and control supervision bound the HJB defect. For bounded finite-energy trajectories, the deployed cost and control deviation are bounded by the teacher cost plus residual/discretization errors, without optimal references or pointwise imitation. LQR anchoring ensures local exponential stability for all network parameters, independent of training accuracy. The method eliminates drift knowledge, accelerates SDRE deployment, and offers provable value recovery, HJB consistency, and stability. Experiments on 40D Cucker–Smale, 51D Allen–Cahn, and 64D Burgers confirm stable closed-loop performance and rollout-value recovery while replacing online Riccati solves with amortized neural feedback.

Primary area:
reinforcement learning
Keywords:
Nonlinear control, value function learning, control demonstrations

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Jyb3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies optimal control along with learning from demonstration (teacher). The authors consider nonlinear systems, noise-free and in continuous representation (instead of the common discrete representation of the dynamical systems). Even though the dynamical system is nonlinear, it is affine with respect to control input and obeys x˙=f(x)+B(x)u\dot{x}=f(x)+B(x)u. It is assumed that B(x)B(x) is known but f(x)f(x) is unknown. The authors learn only the value function for the optimal control problem as the greedy policy is given thanks to a known B and smooth V. The loss function minimizes a linear combination of the typical TD loss and a loss between the optimal policy and teacher actions which brings the paper into the category of RL+BC.

While the paper is timely and well written my recommendation is reject. The reason is that the authors missed seeing this work as RL+BC and thus are unaware of similar works on this topic. This makes their paper a very small subclass of solutions. Moreover, the continuous formulation brings unnecessary complication and limited applicability which could be avoided from very beginning.

Critical strengths

The topic ``RL+BC’’ is a timely. The paper is theoretical and prove the results instead of numerical evaluation. To me the theoretical proofs are sound. The paper is simple, easy to read and well organized.

Critical weaknesses

While this paper is on RL+BC, this topic has not been literature reviewed. There are many approaches on this topic including but not limited to IQL, CQL, TD3+BC, Flow Q learning to name a few. They are not discussed in the paper.

Similarly these approaches have not been implemented in the simulation results.

Seeing the paper under RL+BC umbrella, magnifies the pitfall in the methodology in this paper even more. This paper aims to learn a deterministic policy while the typical formulation in RL+BC allows for a stochastic policy that has a deterministic policy as a special case.

The third major flaw is formulation in continuous domain. Three issues follow:

  • Authors selected continuous formulation and then they needed to sample the trajectory to follow the teacher trajectory in the loss. This brings the discretization error into the picture and authors try to bound it.
  • Using continuous formulation, the authors need to know B(x) which could be avoided by having a discrete formulation and learning QQ function.
  • V should be continuous so \grad V exists however the conditions on the dynamics guaranteeing that are not discussed so one cannot make sure that if a continuous V exists.

Another limiting factor is considering quadratic cost. This limits the class of problem a lot. Note that replacing quadratic costs with convex costs is well established in the control and learning community these days.

In (7) and (8) there are A(x) and B(x) matrices that are functions of x, similarly u is a function of x and the authors simply assumed that u is admissible. How to conclude that one? Of course it holds if A and B are NOT function of the state but such a statement does not hold unless you cite the relevant theorem.

Questions for authors

Why do you parameterize the value function as (12) and not other forms? I understand that you would like to have quadratic structure at the origin but they are many other ways to that.

Equation (19) is not mathematically correct as r_h is a function of x_i and u_i but only x_i is mentioned in the argument. Note that (19) is different from (18) where u_s is explicitly a function of x.

I could not find the details related to C in Theorem 4.2. It is good to discuss how sharps are the bounds in the theorems. Is Theorem 4.4 worth a theorem really? To me it is the normal linearization available in the control textbooks. The abstract starts with “SDRE” which is unknown to the reader. How to read the ratio in the simulation results? To me other methods are also good with respect to this criteria. The CPU time is better than SDRE but worse than QRnet. The authors have not acknowledged this.

In the conclusion, authors mention that they provide simulation results on three high dimensional systems while only one system is of dimension 20 and others are 1 and 2.

Reviewer_bdtgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the teacher-policy value identification problem for SDRE controllers in high-dimensional nonlinear systems. The authors propose a control-supervised approach that learns this teacher-policy value from held-input transitions and control labels, requiring no value labels or statewise drift evaluations during training.

Critical strengths
  1. The paper clearly distinguishes the algebraic value surrogate constructed from SDRE solutions from the infinite-horizon cost-to-go realized by the closed-loop policy.
  2. Theorems 4.1–4.3 form a complete non-asymptotic chain of results, and the stated assumptions candidly acknowledge the limitations of the theory.
  3. The simulation results on the three systems are quantitatively consistent with the theoretical predictions.
Critical weaknesses
  1. For nonlinear systems, approximate dynamic programming (ADP) is a mature family of methods developed over decades. The paper provides neither a survey of this line of work nor a clear account of the main reasons for studying SDRE and its advantages over alternatives. In my view, approximating the optimal value function as x^{\tr}P(x)x also lacks theoretical justification.
  2. The learning mechanisms employed, such as one-step temporal-difference learning, are combinations of known components, and Equations (1)–(14) collect existing results. The novelty of the contribution is therefore not clearly established.
  3. The baselines are insufficient. The paper lacks comparisons with classical ADP and RL algorithms.
Questions for authors
  1. How accurate is the optimal control policy recovered by the proposed algorithm? The authors could evaluate the method on nonlinear optimal control problems that admit analytic solutions.
  2. What are the specific dynamics of the three case studies considered?
  3. The step size h differs considerably across the three case studies. In view of the O(h) discretization terms and the step-size bound h_0 in the theorems, how should h be selected in practice on the basis of both the system and the theory?
Reviewer_CQzwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a control-supervised neural approach that learns the value function and feedback policy of SDRE controllers without value labels or online Riccati solves. Theoretical guarantees establish value identification, performance bounds, and local stability, while experiments on several high-dimensional nonlinear systems demonstrate stable performance and accurate value recovery.

Critical strengths
  1. The theory seems to be solid.
Critical weaknesses
  1. My main concern is that this work falls primarily within the scope of classical control theory, and its relevance to a broader machine learning audience appears limited. Most of the references are drawn from the control literature. Moreover, in the experiments, the only neural-network-related baseline is also taken from a control journal. The authors should better clarify the paper’s relevance and potential contributions to the ML community and provide stronger comparisons with methods from the ML literature.

  2. The opening statement about optimal control theory is inaccurate, particularly given the repeated use of “must.” Quadratic regularization represents only a specific class of objectives in optimal control. Many optimal control problems do not require the state to converge or decay; in some applications, sustained oscillations or even growth of the state may be desirable (e.g., certain economic models). I suggest revising this statement to avoid an unnecessarily restrictive characterization of optimal control.

  3. There is a substantial gap between the theoretical results and the empirical evaluation. In particular, the experiments do not appear to verify whether the hypotheses of the main theorems are satisfied. This makes it difficult to assess to what extent the theoretical results actually explain or support the observed empirical behavior.

Questions for authors

see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.