- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies optimal control along with learning from demonstration (teacher). The authors consider nonlinear systems, noise-free and in continuous representation (instead of the common discrete representation of the dynamical systems). Even though the dynamical system is nonlinear, it is affine with respect to control input and obeys . It is assumed that is known but is unknown. The authors learn only the value function for the optimal control problem as the greedy policy is given thanks to a known B and smooth V. The loss function minimizes a linear combination of the typical TD loss and a loss between the optimal policy and teacher actions which brings the paper into the category of RL+BC.
While the paper is timely and well written my recommendation is reject. The reason is that the authors missed seeing this work as RL+BC and thus are unaware of similar works on this topic. This makes their paper a very small subclass of solutions. Moreover, the continuous formulation brings unnecessary complication and limited applicability which could be avoided from very beginning.
Critical strengths
The topic ``RL+BC’’ is a timely. The paper is theoretical and prove the results instead of numerical evaluation. To me the theoretical proofs are sound. The paper is simple, easy to read and well organized.
Critical weaknesses
While this paper is on RL+BC, this topic has not been literature reviewed. There are many approaches on this topic including but not limited to IQL, CQL, TD3+BC, Flow Q learning to name a few. They are not discussed in the paper.
Similarly these approaches have not been implemented in the simulation results.
Seeing the paper under RL+BC umbrella, magnifies the pitfall in the methodology in this paper even more. This paper aims to learn a deterministic policy while the typical formulation in RL+BC allows for a stochastic policy that has a deterministic policy as a special case.
The third major flaw is formulation in continuous domain. Three issues follow:
- Authors selected continuous formulation and then they needed to sample the trajectory to follow the teacher trajectory in the loss. This brings the discretization error into the picture and authors try to bound it.
- Using continuous formulation, the authors need to know B(x) which could be avoided by having a discrete formulation and learning function.
- V should be continuous so \grad V exists however the conditions on the dynamics guaranteeing that are not discussed so one cannot make sure that if a continuous V exists.
Another limiting factor is considering quadratic cost. This limits the class of problem a lot. Note that replacing quadratic costs with convex costs is well established in the control and learning community these days.
In (7) and (8) there are A(x) and B(x) matrices that are functions of x, similarly u is a function of x and the authors simply assumed that u is admissible. How to conclude that one? Of course it holds if A and B are NOT function of the state but such a statement does not hold unless you cite the relevant theorem.
Questions for authors
Why do you parameterize the value function as (12) and not other forms? I understand that you would like to have quadratic structure at the origin but they are many other ways to that.
Equation (19) is not mathematically correct as r_h is a function of x_i and u_i but only x_i is mentioned in the argument. Note that (19) is different from (18) where u_s is explicitly a function of x.
I could not find the details related to C in Theorem 4.2. It is good to discuss how sharps are the bounds in the theorems. Is Theorem 4.4 worth a theorem really? To me it is the normal linearization available in the control textbooks. The abstract starts with “SDRE” which is unknown to the reader. How to read the ratio in the simulation results? To me other methods are also good with respect to this criteria. The CPU time is better than SDRE but worse than QRnet. The authors have not acknowledged this.
In the conclusion, authors mention that they provide simulation results on three high dimensional systems while only one system is of dimension 20 and others are 1 and 2.