acceptodds
Under review as a conference paper at ICLR 2027

The Improvement Guarantee of Learning MPC Leaks on Nonlinear Systems, and How to Bound It

Abstract

Learning Model Predictive Control (LMPC) improves at a repetitive task by storing executed trajectories and reusing their realized cost-to-go as terminal cost. Its improvement guarantee is proved on the stored states, but a nonlinear closed loop only ever reaches a tolerance-neighbourhood of them, so the guarantee must survive whatever extends the value off the store. We define the consistency leak, the amount by which one step of true progress can fail to buy one step of promised value, and prove that the guarantee degrades by at most the leak per step. Measured on stores that a minimum-time LMPC loop generates for itself, the position-indexed constructions the literature runs (convex safe sets, nearest-neighbour lookups, a learned value) leak more as experience accumulates: within nine iterations the improvement bound exceeds the cost it bounds. State-aware constructions keep the leak bounded but expose almost none of the stored improvement. The remedy is to take the nearest neighbour within each stored trajectory and then the minimum over trajectories; its leak is bounded by plant and design constants alone. In closed loop the per-trajectory minimum completes all 176 iterations across two plants while the pooled lookup completes fewer than half, at 0.58-2.2x the cost per MPC solve. Code is available at https://anonymous.4open.science/r/lmpcleak-54F8.

Primary area:
reinforcement learning
Keywords:
learning model predictive control, iterative learning control, model predictive control, safe sets, terminal cost, improvement guarantee, nonlinear systems, minimum-time control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YK5VOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies Learning MPC (LMPC) on nonlinear systems. LMPC's improvement guarantee is proved for a terminal state that lies exactly on a stored state. The authors argue that implementations only reach a neighborhood of the stored states, so the guarantee depends on how the terminal cost is extended off the store. They define a "consistency leak," a one-step Bellman residual, and show that iteration cost can increase by at most the leak times the iteration length. They then compare several terminal-cost constructions and propose a per-trajectory minimum: take the nearest neighbor within each stored trajectory, then the minimum over trajectories. Experiments on a kinematic bicycle, a planar quadrotor and a MuJoCo quadrotor report better completion and cost than a pooled nearest-neighbor lookup.

Critical strengths

The proposed per-trajectory minimum is simple, computationally cheap and easy to implement. The paper correctly points out that relaxed nonlinear LMPC implementations have no improvement guarantee. The authors mention limitations and report negative and null results. The experiments include sanity checks and paired statistical tests.

Critical weaknesses

The paper's language does not follow the usual tone and standards in the field, with lots of new terms and definitions introduced ad hoc, which makes it difficult to connect to the theoretical development already existing in the field. This reduces readability. The authors should do a major overhaul to the presentation and framing of the paper in light of existing works in the literature and standard terminology. Already in the opening section, "promising exactly what the controlled vehicle (the ego) is already doing; such a value is inert" is difficult to understand and quite uncommon in similar scientific writing in the field. The paper has numerous such instances where the reader is lost.

The original body of work uses the safe set points as terminal conditions on the optimization problem and does not require reaching the exact state. The argument about reaching a specific state being a measure zero event is difficult to argue as this does not mean the state is unreachable but rather changes the nature of the required optimization problem (using integer variables) and the original proof relies on reaching the set, which is always feasible by construction. Points about relaxations losing some of the guarantees already appear in published work; therefore, the niche presented in this paper is not clear. The problem claimed in the submission is not clear and not consistent with the published body of work, severely undermining the contribution of the paper.

The optimal control problem posed in (4) is not the original LMPC algorithm, but a relaxation of it on the terminal constraints, which results in the leak structure the authors are posing. Therefore, the authors should frame their claimed contributions carefully as the approximation and the recovered solution for exact penalization are not well founded. The original problem poses a mixed-integer nature, forcing the trajectories to reach one of those in the safe set, which is always possible by construction, so the presentation in the submission seems somewhat contrived. Do the authors take a relaxed practical implementation of LMPC and study the suboptimality of this method with respect to an ideal one? This point is not clear and the rest of the paper becomes confusing due to it.

The benefit of the presented work over the existing body of work is not clear and the authors do not justify the improvement over the baselines sufficiently.

Questions for authors
  • The definition of p() in Definition 2 is not clear.
  • Please define where problem (4) is taken from, or whether you have developed this solution concept yourselves. By allowing a distance to the SS, you are indeed relaxing the problem and changing the LMPC structure and Algorithm 1 does not reflect the original algorithm. Please explain this choice and what it means for the rest of the paper.
  • The statement in Section 4.2 about how the terminal set is enforced in the original LMPC work is incorrect. The authors should either point out to how and where in the cited paper this relaxation is used with the stated theoretical results or revise the paper significantly to reword their developments.
  • Can you clarify the definitions of nonstandard terminology of lap, circuit, etc., which are difficult to follow and are used inconsistently in some places?
  • Please compare your proposed method to the exact formulation on the original paper for the experimental results and report the performance differences. How did you verify the exactness of your solutions with the penalization method in the implementations?
Reviewer_jPBQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the theory with the learning MPC on nonlinear where the terminal state can only reach a neighborhood of the stored states. The consistency leak is the main theoretical results of the paper, which corresponds the amount by which one step of true progress can fail to buy one step of promised value. The paper finds the degrades of convex safe sets, nearest-neighbour lookups, a learned value during the iteration. Closed-loop experiments on a kinematic bicycle and a planar quadrotor support the theoretical findings.

Critical strengths

The paper combines theory, analysis and experiments, and evaluates a broad range of terminal-value constructions used in the LMPC literature. The results illustrate a clear trade-off: constructions that keep the leak bounded tend to expose little of the improvement contained in the stored data.

The theorical results are important in the LMPC fields. The consistency leak provides a measurable quantity that can serve as a diagnostic index for comparing terminal-value constructions.

The experimental design supports the theoretical findings well: the leak is measured on stores generated by the LMPC loop itself.

Critical weaknesses

For such theoretical paper, a clear figure to present the abstract indexes including sampled safe set, consistency leak, store is required.

The proposed remedy is a small change to how the terminal value is looked up. Not sufficient as a large contribution. The problem also lies in consistency leaks, which of course a nicely designed index, but it is not used in the controller itself and cannot be computed exactly.

Questions for authors

Is the consistency leak intended only as an offline diagnostic, or can it also be used within the controller?

Can you provide which experiments directly support the main claim and move the secondary studies out of the main discussion? The experiment’s part contains too much information.

The consistency leak is not evaluated in every setting. In particular, it is never measured for the MuJoCo quadrotor.

Reviewer_L8EAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the reason why iterative improvement guarantees of LMPC may fail when the realized cost-to-go is evaluated in a neighborhood of the stored trajectories. It defines a one-step consistency leak as an upper bound measure of the performance degradation, and proposes to take the minimum of nearest-neighbor values as a remedy. Simulations show substantially better performance (completion rate and cost) on two representative tasks.

Critical strengths
  1. The topic is interesting and important. The paper identifies a critical gap between theoretical guarantee and practical implementations, in that the "leak" might propagate over time.
  2. The formulation of the consistency leak, along with the upper bound, is intuitive, and the theoretical properties (consistency, drive, achievability) provide useful ways to identify potential issues with the terminal values.
  3. The numerical simulations are extensive, and the results are presented in a candid way. Limits like weak numerical certificate are honestly reported.
Critical weaknesses
  1. The theoretical analysis assumes exact optimization, which is infeasible for practical purposes (SLSQP is used in the simulations, the optimality guarantee of which is pretty weak). If technically possible, a theoretical discussion regarding how optimization errors may accumulate over time and affect the results would be beneficial.
Questions for authors
  1. Remark 2 discusses how Assumption 2 may fail and how it could be settled by phase-indexed candidates. I'm just wondering if this fix is universal, or if such failure might not be fixable in certain cases.
  2. In the simulations, the reported leak estimates are based on sampled statistics rather than the supremum used in the definition. It seems that this leads to wider bands and bad numerical certificates. I wonder if there are ways to make this a little tighter to be practically more informative.
  3. In Appendix C, the upper bound of ηT\eta_T seems to be used directly as a proxy for it ("that is a functional form, not merely a ceiling). I wonder why this argument is valid.
  4. A few typos/errors are observed:
    • Page 9, line 485: "extrapolations in τ\tau, statistics of the instrument" is not intelligible.
    • Appendix F, line 1004: it says only those results with p≤0.005p \leq 0.005 are load-bearing. I wonder if the MuJoCo result (p=0.0078p=0.0078, in Table 2) is valid under this premise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.