- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Cost-Aware Adaptive Conformal Inference (ACI) for runtime assurance of dynamical systems with unknown and potentially time-varying dynamics. The key motivation is that standard ACI adapts its uncertainty margin using a binary miscoverage signal and therefore does not distinguish between mild and severe constraint violations. The proposed method addresses this limitation by introducing a cost-aware loss that weights miscoverage events according to the severity of the resulting constraint violation, causing more severe failures to trigger stronger adaptation and more conservative subsequent control actions.
The proposed adaptation mechanism is integrated into an online MPC-based control framework with learned surrogate models. The authors derive theoretical bounds on both the long-run frequency of constraint violations and their average cumulative cost. The method is evaluated on Vanderpol, Pendulum, MountainCar, and Lorenz benchmarks against standard ACI, Conformal PID, Rolling RC variants, and a non-adaptive baseline. The results generally show lower violation frequency and violation cost, with a modest degradation in task performance.
Critical strengths
-
The paper identifies a meaningful limitation of standard ACI and addresses it with a simple and interpretable mechanism. Standard ACI reacts only to whether a miscoverage event occurs, whereas the proposed loss additionally incorporates the physical severity of the resulting constraint violation. This is particularly relevant for runtime assurance, where two prediction failures with similar statistical status can have very different operational consequences. The resulting adaptation mechanism is intuitive: more severe violations lead to larger corrective updates and more conservative subsequent control.
-
The proposed modification is supported by a relatively complete theoretical and empirical analysis. In particular, the paper derives separate guarantees on long-run violation frequency and average violation cost, rather than only on a generic calibrated loss. The experimental comparisons also consistently show reductions in both violation frequency and violation cost across several control benchmarks, with a relatively modest degradation in task performance.
Critical weaknesses
-
The methodological contribution is clear, but its distinction from existing adaptive conformal and online risk-control frameworks could be articulated more explicitly. The central modification is to replace binary miscoverage feedback with the cost-aware loss , which is well motivated for runtime assurance and leads to useful guarantees on both violation frequency and cumulative harm. Since prior online risk-control frameworks can accommodate user-defined losses, it would strengthen the paper to clarify more precisely what additional capability or theoretical insight is enabled by this particular coupling of miscoverage and physical violation cost.
-
The empirical evaluation is convincing on the selected benchmarks, but it is currently limited to relatively low-dimensional classical control systems. The experiments on Vanderpol, Pendulum, MountainCar, and Lorenz clearly demonstrate the intended adaptation behavior and the reduction in violation frequency and cost. At the same time, because the paper is motivated by safety-critical autonomous systems, evaluation on a more realistic or higher-dimensional control setting would further strengthen the practical relevance of the results.
Questions for authors
-
The paper notes that prior online risk-control frameworks can accommodate user-specified losses. Could the authors further clarify the specific advantages of the proposed cost-aware loss in comparison with directly using a severity-aware loss within a more general risk-control framework? In particular, it would be helpful to understand which aspects of the separate guarantees on violation frequency and cumulative violation cost are enabled by the proposed formulation.
-
The experimental results consistently demonstrate reduced violation frequency and violation cost across the four benchmarks. Could the authors provide some additional discussion on how the method may extend to more realistic or higher-dimensional safety-critical control settings, where the surrogate models and safety functions may be more complex? It would also be helpful to comment on any computational or modeling challenges that may arise in such settings.
-
The theoretical guarantees rely on the assumption that the robust control optimization remains feasible for every reachable state. Could the authors provide more intuition regarding how this assumption is expected to hold in practice? It would also be useful to discuss how the method might behave if the robust constraint becomes temporarily infeasible.
4 comments
Response 2
Results on F1TENTH across random seeds 0, 1, …, 9. Lower is better for all three metrics.
| Method | Task Cost | Violation Frequency: V(%) | Violation Cost: J () |
|---|---|---|---|
| Ours | |||
| Standard ACI | |||
| Conformal PID | |||
| MC | |||
| BSA | |||
| No ACI |
Results on RotorPy across random seeds 0, 1, …, 9. Lower is better for all three metrics.
| Method | Task Cost | Violation Frequency: V(%) | Violation Cost: J () |
|---|---|---|---|
| Ours | |||
| Standard ACI | |||
| Conformal PID | |||
| MC | |||
| BSA | |||
| No ACI |
3. Questions 3: Concern about Assumption 4
Thank you for raising this concern. We first clarify the practical interpretation of Assumption 4, then present the theoretical guarantees available when it fails, and finally provide supporting experimental evidence.
(1). Practical interpretation of Assumption 4.
The practical role of Assumption 4 is to ensure that the ACI-calibrated robust constraint remains feasible, allowing the controller to select an input satisfying this constraint at every time step. In practice, we aim to exploit the information provided by ACI to obtain quantitative long-run safety guarantees while balancing conservatism and task performance. The robust constraint is imposed with the same underlying safety objective as well-known shielding or fallback control mechanisms in the existing literature, namely, to keep the system within the safe set . However, our ACI-based approach provides weaker safety guarantees than safety-preserving shielding mechanisms that ensure safe-set invariance under appropriate assumptions, since the feasibility of the ACI-calibrated robust constraint does not itself guarantee that the system always remains within the safe set. When Assumption 4 does not hold, shielding or fallback control mechanisms can be directly integrated into our approach. We assume the availability of a suitable safety-preserving mechanism that can be activated when the ACI-calibrated robust constraint becomes infeasible. In particular, if the system is still within the safe set, a safety-preserving controller can be activated to maintain its invariance. If the system has already entered the unsafe set, a recovery controller can instead be employed to drive the system back into the safe set.
(2). Theoretical guarantees when Assumption 4 fails.
We note that our theoretical results are not entirely contingent on Assumption 4. Whenever optimization (8) is infeasible, we can use a shielding or fallback controller. Let if the backup input is applied at time and otherwise. In this case, we can obtain stronger violation frequency guarantees. The corresponding long-run bound in Theorem 1 becomes
\frac{\sum_{k=1}^{T}I_k e_k}{T}.$$ The violation cost bounds in Theorem 2 and Corollary 1 continue to hold in their original form. Thus, even when Assumption 4 fails, such shielding or fallback control preserves the original cost guarantees while strengthening the violation frequency bounds.Response 3
More generally, even when such a shielding or fallback controller is unavailable, we can still establish a full set of theoretical guarantees. Whenever optimization (8) is infeasible, we apply a prescribed backup controller and continue the original Cost-Aware ACI updates. This controller may be chosen to mitigate risk, but it is required only to supply an admissible input, which may violate the constraint in optimization (8). Under the conditions stated above, Lemmas 1 and 2 hold as stated, ensuring boundedness of the adaptive parameter and convergence of the average boosted loss to . The conservation identity in Proposition 2 and the regret bound in Proposition 3 for the realized linearized losses also remain valid.
The bounds in Theorems 1 and 2 and Corollary 1 require an additional term to account for the number of infeasible steps. Specifically, let count the steps at which optimization (8) is infeasible and the backup controller is used over the first time steps. For fixed algorithm parameters and , the finite-time bounds become
J_T\leq\frac{\alpha}{\beta}+O(1/T)+\frac{N_T}{T}.$$ The corresponding long-run bounds are $$V\leq\alpha+\limsup_{T\to\infty}\frac{N_T}{T},\ \ J\leq\frac{\alpha}{\beta} +\limsup_{T\to\infty}\frac{N_T}{T}.$$ If $N_T/T\to0$, the original long-run conclusions of Theorems 1 and 2 still hold, namely $V\leq\alpha$ and $J\leq\alpha/\beta$, even if infeasibility occurs infinitely often. If $N_T=O(1)$, the original $O(1/T)$ finite-time rate is also preserved. These results directly address the concern that the theoretical guarantees would cease to apply once Assumption 4 fails. Under the remaining assumptions, safety-preserving fallback control retains the original guarantees and can further tighten the violation frequency bounds. A general admissible backup controller also yields theoretical guarantees, with an explicit correction that quantifies the effect of infeasibility. We have added the complete theoretical results and proofs in Appendix D of the revised paper. ## (3). Additional experimental evidence To complement the theoretical analysis, we have added two more challenging, higher-dimensional benchmarks, F1TENTH and RotorPy, to evaluate the practical effectiveness of Cost-Aware ACI when Assumption 4 fails. In both environments, we deliberately impose challenging safety constraints and restrict the admissible control inputs, leading to frequent infeasibility of optimization (8). For example, in RotorPy, optimization (8) is infeasible at approximately $17.62\\%$ of the time steps under our method. Neither experiment uses a shielding or fallback controller that guarantees system safety. Instead, whenever optimization (8) is infeasible, we simply apply an admissible input supplied by a hand-designed backup controller, which is not required to satisfy the constraint in optimization (8). Despite this frequent infeasibility, our method achieves substantially lower violation cost and violation frequency than baselines such as No ACI in these experiments. These results provide empirical evidence that Cost-Aware ACI continues to operate effectively and reduces both the severity and frequency of constraint violations even when Assumption 4 fails. They complement the theoretical guarantees established in Appendix D and demonstrate the practical applicability of the method under challenging control conditions. The detailed experimental settings and results are provided in the revised paper.Further Clarification on Novelty
Dear Reviewer hBgY,
Thank you for your recognition of our paper and for raising the score to 3. To more comprehensively address your concerns regarding the novelty of our work, we would like to provide some additional clarification. The connection to Rolling RC lies in our adaptive update, which shares the same general online risk-control structure as Rolling RC. We believe that this overlap does not detract from the novelty of our work. Rolling RC provides a general framework for calibrating a chosen loss. However, which loss to choose for a given task, and what can be gained from that choice, remain open questions. Subsequent work illustrates this point. The proportional update in Conformal PID [1], as well as the calibration updates used by [2] and [3], fit within the same general framework of Rolling RC. This shared methodological foundation does not diminish the value of the distinct insights contributed by these works. The same reasoning applies to our work. As you noted under Critical Strengths, our paper identifies “a meaningful limitation of standard ACI”. To our knowledge, we are the first to systematically study the benefits of weighting ACI miscoverage feedback by the violation cost. This question was left open by both the analysis and the experiments in Rolling RC. We highlight two aspects of our work’s novelty below.
(1). We are the first to systematically study the benefits of weighting ACI miscoverage feedback by the violation cost, and our experiments provide strong evidence for the benefits of this design. Rolling RC provides a general framework for calibrating a user-specified loss. Its theory establishes how the update controls the chosen loss, but neither its analysis nor its experiments examine the role of physical violation severity in ACI. Our contribution is complementary to that of Rolling RC. We show the value of making the ACI update sensitive to the severity of physical constraint violations.
Our experiments compared Cost-Aware ACI with two alternative loss designs, including the miscoverage-counter loss used by Rolling RC. During the rebuttal, we added Tunable BSA to compare methods at similar levels of task performance. Taken together, these comparisons more clearly isolate the effect of loss design within a common online risk-control framework and provide stronger empirical evidence that Cost-Aware ACI can reduce both violation frequency and average violation cost.
(2). We develop a closed-loop runtime assurance framework that integrates Cost-Aware ACI with MPC and provides explicit bounds on both physical constraint violation frequency and average violation cost. Our contribution extends beyond the design of the cost-aware loss. By incorporating the calibrated uncertainty margin into the MPC constraint, we connect calibration to the safety of the executed trajectory. Beyond ACI’s long-run empirical coverage guarantee, our framework bounds the violation frequency and the average violation cost.
Taken together, our analysis and experiments identify a design principle for ACI in safe control. Uncertainty margins should adapt to both the frequency and the severity of constraint violations. We therefore maintain that identifying this design principle and demonstrating its value in closed-loop control together constitute a novel and substantive contribution to the use of ACI in safe control.
References:
[1] Angelopoulos, Anastasios, Emmanuel Candes, and Ryan J. Tibshirani. "Conformal pid control for time series prediction." Advances in neural information processing systems 36 (2023): 23047-23074.
[2] Wu, Junxi, Dongjian Hu, Yajie Bao, Shu-Tao Xia, and Changliang Zou. "Error-quantified conformal inference for time series." In International Conference on Learning Representations, vol. 2025, pp. 81566-81593. 2025.
[3] Muthali, Anish, Haotian Shen, Sampada Deglurkar, Michael H. Lim, Rebecca Roelofs, Aleksandra Faust, and Claire Tomlin. "Multi-agent reachability calibration with conformal prediction." In 2023 62nd IEEE Conference on Decision and Control (CDC), pp. 6596-6603. IEEE, 2023.
Response 1
1. Weaknesses 1 and Questions 1: Relationship to Rolling Risk Control and Our Contributions
Thank you for raising this important question. We would like to clarify the relationship between our work and the general Rolling Risk Control (Rolling RC) of Feldman et al. (2023), and emphasize that our contributions extend beyond the design of the cost-aware loss function in Cost-Aware ACI.
We acknowledge that our adaptive update shares the same general online risk-control structure as Rolling RC. However, our contribution is not a new online calibration algorithm. Rather, it lies in the design and analysis of a structured loss tailored to safety-critical control. Specifically, we show that the proposed cost-aware loss admits an economic risk-budget interpretation and yields simultaneous guarantees on two operational safety metrics:
J \le \frac{\alpha}{\beta},$$ where $V$ denotes the long-run violation frequency and $J$ denotes the long-run violation cost. These guarantees are not immediate consequences of the generic Rolling RC analysis. The ability of a general framework to accommodate user-specified losses does not automatically establish which loss is appropriate for a particular safety objective, nor the resulting guarantees on physical safety metrics. Our analysis exploits the specific structure $$L_k = e_k\bigl(1+\beta c(\mathbf{x}_k)\bigr)$$ to derive separate long-run guarantees on both violation frequency and cumulative violation cost. Furthermore, our original experiments already included two representative Rolling RC baselines, and we have additionally introduced a Tunable BSA baseline during this rebuttal. These comparisons more rigorously isolate the effect of different loss designs within the same general risk-control mechanism, providing stronger empirical evidence for the advantages of our proposed Cost-Aware ACI. Finally, our work addresses a closed-loop control problem rather than online prediction alone. The calibrated uncertainty margin directly affects the MPC optimization and the resulting control actions, which in turn influence future system states, physical violations, and subsequent losses. Our analysis and algorithm are developed specifically for this coupled setting, with the objective of balancing task performance and runtime assurance under unknown and time-varying dynamics. We will further clarify our contributions and provide a more detailed discussion of the relationship between our work and Feldman et al. (2023) in the final version of the paper. # 2. Weaknesses 2 and Questions 2: Scalability to Higher-Dimensional Environments Thank you for the constructive suggestion. We would first like to clarify that the proposed framework and Cost-Aware ACI mechanism impose no restriction on the state dimension. Our framework can be combined with different surrogate models and model-fitting methods, selected according to the dimension and complexity of the application. Moreover, Cost-Aware ACI operates on scalar safety prediction errors and violation costs, rather than on the full state vector, and its theoretical bounds have no explicit dependence on the state dimension under the stated assumptions. Consequently, the practical scalability to higher-dimensional systems depends primarily on the chosen surrogate models, model-fitting methods, and control optimization methods. Developing scalable methods for these application-specific components is beyond the scope of this work, which focuses on cost-aware conformal adaptation and its integration into online control. Nevertheless, we fully agree that more realistic and higher-dimensional experiments strengthen the practical relevance of the work. We have therefore added the F1TENTH and RotorPy benchmarks, with ten- and fifteen-dimensional augmented states, respectively. These benchmarks incorporate more realistic nonlinear dynamics and application-specific safety functions that connect physical operating constraints to the system state. In F1TENTH, the ego vehicle follows a lead vehicle under sudden braking, reduced tire grip, and increased actuator lag; its nonlinear safety function combines vehicle geometry, following distance, reaction time, and relative braking distance. In RotorPy, a quadrotor performs repeated inspection visits under nonlinear six-degree-of-freedom flight dynamics, aerodynamic forces, motor dynamics, and randomized upward gusts that intensify with altitude. As shown in the tables below, Cost-Aware ACI retains its advantage on both additional benchmarks, achieving lower violation frequency and violation cost than the compared baselines. We have also conducted further experiments on these two benchmarks, with complete results and experimental details provided in the revised paper. Given the time available for this revision, further evaluation on larger and more complex systems, together with experiments on physical platforms, is left for future work.