acceptodds
Withdrawn from ICLR 2027

Risk Bounds for Physical AI: Decision Ambiguity and Model Incompleteness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Physical AI safety depends not only on what a system perceives, but on what its action will do and whether a response can still work in time. From classical decision, identification, and testing results we compose one risk bound for a single irreversible decision that separately budgets the loss of continuing while the consequence model is valid, of continuing after it has silently failed, and of responding, including responses that fail or come too late. A hand-constructed finite example computes every term of the bound, and a synthetic model instantiates its monitoring term. A MetaDrive study, which does not estimate the bound, asks whether detect-and-brake monitors pass a preset intervention screen in intact driving and how responses change repaired and new failures. A hidden fault that scales braking commands to 70% produces 21 failures in 120 drives selected for nominal success. None of twelve detect-and-brake candidates frozen before the audit passes the screen: the least-triggering ones intervene in 3 of 24 intact scenes where at most 2 were allowed; with 24 scenes this fails the screen but cannot show that the true rate exceeds 10%. Offline re-scoring with a post-hoc predictor whose one-step error is 37.7% lower detects more faults, but its best variants still cross the threshold in 3 of 24 intact scenes. In post-hoc same-scene comparisons, compensation with the true fault gain, started at the known onset in fault drives only, repairs 15 of 21 baseline failures and introduces two; started by an onset-aligned trigger in every drive, it leaves 16 fault failures and causes 14 intact-drive failures. Three braking responses on that trigger each introduce more failures than they repair in point estimates (paired intervals include or touch zero). Together, the analysis and experiments motivate treating information sufficiency, consequence validity, and response execution as separate assurance obligations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
physical AI, AI safety, runtime assurance, risk bounds, model misspecification, partial identification, fault detection, safe fallback, autonomous driving

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_kX35OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the quantification of risk in physical AI systems under decision ambiguity, incomplete consequence models, and potentially ineffective safety interventions. It develops a mathematical framework that combines classical Bayes-risk, partial-identification, and statistical testing results to derive upper and lower bounds on expected loss. The paper further evaluates fault detection and mitigation strategies using synthetic examples and a MetaDrive autonomous driving simulator. The experiments highlight that improved prediction or fault detection does not necessarily translate into reduced system-level risk, particularly when safety interventions introduce additional hazards.

Critical strengths

Relevant system-level safety perspective: The paper highlights an important distinction between accurate perception, valid predictions of action consequences, and effective safety interventions. This is particularly relevant for learning-enabled autonomous systems, where detecting a fault does not necessarily guarantee successful mitigation.

Rigorous mathematical formulation and informative empirical observations: The proposed risk decomposition is mathematically well-founded under explicitly stated assumptions. The experiments also provide useful evidence that safety interventions can introduce additional failures, motivating evaluation of the complete controller–monitor–response mechanism rather than individual components.

Critical weaknesses

Limited novelty and insufficient positioning against established safety and reliability engineering: The main theoretical contribution combines established results from Bayes decision theory, partial identification, hypothesis testing, and conditional risk decomposition. The resulting formulation closely resembles classical event-tree and probabilistic risk assessment approaches. The paper does not sufficiently demonstrate what fundamentally new risk-quantification capability it offers beyond existing system reliability methods, particularly when model uncertainty, diagnostic coverage, and response failures are already considered.

Incomplete assurance framework and insufficient empirical validation: The three proposed assurance obligations do not constitute a systematic or demonstrably complete safety-analysis methodology. In particular, the framework assumes that relevant hazards, losses, actions, and model-validity conditions have already been identified, without providing a mechanism comparable to .e.g, STPA (and other traditional safety analysis techniques) for identifying unsafe control actions and causal scenarios. More importantly, the central risk bound is not instantiated or evaluated in the MetaDrive experiments. Its practical usefulness remains uncertain because several key quantities, including model-escape probabilities and conditional response risks, require independent evidence that may be difficult to obtain.

Questions for authors

How does Theorem 5.2 fundamentally differ from conventional event-tree or probabilistic risk assessment methods that decompose system risk into initiating events, detection failures, and mitigation outcomes? Can the authors demonstrate a concrete advantage over these established approaches?

How were the three assurance obligations derived, and in what sense are they sufficient for physical AI safety? How would the framework integrate with systematic hazard-analysis approaches such as STPA, particularly for identifying unsafe control actions, hazardous interactions, and previously unrecognised failure scenarios?

Can the authors instantiate the full risk bound on the MetaDrive system using independently justified estimates of its constituent terms, including uncertainty bounds, and compare the resulting risk estimates with observed failure rates? How can the model-validity envelope and escape probability be credibly established when the relevant failure mechanisms may be unknown?

Reviewer_Vq6DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies safety in physical AI systems by distinguishing three sources of risk: making decisions with insufficient information, relying on an inaccurate consequence model, and executing a fallback action that may fail or arrive too late. The authors combine several classical results into a unified risk bound and use synthetic examples and MetaDrive experiments to illustrate the framework.

Critical strengths

The paper addresses an important problem in safety-critical AI. Separating decision uncertainty, model validity, and response effectiveness is a useful perspective.

The theoretical presentation is generally clear, and the authors explicitly acknowledge that several mathematical ingredients are classical.

Critical weaknesses

I am not fully convinced by the theoretical novelty. The main risk bound is derived by combining several well-established results, and the proof of Theorem 5.2 is essentially a decomposition of different risk sources. While I agree that considering these risks together is useful, it is not clear what new theoretical insight the formulation provides beyond existing safety assurance frameworks.

My main concern is the connection between the theory and the experiments. The MetaDrive experiments do not actually evaluate the proposed risk bound, while the synthetic examples rely on predefined models and assumptions. As a result, the theoretical and experimental parts feel somewhat disconnected. I would expect at least one experiment showing how the bound can be calculated and used to assess or improve a controller.

Some of the key assumptions are quite strong. For example, the framework requires information about whether the consequence model remains valid and how reliable the fallback response is. In practice, these quantities can be difficult to determine, especially under previously unseen faults. I think the paper needs to explain more clearly how these assumptions could be justified in realistic applications.

The experimental evidence is also somewhat limited. The authors consider only one driving controller and one type of actuator fault. In addition, the evaluation scenes are selected based on successful nominal operation. This makes the experiments useful as a controlled case study, but it is difficult to know whether the observations would hold for other controllers, fault mechanisms, or more challenging driving situations.

I am not sure the frozen audit result is sufficiently convincing. The best monitors are rejected because they trigger in 3 out of 24 nominal scenes, while the threshold allows 2. This is a very small difference, and the sample size is limited. Although the authors acknowledge this issue, the rejection of all candidates is still emphasized as an important experimental finding.

Finally, the fallback strategies appear relatively basic. Applying a fixed braking command while retaining the original steering behavior may itself create unsafe situations, particularly when braking authority is already degraded. The reported increase in failures is interesting, but without comparisons against stronger safety controllers, it is difficult to tell how much of this result comes from the underlying safety problem and how much comes from the particular fallback design.

Questions for authors

Could the authors clarify what the main theoretical advance of Theorem 5.2 is compared with existing risk decomposition and runtime assurance methods? Is there a particular safety guarantee or design problem that becomes possible to address through this formulation?

Is it possible to calculate the proposed risk bound for the MetaDrive experiments, even under simplified assumptions? I think this would help demonstrate the practical value of the framework.

The best monitors fail the admission criterion by only one nominal intervention. How sensitive are the conclusions to the chosen threshold and the number of evaluation scenes?

Have the authors considered comparing the fixed braking fallbacks with a feedback-based or fault-aware safety controller? This might help explain whether the additional failures are caused by delayed detection or by the response strategy itself.

The oracle compensation strategy performs considerably better than the adaptive variants. Can the authors further investigate whether this difference is mainly caused by inaccurate fault estimation, delayed intervention, or limitations of the compensation policy?

Reviewer_pMuMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The goal of this paper is to create an estimate for whether a single decision to act (or not) would result in irreversible failure of the system. The approach is to create estimates for whether continuing or failing to continue can be understood, including whether the model that generates the evaluation estimate might have failed. The paper takes two routes to validate the approach. The first is through exhaustive computation of the bound on a synthetic model. The second is to utilize a MetaDrive study to examine whether using the approach can improve performance.

Critical strengths
  • The paper includes some exhaustive examples of potential outcomes: different braking mechanisms, different bounds of deceleration, etc., for the experiments using metadrive. The experiment uses a stochastic model to evaluate different response constraints such as how the brakes are used.
  • Detailed examination is given in Section 5 on defining the safety for understanding the response (or lack thereof) for braking, steering, or other changes in response.
Critical weaknesses
  • The paper does not describe the results in a striking way. While the responses and commands are thoroughly described in Section 6.4, for example, I do not come away with a compelling understanding of the impact of the work from a clinical description of the experiments. It is not clear what is the main takeaway.
  • Many details are described in Section 6 for the experiments. What is missing is a non-jargon explanation of the performed experiment set, and the parameters of the experiments. For example, the "safe side" of an obstacle is perhaps understood by other readers better than this reviewer, but it is not expanded upon in the text. Similarly, the analytic floors are compared to the model in this section, though the evidence for why the simulated risk is within the prescribed tolerance is treated in prose rather than with comparison to a table or other information. The section 6.1 would, I expected have been what was promised in the abstract: the 'hand-constructed finite example.' If that is what this section is, then some clarity should be brought.
  • In general, the paper's style comes across more as a response to an AI prompt, than a technical paper that could be utilized to apply the approach to a different problem. This comment is not intended to predict whether AI was used in any way for the paper's proofreading or production, but moreover a statement that regardless of how that style came to be, it comes across in that way. Other readers may not see this as a weakness, but for this reader the main challenge is that the mix between overly-specific data description in prose, and overly-layperson style of leadins to paragraphs, can be somewhat frustrating to understand where to find technical details when reading the paper. Given that the extensive appendix may hold much of this information, the paper then becomes a bit of a challenge to interpret without also parsing and reading through the appendix, causing additional fraction of time. This challenge to the paper's style goes beyond one reader or writer's preference, and is (I think) a weakness, that the paper comes across as a summary of the appendix, which is perhaps not as useful for a paper in the conference. Other reviewers may differ in their opinions on this front.
Questions for authors
  • Can you justify why the 'one-step error' reduction of 37.7% is what you chose in your fault detection exploration?
  • Did you expect that a potential design of the system could prevent all crashes of the selected scenes, or was a design intention to explore the actions taken (or not) to attempt to hone in on whether some system designs were just not able to be de-risked?
Reviewer_BfvzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a combined bound on the risk of an autonomous system. The bound combines multiple potential sources of error: the uncertainty that comes from observation histories being consistent with multiple state beliefs, and also the error that comes from potentially incorrect model of how the system will behave under a chosen action (or commitment to subpolicy, which counts as an action for the purpose of this study). It also treats the fact that for a risk-mitigating intervention monitor to be useful, it must be able to intervene while there is still time for the intervention to be useful. The paper builds their derivation on some reasonable assumptions (for example, the "Decision-validity contract" of Definition 2.1) and derives a bound that includes multiple kinds of error. An empirical study on MetaDrive under perturbed dynamics (e.g. brakes working at diminished capacity) measures how different kinds of error contribute to system-level failure, but does not quantitatively relate to the main bound derived in Theorem 5.2.

Critical strengths

This paper treats an important problem: understanding how different forms of uncertainty and modeling error contribute to the error of an autonomous system.

The initial assumptions make sense and render the theoretical derivations tractable. Many of the propositions are relatively straightforward to prove, which is good.

Critical weaknesses
  1. This paper is extremely poorly organized and suffers from severe presentational errors. For example, the abstract is rather impenetrable- instead of clearly situating the key contributions of the paper and situating it in the literature, it includes many sentences of quantitative results for experiments that are not yet introduced, and which are not clearly related to the key contribution of this paper (the derived risk bound). Much of this same "abstract" paragraph is repeated verbatim later in section 1, where it is equally hard to understand. More generally, the paper suffers from severe lack of structure and signposting: there are many small paragraph headings that ask intermediate questions, but it is unclear what the paper is building towards. Sections 2, 3, 4, and 5 are all building towards theorem 5.2, but this is not at all clear until the reader reaches theorem 5.2. There should be a clear "problem setup" section that makes it clear what the paper is building towards. Also, as much as possible, key notation should be introduced early in the problem statement.

  2. The paper does not include proper related work. Section 7 "Related work and conclusion" briefly lists some related work and points the reviewer to appendices for comparison with this paper. Situating your work in the existing literature is actually the most important part of a paper, and should be in the main body of the paper.

  3. The primary contribution, Theorem 5.2, is not validated to be practically useful. There are many components that go into deriving this bound, including a final union bound, meaning the bound is likely not very tight in practical systems. It is of course common for papers to have conservative or loose bounds, but they must be either 1. argued to be practically useful or 2. be for a clean, well-motivated, theoretically interesting problem. This paper seems to fall more into the applications focused category, so there should be some argument of why the bound is practically useful. The current experiments, which the abstract even says for which "does not estimate the bound" do not relate directly to the bound attained by Theorem 5.2. The abstract and contributions section does not show a clear connection between the differerent claimed contributions: overall the paper seems unfocused and disorganized.

Questions for authors
  1. Do the empirical results show the practical value of the bound derived in theorem 5.2?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.