acceptodds
Withdrawn from ICLR 2027

When Does an Observable Lose Failure Evidence? Exact Ranking Losses for Runtime Detection in Vision–Language–Action Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

No runtime failure detector for a vision–language–action (VLA) policy can beat the Bayes-optimal ranking of the observation it reads. This ceiling is standard; what a practitioner needs to know is what separates it from the score actually deployed. We decompose that gap exactly into a representation loss from compressing the observation (for example, into sparse-autoencoder atoms), a projection bias from fitting a misspecified likelihood family, and an estimation term, and we characterize each. A compression is lossless if and only if it preserves the success posterior; otherwise its AUROC cost has an exact form and is bounded by the missing conditional information with a sharp constant . Ignoring the direction of time costs at most AUROC, where is the path-reversal divergence of outcome class ; the constant is sharp, yet irreversibility does not identify this value, and fully reversible paths can separate the outcomes perfectly. Low-order Markov likelihood projections can erase all evidence, and score consistency yields AUROC consistency exactly when the limit ties only states with equal likelihood ratio, a condition that a consistent maximum-likelihood fit can violate. For evaluation, early termination adds an exact bonus to prefix AUROC, and without structural assumptions no finite-sample procedure can certify a low ceiling. On seven LIBERO simulation interfaces spanning OpenVLA, OpenVLA-OFT, and 0.5 (7,000 report rollouts; no physical robot), episode length alone reaches AUROC 0.9996–1.0000 as the termination identity predicts, other atom-level families exceed the pre-specified Grammar-LLR witness by up to 0.42 on the same rollouts, and the ordering of atom-level and hidden-state witnesses changes across interfaces.

Primary area:
learning theory
Keywords:
vision-language-action models, runtime failure detection, Bayes-optimal ranking, AUROC, learning theory, information theory, representation compression, likelihood-ratio estimation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_FXWCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a theoretical analysis of the components of AUROC score in VLA failure detection, and leverages this view to gain empirical insight about reporting practices. The analysis centers around a decomposition of AUROC into components corresponding to representation loss, model misspecification, and estimation loss, with each component corresponding to a step in a typical machine learning pipeline. Experiments are performed in the LIBERO simulation environment, using two OpenVLA variants and π0.5\pi_{0.5}.

Critical strengths

The goal of rigorous end-to-end understanding of failure detection pipelines in robot policies is important and understudied.

Critical weaknesses

I find that the paper's main analysis is largely a restatement of well-established principles in modern machine learning, while the experiments do not convincingly demonstrate the work’s practical significance.

On the theory side, the decomposition is well-known, that to learn a predictor atop a frozen encoder, you need the right representation, the right model, and enough data. In my opinion this is common knowledge to machine learning practitioners, both in industry and academia. Offering some novel way to leverage this framing to improve performance could be a meaningful contribution (see below), but the observation itself is somewhat trivial. I am not familiar enough with the relevant statistical learning literature to judge the novelty of the theoretical results themselves, but I find that the authors do not convincingly establish their significance for the problem of VLA failure detection, which I believe is the goal here.

I find that the usefulness of the theory is not demonstrated via experiments. Given the setup, one might expect experiments that measure the different terms in Equation 6, and use that information to inform how we might better build or choose our systems. However, the experiments do not separately quantify any of these terms (e.g., the representation loss, which is arguably the most challenging to evaluate), and thus do not demonstrate how the proposed decomposition can be used to address challenges in building failure detection pipelines. Overall, it is hard to identify any new empirical insights that this work gives us.

Also, the focus on SAEs is not motivated, especially for a reader who is not deeply familiar with them. Table 3 seems to show that detectors trained atop the typical hidden state perform better, and I think that hidden states are the most standard choice for such systems.

Questions for authors

Equation 6 seems straightforward, which subsequent results do the authors consider most significant, and what do they show about building VLA failure detectors that was not previously understood?

Reviewer_sM1hOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies runtime failure detectors for vision–language–action (VLA) policies, evaluated by AUROC. It takes as reference the AUROC of the Bayes-optimal likelihood-ratio score for a given observable. It then decomposes the gap between the Bayes-optimal and the AUROC of a deployed detector into a representation loss, a projection bias, and an estimation term, and it states several results about each term. Experiments use LIBERO simulation rollouts of OpenVLA, OpenVLA-OFT, and \pi_{0.5}.

Critical strengths
  1. Runtime failure detection for learned robot policies is an important problem for deploying robots safely. This makes the topic adequate for ICLR, but it does not by itself outweigh the weaknesses below.
  2. The paper shows that on LIBERO, episode length alone separates successes from failures almost perfectly, because failed episodes run until the time limit. This is a useful point to know for anyone evaluating failure detectors on this benchmark. However, the effect was already observed in SAFE (Gu et al., 2025), so its significance as a contribution is limited.
Critical weaknesses
  1. The writing and presentation fall well short of an ICLR paper. This is my main concern. From the very start of the paper, I found the paper extremely hard to follow, and I do not think this is only due to the material. --> The abstract of a scientific paper should be at a level of abstraction so that even non-expert should be able to grasp the general content and contribution of it. I suggest anyone to try this for this submission. --> The introduction does not motivate the problem; it summarises the paper. The third paragraph lists theorem statements with their constants, and the fourth lists proof techniques. A reader is never told, in plain terms, what question the paper answers, why a practitioner should care, or what the reader should take away. --> Table 1 is pure bookkeeping and does not help the reader understand the results. --> The "discussion" in Section 6 largely restates the abstract and shows little reflection on the meaning of the results or where they fall short. --> The central reference quantity "Bayes ceiling" should be explained much more plainly. If I understand correctly, the "ceiling" is the AUROC of the best possible detector, i.e., the likelihood ratio between the success and failure distributions of the observable. Attaining it requires exact knowledge of these distributions (e.g. a model), so it is an upper bound that is unattainable in practice without model knowledge. By the authors' Proposition 4, it cannot even be certified from finite data beyond the first time step without structural assumptions. This should be stated upfront in plain words. Instead, the introduction says that the ceiling "cannot say whether a low AUROC comes from weak evidence or from a detector that uses the evidence badly", which I found confusing.

Overall, the paper does not present its ideas at different levels of abstraction. Detailed mathematics comes before the reader knows why it matters, and as a result it remains largely unclear to me what the authors are ultimately trying to accomplish.

  1. The paper does not mention runtime monitoring at all. Deciding during the execution of a system whether it is about to fail, and when to raise an alarm or hand over control, is the core problem of runtime monitoring! This is a mature research area spanning control, formal methods, and statistics. The paper does not mention it at all, and the authors seem unaware of this entire research direction. Central notions from that area, such as detection delay, false-alarm rate, and prediction lead time, are absent here. For a detector to be useful for intervention, it must flag a failure before it happens (i.e., predictive failure); a ranking metric on (partial) episodes does not capture this. The paper needs to position itself clearly against this body of work and explain what it adds. In fact, this nicely leads into the next critical weakness:

  2. The main empirical comparisons do not test runtime detection. As I understand Section 5.1, in LIBERO a successful episode ends as soon as the task is completed, whereas a failed episode runs until the time limit. Simply knowing how long an episode lasted therefore separates successes from failures almost perfectly; AUROC 0.9996–1.0000, as recited from SAFE. From a monitoring perspective, this information is available only once the episode is over, so it is of no use for deciding when to intervene. Several places in the submission suggest that the authors provide a "a score, computed while the episode runs, that decides when to stop the robot or hand control to a human". I would ask the authors to clarify what theirs results tell a practitioner who must intervene before a failure occurs.

  3. Appendix L states that most of the core components of the paper are classical notions, and that their contribution is "their assembly into an exact decomposition", with "no claim of priority beyond this assembly". Several results in the main text are labelled "standard". Taken together, the theoretical contribution reads as incremental, and this is not apparent from the abstract or the introduction.

Questions for authors
  1. In two or three plain sentences: what problem does the paper solve, and what should a practitioner do differently after reading it?
  2. How does this work relate to runtime monitoring, and what does it add beyond that field?
  3. What do the full-horizon results tell someone who must intervene before a failure occurs? Can you report detection lead time at a fixed false-alarm rate?
  4. What does it mean that the primary-study data "became inaccessible", and which primary results can be reproduced from the released material?
  5. Given Appendix L, which results do you consider genuinely new, and why are they significant?
Reviewer_QfuNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes why the AUROC of VLA failure detectors may fall below the best achievable from their observables. The authors decompose this gap into representation loss, likelihood-model bias, and estimation effects from fitting the model on finite data, and examine how episode termination affects AUROC. Across seven LIBERO simulation settings, episode length alone nearly perfectly separates success from failure at the full episode horizon, while the relative performance of atom-based and hidden-state detectors varies by setting.

Critical strengths

None

Critical weaknesses
  • The paper’s practical contribution to VLA failure detection is limited. Although authors present many theorems and propositions, the main message is that reported AUROC is affected by representation, likelihood-model bias, finite-data estimation, and the evaluation setup, rather than detector performance alone. Even though this analysis is useful, the authors do not provide a concrete alternative for evaluating runtime failure detectors beyond guidances on interpreting AUROC. I am therefore not convinced that the contribution is sufficient for a top-tier venue such as ICLR.

  • The experimental results in Section 5 provide limited insight beyond the termination-length effect. First, it is too straightforward and unsurprising that episode length separates outcomes in LIBERO when successful episodes terminate early and failures run to the maximum episode length. Second, the relative AUROC of atom-based scores and SAFE changes across interfaces, but the comparisons do not isolate which changes account for these differences. Thus, they do not establish a clear conclusion about the observables themselves.

Questions for authors

Q1. I think the paper is motivated by analyzing AUROC as a metric for evaluating VLA failure detectors. Is it really important to analyze AUROC alone? Are there other metrics relevant to failure detection? Also, I think representation loss, model bias, and finite-data estimation can affect many other machine-learning metrics, rather than AUROC alone. Can the authors clarify whether these findings are specific to AUROC?

Q2. Why do the authors focus on classical likelihood-based models, such as a first-order Markov model? How do these models relate to current VLA failure detectors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.