acceptodds
Under review as a conference paper at ICLR 2027

Introspective Signals for Failure Prediction in Vision-Language-Action Models

Abstract

Existing approaches for failure prediction in VLAs rely on information derived from policy inputs, outputs, and latent representations. However, the predictive value of relationships among internal signals (such as attention, value vectors, and hidden states) and their evolution during execution remains underexplored. We hypothesize that these patterns can support early and accurate prediction of a VLA’s eventual failure. We construct hypothesis-driven introspective signals organized around four categories – world sensitivity, temporal consistency, attention–attribution agreement, and internal coordination – and learn temporal patterns over them that align with with episode-level outcome labels. Across three policy families and two LIBERO-based task settings, our models predict failures earlier and more accurately than existing methods and generalizes to unseen tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Failure Prediction, Vision-Language-Action Models, Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jcocOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether relationships among internal signals in vision-language-action (VLA) models can predict eventual task failure more accurately and earlier than existing failure predictors. The method collects attention weights, value vectors, hidden states, embedding gradients, observations, and actions from VLA inference. Handcrafted statistics such as cosine similarity, Spearman correlation, top-k IoU, and Wasserstein distance characterize relationships between these signals.

Critical strengths
  • Important and practically motivated problem. Predicting failure before a robot irreversibly executes an unsuccessful trajectory is an important challenge.

  • Comprehensive experiments. The inclusion of autoregressive and continuous-action VLA architectures increases the relevance of the study. It also highlights that different architectures expose different internal signals. Meanwhile, this paper also has a detailed feature analysis.

Critical weaknesses
  • The approach primarily constructs manually designed features from existing internal signals and trains a GRU classifier. It is unclear whether the proposed relationships are fundamentally better than appropriately trained predictors using the same underlying signals.

  • The experiments do not isolate the source of improvement. Better performance might come from richer input features, temporal modeling, feature selection, or different training procedures rather than the proposed introspective relationships.

  • The practical claim of early failure prediction is insufficiently demonstrated. Per-step AUROC improvements do not necessarily imply useful early warnings, especially without reporting detection lead time, false-alarm tradeoffs, and inference overhead.

Questions for authors
  • How much of the proposed method's improvement remains when all internal attention, hidden-state, and gradient-based features are removed, retaining only observation and action-consistency features?

  • Can a GRU trained on the same underlying signals, without handcrafted pairwise relationships, achieve comparable accuracy?

  • How sensitive is the performance to the action-related scalar used to compute embedding gradients?

3 comments
AuthorsOpenReview ↗

Response to Reviewer jcoc (1/3)

Thank you for taking the time to review and providing suggestions to strengthen our analysis. We would like to address your concerns and questions as follows:

The approach primarily constructs manually designed features from existing internal signals and trains a GRU classifier. It is unclear whether the proposed relationships are fundamentally better than appropriately trained predictors using the same underlying signals.

Can a GRU trained on the same underlying signals, without handcrafted pairwise relationships, achieve comparable accuracy?

The experiments do not isolate the source of improvement. Better performance might come from richer input features, temporal modeling, feature selection, or different training procedures rather than the proposed introspective relationships.

Thank you for suggesting this analysis. We address these questions in two parts:

(1) New Analysis: Very Limited Gain from Richer Inputs. We trained a compact raw signal predictor using six inputs selected from attention weights, value vectors, hidden states, gradient-based attribution signals, and action outputs, etc. The predictor uses the same GRU architecture and training procedure as our relational feature predictor. Please see a more detailed training description in Section 6 of the updated manuscript. We compare it with SAFE, which operates on policy hidden states only. The broader raw signal pool provides a modest improvement for π0.5 and similar performance for π0-FAST and OpenVLA-OFT. Thus, access to richer raw inputs alone produces limited gains and does not explain the improvement from our relational feature construction. Please see a figure of per-step task-disjoint test AUROC for SAFE and the compact raw signal set predictor in Section A.3.

New Analysis: Failure Prediction from Raw Signals vs. Relational Features To address whether improvements come from relational feature design and temporal modeling, we also provide comparison between the compact raw signal set predictor and a compact feature set predictor (Training detail described in Section 6.1 of the updated manuscript). The relational feature set substantially outperforms the raw signals throughout the rollout for π0.5 and π0-FAST. For OpenVLA-OFT, the two predictors perform similarly during the first 30 steps, after which the relational feature set is generally stronger, with some variation near the end of the rollout. These results show that explicitly modeling relationships among internal signals provides predictive information beyond the raw quantities and their direct temporal changes. We have included this analysis in our updated manuscript Section 6.3. Please also see a figure of per-step task-disjoint test AUROC for compact raw signal and relational feature sets in Section 6.3.

AuthorsOpenReview ↗

Response to Reviewer jcoc (2/3)

The practical claim of early failure prediction is insufficiently demonstrated. Per-step AUROC improvements do not necessarily imply useful early warnings, especially without reporting detection lead time, false-alarm tradeoffs, and inference overhead.

We address this concern in two parts:

(1) Lack of Ground Truth for Failure Onset. Among the four baselines (i.e. SAFE [1], FIPER [2], INSIGHT [3], FAIL-Detect [4]), only SAFE compares detection timing with human-annotated failure times (INSIGHT only annotates steps that don’t make progress towards the goal as a help trigger signal). However, SAFE uses judgments of when intervention is needed for ambiguous cases, not of when a failure clearly onsets. Since there is no gold standard or ground truth to identify when failure happens, we attempted at having a human annotator (one of the authors) and an LLM (GPT 6.1-sol) independently annotate 50 failed rollouts from each of the three policies. The audit revealed substantial disagreement in the failure onset steps between the human annotator and the LLM, with Mean Absolute Differences (MAE) of 38.1, 32.6, and 30.7 policy steps for OpenVLA-OFT, π0.5, and π0-FAST, respectively. This disagreement suggests that judgement of failure onset is ambiguous and difficult to scale for large evaluations. For example, one annotator may rule the step where the robot mis-grasps and drops an object as failure onset, whereas another may wait until the robot reaches an unrecoverable failure state. SAFE also discusses similar ambiguity in their human annotation process in Appendix C.3.

(2) New Analysis: Normalized First-Detection Time in Failure Episodes. We aim to predict eventual failures as early as possible within the rollout while using the final episode outcome as the supervision and evaluation target. Following the rollout-relative timing evaluations in FIPER and SAFE, we define normalized first-detection time as the first-detection step divided by episode length. Following FIPER, we compute it over correctly detected failure episodes. We report results averaged over 40 folds at thresholds calibrated to target episode-level FPRs of 5%, 10%, and 20%. Baselines are evaluated at their original operating points. Lower values indicate earlier detection. At the 10% FPR target, our predictor first detects failure after 13.1%, 13.0%, and 13.6% of the rollout for π0.5\pi_{0.5}, π0\pi_0-FAST, and OpenVLA-OFT, respectively. These detection times are earlier than those of every applicable baseline at its original operating point. We have included this analysis in more detail in the updated manuscript Analysis section.

VLA modelOurs 5% FPROurs 10% FPROurs 20% FPRSAFEFAIL-DetectINSIGHTFIPER
π0.5\pi_{0.5}0.169 ± 0.0430.131 ± 0.0380.089 ± 0.0320.214 ± 0.1420.383 ± 0.176—0.476 ± 0.104
π0\pi_0-FAST0.131 ± 0.0500.130 ± 0.0450.118 ± 0.0480.164 ± 0.1490.276 ± 0.1720.291 ± 0.142—
OpenVLA-OFT0.178 ± 0.0420.136 ± 0.0350.106 ± 0.0290.169 ± 0.1450.232 ± 0.100——

[1] Gu, Qiao, et al. "Safe: Multitask failure detection for vision-language-action models." Advances in Neural Information Processing Systems 38 (2026): 40041-40076.

[2] Römer, Ralf, et al. "Failure prediction at runtime for generative robot policies." Advances in Neural Information Processing Systems 38 (2026): 7631-7670.

[3] Karli, Ulas Berk, et al. "Insight: Inference-time sequence introspection for generating help triggers in vision-language-action models." 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.

[4] Xu, Chen, et al. "Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies." arXiv preprint arXiv:2503.08558 (2025).

AuthorsOpenReview ↗

Response to Reviewer jcoc (3/3)

How much of the proposed method's improvement remains when all internal attention, hidden-state, and gradient-based features are removed, retaining only observation and action-consistency features?

We provide a comparison between the compact feature set predictor and a predictor trained on only observation and action-consistency features for OpenVLA-OFT, π0.5, and π0-FAST, respectively. When all internal attention, hidden-state, and gradient-based features are removed, the test AUROCs worsen drastically at step 0-5 across all three policies and remain relatively inferior at step 5-15 for π0.5 and OpenVLA-OFT. This suggests the features constructed on the internal signals are very helpful at the early phase of policy rollouts. Please see a figure of per-step task-disjoint test AUROC comparison in Section A.4 in the updated manuscript.


How sensitive is the performance to the action-related scalar used to compute embedding gradients?

We'd like to clarify that the action-related scalar is used only to compute gradient-based attribution; it is not provided directly to either the compact raw signal set predictor or the compact feature set predictor. As reported in Appendix A.1 of the updated manuscript, gradient-based attribution features are frequently useful: several rank among the top 30 single-feature GRUs and enter the starter pool for forward selection, and some are selected in the final six-feature sets. However, the predictor does not depend on them. The fixed six-feature GRUs for π0.5\pi_{0.5} and π0\pi_0-FAST used in our main comparison to baselines contain no gradient-derived features and still achieve similar per-step AUROC. Thus, gradient-based attribution provides useful complementary evidence when selected, while compact predictors can also perform well without it. Our current experiments establish the contribution of gradient-based attribution features but do not compare alternative definitions of the action-related scalar directly.

Reviewer_4WcKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether internal signals of Vision-Language-Action (VLA) models can predict eventual task failure during execution. The authors construct introspective features from attention weights, hidden states, gradient-based attribution, and action predictions, organized into four categories: world sensitivity, temporal consistency, attention–attribution agreement, and internal coordination.

These features are used to train GRU-based failure predictors with episode-level success/failure labels. Experiments across three VLA architectures on LIBERO-based benchmarks demonstrate improved prediction performance over existing baselines, including under task-disjoint evaluation. The authors also analyze the contribution of individual feature categories and feature-set sizes.

Critical strengths

S1. Important problem and meaningful evaluation setting.

Early failure prediction is an important problem for reliable robotic manipulation. The task-disjoint evaluation protocol is particularly valuable, as it examines whether learned failure predictors generalize beyond the tasks used for training.

S2. Promising empirical performance.

The proposed features achieve competitive prediction performance across three VLA architectures. The extensive feature ablation also provides useful observations about the predictive value of different internal signal relationships.

Critical weaknesses

W1. Insufficient distinction between failure-specific signals and task difficulty or uncertainty.

The central hypothesis is that relationships among introspective signals provide useful indicators of eventual task failure. However, these signals may also reflect task difficulty, uncertainty, or normal exploratory behavior rather than impending failure.

For example, attention inconsistency and frequent action replanning may occur when multiple visually similar objects are present, even if any of these objects can be selected successfully. Conversely, a VLA may consistently execute an incorrect action with stable internal representations.

Although the predictor is supervised using episode-level outcomes, the experiments do not directly distinguish failure-related information from correlations with task difficulty or uncertainty. The task-disjoint evaluation demonstrates generalization within LIBERO, but does not establish robustness when the relationship between uncertainty and task success changes.

Consequently, it remains unclear whether the proposed signals provide reliable failure-specific information or primarily exploit correlations present in the evaluated tasks.

W2. The contribution of relational features and the early-prediction capability are not sufficiently isolated.

The paper argues that relationships among internal signals provide more informative failure indicators than individual representations. However, the main comparisons involve different feature representations and predictor configurations.

A controlled comparison using the same GRU architecture with raw internal signals, simple temporal differences, and the proposed relational features is missing. In particular, it is unclear whether much of the improvement could be explained by simple action replanning or uncertainty-related signals.

Furthermore, the evaluation primarily reports per-step classification performance against eventual episode outcomes. It does not directly quantify how far in advance failures are detected relative to their physical occurrence. Thus, the reported AUROC improvements do not fully establish the practical benefit of early warning.

Questions for authors

Q1. Failure versus uncertainty and difficulty.

How does the proposed predictor distinguish impending failure from uncertainty or difficult but successful execution? Can the authors evaluate high-uncertainty successful episodes and low-uncertainty failed episodes, preferably under controlled task difficulty?

Q2. Contribution of relational features.

Can the authors compare the proposed relational features against raw internal signals and simple temporal-change features using an identical GRU architecture? This would help isolate the contribution of the proposed feature construction.

Q3. Early failure detection.

Can the authors report detection lead time or pre-failure recall at matched false-positive rates? Such results would clarify whether the proposed method provides actionable warnings before physical failures occur.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4WcK (1/3)

Thank you for taking the time to review and providing suggestions to strengthen our analysis. We would like to address your concerns and questions as follows:

W1. Insufficient distinction between failure-specific signals and task difficulty or uncertainty. The central hypothesis is that relationships among introspective signals provide useful indicators of eventual task failure. However, these signals may also reflect task difficulty, uncertainty, or normal exploratory behavior rather than impending failure. For example, attention inconsistency and frequent action replanning may occur when multiple visually similar objects are present, even if any of these objects can be selected successfully. Conversely, a VLA may consistently execute an incorrect action with stable internal representations. Although the predictor is supervised using episode-level outcomes, the experiments do not directly distinguish failure-related information from correlations with task difficulty or uncertainty. The task-disjoint evaluation demonstrates generalization within LIBERO, but does not establish robustness when the relationship between uncertainty and task success changes. Consequently, it remains unclear whether the proposed signals provide reliable failure-specific information or primarily exploit correlations present in the evaluated tasks.

Q1. Failure versus uncertainty and difficulty. How does the proposed predictor distinguish impending failure from uncertainty or difficult but successful execution? Can the authors evaluate high-uncertainty successful episodes and low-uncertainty failed episodes, preferably under controlled task difficulty?

Thank you for suggesting this analysis. We will respond in two parts:

(1) New Analysis Regarding Task Uncertainty: Since the rollouts do not have ground truth uncertainty annotations, we adopt INSIGHT [1], a learned help-trigger predictor based on token-level uncertainty signals, as an operational proxy. Across 40 folds, we treat a task as high uncertainty only if >50% of the 40 folds that held it out for the test set flagged it as “uncertain” at any timestep. We treat the 121 successful episodes that INSIGHT flags as high-uncertainty successes and the 154 failure episodes that INSIGHT doesn’t flag as low-uncertainty failures.

Episode outcomeMajority flaggedNot flagged
Failure (569)415154 (low-uncertainty failures)
Success (2,067)121 (high-uncertainty successes)1,946
  • Failure signals present in low-uncertainty failures: On the 154 failures that INSIGHT doesn’t flag as uncertain, our compact feature set predictor’s (training details updated in Section 6.1) per-step test AUROC curve resembles that of all failures, though it is slightly during early timesteps (1-30). The failure signal is therefore present when the uncertainty signal is absent.
  • Failure signals capture more than uncertainty: high-uncertainty success episodes are the hard cases, but on the 121 successes where INSIGHT flags for uncertainty, our predictor’s per-step test AUROC curve still beats INSIGHT, suggesting that the failure signals capture more than uncertainty signals. Please see Section A.2 in the updated manuscript for a figure of the task-disjoint test AUROC on low-uncertainty failures and high-uncertainty successes.

(2) New Analysis Regarding Task Difficulty: We adopt LIBERO-Plus’s task difficulty categorization as an operational proxy and show the number of failures and test AUROC across different difficulty levels, for the three policies respectively. We show that there is an increasing trend of predictive power for π0.5 as the tasks get more difficult, but we do not observe the same trends on OpenVLA-OFT and π0-FAST. Overall, these results do not indicate that our model is simply identifying “difficult” task variations.

DifficultyEpisodesπ0.5\pi_{0.5} success rate / test AUROCπ0\pi_0-FAST success rate / test AUROCOpenVLA-OFT success rate / test AUROC
Original LIBERO-1050093.4% / 0.63284.4% / 0.68794.6% / 0.677
129292.5% / 0.66686.3% / 0.75984.9% / 0.812
238692.7% / 0.62785.5% / 0.75286.0% / 0.819
340788.5% / 0.78879.4% / 0.83973.7% / 0.787
441979.2% / 0.78575.2% / 0.87460.9% / 0.862
563250.0% / 0.81567.2% / 0.77637.3% / 0.861
All episodes2,63679.8% / 0.82478.4% / 0.79270.0% / 0.856

[1] Karli, Ulas Berk, et al. "Insight: Inference-time sequence introspection for generating help triggers in vision-language-action models." 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.

AuthorsOpenReview ↗

Response to Reviewer 4WcK (2/3)

W2. The contribution of relational features and the early-prediction capability are not sufficiently isolated. The paper argues that relationships among internal signals provide more informative failure indicators than individual representations. However, the main comparisons involve different feature representations and predictor configurations. A controlled comparison using the same GRU architecture with raw internal signals, simple temporal differences, and the proposed relational features is missing. In particular, it is unclear whether much of the improvement could be explained by simple action replanning or uncertainty-related signals. Furthermore, the evaluation primarily reports per-step classification performance against eventual episode outcomes. It does not directly quantify how far in advance failures are detected relative to their physical occurrence. Thus, the reported AUROC improvements do not fully establish the practical benefit of early warning.

Can the authors compare the proposed relational features against raw internal signals and simple temporal-change features using an identical GRU architecture? This would help isolate the contribution of the proposed feature construction.

New Analysis: Relational Features vs. Raw Signals. We trained a compact raw signal set predictor for each VLA using the same six-input GRU architecture, training procedure, task-disjoint splits, and forward-selection procedure as the compact relational feature-set predictor in Section 6.1. Its candidate pool contains the attention weights, value vectors, hidden states, gradient-based attributions, inputs and action outputs, etc. The relational feature set substantially outperforms the raw signals throughout the rollout for π0.5 and π0-FAST. For OpenVLA-OFT, the two predictors perform similarly during the first 30 steps, after which the relational feature set is generally stronger, with some variation near the end of the rollout. These results show that explicitly modeling relationships among internal signals provides predictive information beyond the raw quantities and their direct temporal changes. We have included this analysis in our updated manuscript Section 6.3. Please see a figure of per-step task-disjoint test AUROC for compact raw signal and relational feature sets in Section 6.3.

AuthorsOpenReview ↗

Response to Reviewer 4WcK (3/3)

Can the authors report detection lead time or pre-failure recall at matched false-positive rates? Such results would clarify whether the proposed method provides actionable warnings before physical failures occur.

We will address this question in two parts:

(1) Lack of Ground Truth for Failure Onset. Among the four baselines (i.e. SAFE [2], FIPER [3], INSIGHT [1], FAIL-Detect [4]), only SAFE compares detection timing with human-annotated failure times (INSIGHT only annotates steps that don’t make progress towards the goal as a help trigger signal). However, SAFE uses judgments of when intervention is needed for ambiguous cases, not of when a failure clearly onsets. Since there is no gold standard or ground truth to identify when failure happens, we attempted at having a human annotator (one of the authors) and an LLM (GPT 6.1-sol) independently annotate 50 failed rollouts from each of the three policies. The audit revealed substantial disagreement in the failure onset steps between the human annotator and the LLM, with Mean Absolute Differences (MAE) of 38.1, 32.6, and 30.7 policy steps for OpenVLA-OFT, π0.5, and π0-FAST, respectively. This disagreement suggests that judgement of failure onset is ambiguous and difficult to scale for large evaluations. For example, one annotator may rule the step where the robot mis-grasps and drops an object as failure onset, whereas another may wait until the robot reaches an unrecoverable failure state. SAFE also discusses similar ambiguity in their human annotation process in Appendix C.3.

(2) New Analysis: Normalized First-Detection Time in Failure Episodes. We aim to predict eventual failures as early as possible within the rollout while using the final episode outcome as the supervision and evaluation target. Following the rollout-relative timing evaluations in FIPER and SAFE, we define normalized first-detection time as the first-detection step divided by episode length. Following FIPER, we compute it over correctly detected failure episodes. We report results averaged over 40 folds at thresholds calibrated to target episode-level FPRs of 5%, 10%, and 20%. Baselines are evaluated at their original operating points. Lower values indicate earlier detection. At the 10% FPR target, our predictor first detects failure after 13.1%, 13.0%, and 13.6% of the rollout for π0.5\pi_{0.5}, π0\pi_0-FAST, and OpenVLA-OFT, respectively. These detection times are earlier than those of every applicable baseline at its original operating point. We have included this analysis in more detail in the updated manuscript Analysis section.

VLA modelOurs 5% FPROurs 10% FPROurs 20% FPRSAFEFAIL-DetectINSIGHTFIPER
π0.5\pi_{0.5}0.169 ± 0.0430.131 ± 0.0380.089 ± 0.0320.214 ± 0.1420.383 ± 0.176—0.476 ± 0.104
π0\pi_0-FAST0.131 ± 0.0500.130 ± 0.0450.118 ± 0.0480.164 ± 0.1490.276 ± 0.1720.291 ± 0.142—
OpenVLA-OFT0.178 ± 0.0420.136 ± 0.0350.106 ± 0.0290.169 ± 0.1450.232 ± 0.100——

[2] Gu, Qiao, et al. "Safe: Multitask failure detection for vision-language-action models." Advances in Neural Information Processing Systems 38 (2026): 40041-40076.

[3] Römer, Ralf, et al. "Failure prediction at runtime for generative robot policies." Advances in Neural Information Processing Systems 38 (2026): 7631-7670.

[4] Xu, Chen, et al. "Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies." arXiv preprint arXiv:2503.08558 (2025).

Reviewer_ApCPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether the relationships and temporal evolution of the internal signals within VLA can predict the eventual mission failure in advance. The method starts from four types of hypotheses: world sensitivity, temporal consistency, attention-attribution agreement, and internal coordination. It constructs features such as attention, value, hidden state, gradient attribution, and cross-time relationships, and uses GRU to learn temporal predictors from episode-level success/failure labels. The experiments cover three types of strategies: π0-FAST, π0.5, and OpenVLA-OFT, as well as LIBERO-10/LIBERO-Plus. They compare various failure detection baselines under episode-random and task-disjoint divisions, and conduct large-scale feature ablation analysis to examine the role of complementary signals.

Critical strengths
  1. The problem setting progresses from "whether a single internal quantity is abnormal" to "whether the relationships among multiple internal signals are mismatched", presenting clear novelty and reasonable motivation.
  2. The four types of assumption-driven feature construction have certain interpretability, and can associate failure evidence with different mechanisms such as world sensitivity, time consistency, attribution consistency, and internal coordination.
  3. At the same time, using episode-random and strict task-disjoint divisions, it avoids overestimating generalization performance by relying solely on random division of the same task rollout.
  4. The experiments cover three VLA structures with significant differences, and clearly specify the scope of application for the interface differences required by the baseline, and make the comparison settings transparent overall.
Critical weaknesses
  1. The predictor uses the final episode label to supervise all prefixes. The early states may not yet contain conclusive evidence of failure, thereby introducing label noise; there is a lack of systematic analysis of the alarm lead time, false alarm duration, and actual intervention benefits.
  2. The validation focused on LIBERO, lacking real robots, and different simulation platforms. Therefore, the external validity of "universal failure prediction" remains limited.
  3. Without conducting an end-to-end assessment of the downstream recovery loop, and merely making "isolated failure predictions", it cannot be proven that this monitoring truly enhances the success rate of the task.
  4. The interpretability and universality of the feature design are insufficient, and the model relies on the specific structure within the VLA, with a high cost for cross-architecture migration.
Questions for authors

Please refer to weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ApCP (1/2)

Thank you for taking the time to review and for recognizing the strengths of our work. We would like to address your concerns as follows:

The predictor uses the final episode label to supervise all prefixes. The early states may not yet contain conclusive evidence of failure, thereby introducing label noise; there is a lack of systematic analysis of the alarm lead time, false alarm duration…

We address this concern in three parts:

(1) Lack of Ground Truth for Failure Onset. Among the four baselines (i.e. SAFE [1], FIPER [2], INSIGHT [3], FAIL-Detect [4]), only SAFE compares detection timing with human-annotated failure times (INSIGHT only annotates steps that don’t make progress towards the goal as a help trigger signal). However, SAFE uses judgments of when intervention is needed for ambiguous cases, not of when a failure clearly onsets. Since there is no gold standard or ground truth to identify when failure happens, we attempted at having a human annotator (one of the authors) and an LLM (GPT 6.1-sol) independently annotate 50 failed rollouts from each of the three policies. The audit revealed substantial disagreement in the failure onset steps between the human annotator and the LLM, with Mean Absolute Differences (MAE) of 38.1, 32.6, and 30.7 policy steps for OpenVLA-OFT, π0.5, and π0-FAST, respectively. This disagreement suggests that judgement of failure onset is ambiguous and difficult to scale for large evaluations. For example, one annotator may rule the step where the robot mis-grasps and drops an object as failure onset, whereas another may wait until the robot reaches an unrecoverable failure state. SAFE also discusses similar ambiguity in their human annotation process in Appendix C.3.

(2) New Analysis: Normalized First-Detection Time in Failure Episodes. We aim to predict eventual failures as early as possible within the rollout while using the final episode outcome as the supervision and evaluation target. Following the rollout-relative timing evaluations in FIPER and SAFE, we define normalized first-detection time as the first-detection step divided by episode length. Following FIPER, we compute it over correctly detected failure episodes. We report results averaged over 40 folds at thresholds calibrated to target episode-level FPRs of 5%, 10%, and 20%. Baselines are evaluated at their original operating points. Lower values indicate earlier detection. At the 10% FPR target, our predictor first detects failure after 13.1%, 13.0%, and 13.6% of the rollout for π0.5\pi_{0.5}, π0\pi_0-FAST, and OpenVLA-OFT, respectively. These detection times are earlier than those of every applicable baseline at its original operating point. We have included this analysis in more detail in the updated manuscript Analysis section.

VLA modelOurs 5% FPROurs 10% FPROurs 20% FPRSAFEFAIL-DetectINSIGHTFIPER
π0.5\pi_{0.5}0.169 ± 0.0430.131 ± 0.0380.089 ± 0.0320.214 ± 0.1420.383 ± 0.176—0.476 ± 0.104
π0\pi_0-FAST0.131 ± 0.0500.130 ± 0.0450.118 ± 0.0480.164 ± 0.1490.276 ± 0.1720.291 ± 0.142—
OpenVLA-OFT0.178 ± 0.0420.136 ± 0.0350.106 ± 0.0290.169 ± 0.1450.232 ± 0.100——

(3) New Analysis: False Alarm Duration in Successful Episodes. We additionally report the fraction of steps above threshold in successful episodes to characterize false-alarm duration. At the 10% episode-level FPR target, successful episodes are above threshold for only 3.8%, 3.4%, and 9.6% of their steps across the three VLA models, showing that false alarms from our predictor are short-lived. Baselines are evaluated at their original operating points. Lower values indicate shorter false alarms.

VLA modelOurs 5% FPROurs 10% FPROurs 20% FPRSAFEFAIL-DetectINSIGHTFIPER
π0.5\pi_{0.5}0.0150.0380.0890.0680.009—0.024
π0\pi_0-FAST0.0170.0340.0690.0400.0180.002—
OpenVLA-OFT0.0630.0960.1490.0320.047——

[1] Gu, Qiao, et al. "Safe: Multitask failure detection for vision-language-action models." Advances in Neural Information Processing Systems 38 (2026): 40041-40076.

[2] Römer, Ralf, et al. "Failure prediction at runtime for generative robot policies." Advances in Neural Information Processing Systems 38 (2026): 7631-7670.

[3] Karli, Ulas Berk, et al. "Insight: Inference-time sequence introspection for generating help triggers in vision-language-action models." 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.

[4] Xu, Chen, et al. "Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies." arXiv preprint arXiv:2503.08558 (2025).

AuthorsOpenReview ↗

Response to Reviewer ApCP (2/2)

The validation focused on LIBERO, lacking real robots, and different simulation platforms. Therefore, the external validity of "universal failure prediction" remains limited.

Thank you for your question. We’d like to clarify that we do not claim the failure features are universal for robot policies; however, in comparison to the baselines, our models demonstrate significantly stronger generalization capability across several axes: (1) tasks unseen during training; (2) two benchmarks; (3) three VLA architectures.

Regarding each axis:

  • (1) We’d like to highlight that in our comparison of episode-random (i.e. tasks seen at predictor training) and task-disjoint (i.e. tasks unseen at predictor training) evaluation settings in Figure 2, we showed that the predictive power of our failure prediction features generalize to tasks unseen at predictor training while all four baseline performances degrade. Although the evaluation settings were focused on LIBERO, the tasks used to train and to test can be very different. For example, a predictor trained on “put both the cream cheese box and the butter in the basket” (a pick-and-place task) also showed predictive power on “turn on the stove and put the moka pot on it” (a manipulation and pick-and-place task), demonstrating generalization capabilities.
  • (2) Additionally, we tested on the LIBERO-10 branch of LIBERO-Plus [5], a benchmark with multiple types of task perturbations (e.g. success rates vary from 28.5% under camera-viewpoint perturbations to 67.5% under background perturbations, compared to 90.6% on original LIBERO) and difficulty levels (e.g. success decreases 85.5% to 63.0% for π₀-FAST and from 97.1% to 72.9% for OpenVLA-OFT, averaged across seven perturbation categories), and showed the effectiveness of our failure prediction models.
  • (3) We’d also like to highlight that our evaluation on three different model family and architectures (OpenVLA-OFT, diffusion-based π0.5, and autoregressive π0-FAST) also provided validity of generalization of our failure prediction features.

... there is a lack of … actual intervention benefits Without conducting an end-to-end assessment of the downstream recovery loop, and merely making "isolated failure predictions", it cannot be proven that this monitoring truly enhances the success rate of the task.

We’d like to clarify that our main contribution lies in failure prediction itself, independent of any downstream recovery method that may take place after a failure is predicted. While we do not prescribe recovery methods in the paper, our failure prediction method is designed to be compatible with those that already exist. We’d love to include suggested experiments as the next steps in our future work. For context, existing literature like SIRIUS-Fleet [6] and ThriftyDAgger [7] have all demonstrated the effectiveness of predicting such anomalies/novelty/risk and engaging human intervention to improve task performance.


The interpretability and universality of the feature design are insufficient, and the model relies on the specific structure within the VLA, with a high cost for cross-architecture migration.

We’d like to clarify that while VLA models are generally not directly interpretable, our features offer interpretability due to how they are designed around four hypotheses (i.e. world sensitivity, attention–attribution agreement, internal coordination, and temporal consistency). In Tables 1-4, we provide a description of the specific aspect of signal relationship that each of the features is capturing. We’d also like to highlight that our evaluation on three different model family and architecture OpenVLA-OFT, π0.5 (diffusion-based), and π0-FAST (autoregressive) also provided validity of generalization of our failure prediction features; thus, we demonstrate that our models do not rely on the specific structure of any particular VLA, and do indeed have cross-architecture compatibility.


[5] Fei, Senyu, et al. "Libero-plus: In-depth robustness analysis of vision-language-action models." arXiv preprint arXiv:2510.13626 (2025).

[6] Liu, Huihan, et al. "Multi-task interactive robot fleet learning with visual world models." arXiv preprint arXiv:2410.22689 (2024).

[7] Hoque, Ryan, et al. "Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning." arXiv preprint arXiv:2109.08273 (2021).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.