acceptodds
Under review as a conference paper at ICLR 2027

FailBench: Benchmark for robotic manipulation failure detection

Abstract

Vision-language models (VLMs) are increasingly used to determine whether a robot manipulation attempt has succeeded. These judgments can serve as reinforcement-learning rewards, training-data filters, policy-ranking signals, or triggers for retry, making reliable failure detection critical for robot learning and evaluation. However, existing benchmarks provide limited evidence of cross-domain generalization. We introduce FailBench, a benchmark for robot failure detection comprising 2,197 manipulation attempts from 14 public sources, including 12 real-world and 2 simulated datasets, using the original outcome labels provided by their respective sources. Six of the real-world sources were originally collected for policy evaluation, reward modeling, or general data collection rather than failure detection, and 76% of failures occur naturally rather than being deliberately constructed. Evaluating 13 VLM-based detectors, we find that the best model achieves only 0.77 mean balanced accuracy, indicating substantial room for improvement. Models fine-tuned specifically for robot failure detection consistently underperform general-purpose VLMs, with the exception of the smallest general-purpose model, and also underperform their corresponding pretrained models. We further find that performance depends more strongly on the visual evidence required to determine the outcome than on the robot or task domain: detection approaches saturation when success is determined by observable object motion, but approaches chance when success depends on establishing contact, with no model exceeding 0.60 balanced accuracy on contact-intensive assembly tasks. Increasing reasoning effort does not alleviate this bias, as incorrect predictions tend to receive longer reasoning traces. Model-level interventions provide little improvement, while an input-level intervention is effective: spatially localizing the outcome-relevant region and cropping the input improves the strongest detector by 2.3 percentage points without additional training. We release FailBench and the accompanying evaluation harness to facilitate reproducible evaluation of robot failure detection.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Robot Manipulation, Failure Detection, Robotics Benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gYHHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark for evaluating failure detection in vision-language models (VLMs), comprising 2,197 samples collected from 14 public sources. The results show that general-purpose VLMs outperform models specifically fine-tuned for robotic failure detection. The authors also propose an input-level intervention method that further improves failure-detection accuracy.

Critical strengths
  1. Failure detection in vision-language models (VLMs) is an important problem, particularly for reliable robotic deployment.
  2. The benchmark covers diverse data sources and includes three distinct types of failure data: organic, planned, and synthetic failures.
Critical weaknesses
  1. The authors are encouraged to improve the overall writing quality and logical flow of the paper. In the current version, many sentences and paragraphs appear disconnected, making the paper difficult to follow. The Related Work section is also overly long and lacks clear organization. In addition, the first figure does not appear until page 5. More figures or illustrative examples could be included earlier to better communicate the motivation, methodology, and main contributions.

  2. The benchmark contains fewer than 3,000 samples, which is relatively small for a representative evaluation benchmark. In addition, the coverage of robotic tasks is unclear, and the embodied platforms appear to be dominated by parallel-jaw robotic arms. The limited scale, task diversity, and embodiment diversity may reduce the representativeness and generalizability of the benchmark.

  3. The proposed input-intervention method is relatively simple, essentially cropping the task-relevant region of the input image. Moreover, it achieves only a 2.3% improvement, which is fairly limited. The effectiveness of this method is also demonstrated only on the Gemini 3 Flash model, making it difficult to assess whether the improvement generalizes across different VLMs.

  4. The benchmark evaluates only binary failure-detection accuracy, i.e., classifying each sample as either failure or success. This evaluation setting may provide limited insight into the capabilities required for real-world robotic deployment, where identifying the failure type, understanding its cause, localizing the failure, and suggesting appropriate recovery actions are also important.

Questions for authors
  1. What tasks and robot embodiments are covered by the benchmark? Are the selected tasks equally fair for evaluating both general-purpose VLMs and models specifically fine-tuned on robotic failure-detection datasets?

  2. Could the authors evaluate more recent models, such as GPT-5, GPT-6, Gemini 3.1 Pro, or Gemini 3.8 Flash? In particular, would models such as GPT-6 Astra, which may have stronger embodied reasoning capabilities, achieve better failure-detection performance?

  3. Does the proposed image-cropping intervention generalize to other VLMs, or is the improvement specific to the evaluated model?

1 comment

Thank you for the detailed review.

Reasons

R1. [Writing] - "many sentences and paragraphs appear disconnected ... The Related Work section is also overly long ... the first figure does not appear until page 5."

Ans. We agree that the presentation could be improved and will improve it in the camera-ready version.

R2. [Scale, tasks, embodiments] - "The benchmark contains fewer than 3,000 samples ... the coverage of robotic tasks is unclear, ... dominated by parallel-jaw ..."

Ans. We appreciate this concern and provide more details below. The submitted FailBench covers more than 15 robot models and 10 task categories across 14 independent data sources, providing diverse settings for evaluating cross-source generalization. The tasks include pick/place, opening and closing, assembly, tool-use, multi-step tasks, deformable-object handling, pouring etc. The platforms include Franka Panda, SO-101, Google Robot, UR5, Franka FR3, Flexiv Rizon, WidowX and others. We acknowledge that the original benchmark primarily focuses on tabletop manipulation with parallel-jaw grippers. Dexterous hands and humanoid robots were outside its initial scope. FailBench was designed to be extensible, and expanding its embodiment coverage is among our planned next steps. We are already actively working on this expansion as part of the ongoing extension of FailBench. We will provide a more detailed breakdown in the revised appendix.

R3. [Crop intervention] - "The proposed input-intervention method is relatively simple ... it achieves only a 2.3% improvement..."

Ans. With the evidence-localization pipeline, our intention was not to introduce a novel methodological contribution or claim generalizability across VLMs, but rather to investigate whether an existing technique could also benefit robotic failure detection.

The observed 2.3% improvement demonstrates that this approach can be effective in the evaluated setting. We present this as an empirical insight rather than a methodological contribution and do not claim that the improvement necessarily extends to other models. We will clarify this distinction in the camera-ready version.

R4. [Binary labels only] - "identifying the failure type, understanding its cause ... actions are also important."

Ans. We fully agree that the suggested dimensions are important. Our focus in FailBench is specifically on binary success/failure detection. Our results demonstrate that even this fundamental task remains challenging for current VLMs. Certain capabilities, such as failure-type classification, could be incorporated as extensions of FailBench as mentioned in future work, while others, such as recovery planning, may become dedicated benchmarks with different evaluation protocols.

Questions

Q1. - "What tasks and robot embodiments are covered by the benchmark? Are the selected tasks equally fair for ... general-purpose VLMs and models specifically fine-tuned ..."

Ans. The submitted FailBench covers more than 15 robot models and 10 task categories across 14 independent sources. More details are in R2. To ensure fairness, each judge runs with its own published prompt and inference settings. A specialist is not at a disadvantage. On its own sources (UR5-Fail and BDV2-Fail), Guardian ranks 2nd of 14 models, and 7th on all 13 subsets. The drop comes from the sources it was not trained on.

Q2. - "Could the authors evaluate more recent models, such as GPT-5, GPT-6, Gemini 3.1 Pro, or Gemini 3.8 Flash? In particular, would models such as GPT-6 Astra ... achieve better failure-detection performance?"

Ans. Although our original experiments were completed before these newer models could be incorporated, we have now extended the evaluation to GPT-6 Astra, Claude Opus 5.5, and Gemini 3.8 Flash from 3 different providers.

ModelMean balanced accuracy95% CI
GPT-6 Astra0.8660.84 to 0.88
Claude Opus 5.50.8340.81 to 0.85
Gemini 3.8 Flash0.8210.80 to 0.83
Gemini 3 Flash (best in paper)0.760.74 to 0.78
Random answer0.500

GPT-6 Astra achieves the strongest performance. Nevertheless, substantial challenges remain: its overall score is 0.86, and performance remains at or below 0.79 on several challenging subsets. Thus, while newer frontier models demonstrate clear progress in robotic failure detection, they do not saturate FailBench. We will extend the results section with these evaluations.

Q3. - "Does the proposed image-cropping intervention generalize to other VLMs, or is the improvement specific to the evaluated model?"

Ans. We have not evaluated the intervention across other VLMs, as our objective was to demonstrate its potential effectiveness rather than establish a methodological contribution. As clarified in R3, we present this result as an empirical finding specific to the evaluated model and do not claim that the improvement generalizes to other VLMs.

Reviewer_NsvMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FailBench, a benchmark comprising robotic manipulation attempts collected from 14 public data sources, with labels mapped to a common binary success/failure format. The authors evaluate 13 general-purpose, embodied, and specialized failure detectors, finding that the best model achieves only 0.77 macro balanced accuracy and that specialized detectors generally underperform their corresponding base models. Through qualitative error analysis, the authors suggest that fine-grained physical contact understanding remains challenging. They further propose an evidence-localization pipeline that identifies and crops task-relevant visual regions, improving detection performance by approximately 2.3 percentage points without retraining.

Critical strengths

The paper systematically evaluates 13 failure detectors across 14 diverse data sources, covering both real-world and simulated robotic manipulation tasks. The unified evaluation protocol and comparisons with the detectors' base models provide useful empirical evidence of performance differences and generalization limitations across data sources.

Critical weaknesses
  1. Lack of Novelty. This work primarily constructs a unified robotic failure detection benchmark by aggregating existing public datasets. Although it incorporates more data sources, prior works such as Guardian and PRIMO R1 have already explored cross-distribution evaluation of robotic failure detection, making it difficult to establish the unique evaluation value of the additional data. Furthermore, the proposed Evidence Localization pipeline mainly uses a VLM to localize task-relevant regions and crop images, which is highly similar to existing approaches such as V* and Zoom-Refine, offering limited methodological novelty.

  2. Limitations in Label Consistency and Evaluation Granularity. Different data sources employ different success criteria, annotation rules, and temporal granularities, and directly unifying them into binary labels may introduce semantic inconsistencies. Moreover, binary classification discards fine-grained information such as failure types and task progress, limiting the benchmark's diagnostic value.

Questions for authors
  1. What unique evaluation capabilities or scientific insights does FailBench provide beyond existing benchmarks such as Guardian and PRIMO Bench? Could the authors provide comparative analyses or source ablations to demonstrate the added value of aggregating 14 data sources?
  2. How do the authors ensure label consistency when converting datasets with different success criteria, annotation rules, and temporal granularities into binary outcomes? How sensitive are the evaluation results to these conversions and the exclusion of ambiguous or near-completion samples?
  3. What evidence supports the conclusion that fine-grained physical contact understanding is a major bottleneck in failure detection? Could the observed performance differences instead be explained by dataset-specific factors such as task difficulty, camera viewpoints, or visual ambiguity?
1 comment

Thank you for the careful review.

Reasons

R1. [Novelty vs Guardian and PRIMO]

Ans. FailBench covers a wide range of robots, tasks and recording conditions. It has 2,197 samples (1,176 failures and 1,021 successes) from 14 sources. They cover 16 robot types, from single arms to bimanual setups, and 12 task types, from pick and place to insertion, assembly, folding, pouring and tool use. The samples come from policy rollouts and human teleoperation, in real and simulated scenes, with one or several camera views. 76% of the failures happened during normal robot runs.

PRIMO. PRIMO Bench measures task progress (0 to 100), not success or failure. PRIMO R1 tests failure detection on one set, RoboFail (100 examples), against 14 sources (2,197 samples) in FailBench.

Guardian. Guardian evaluates models on the benchmarks of its own paper, and there it scores best of all models tested. It evaluates both in-distribution and out-of-distribution performance, but its execution check uses frames before and after the action. FailBench also tests full videos, multi-step and contact-rich tasks, and failures that happened naturally under different policies and in teleoperation. On FailBench, Guardian scores 0.796 balanced accuracy on its own sources (2nd of 14 models), but only 0.596 on the 11 other sources.

R2. [Evidence localization vs V* and Zoom-Refine]

Ans. We agree that the evidence-localization pipeline shares its core idea with existing visual search and zoom methods, including V* and Zoom-Refine. We do not claim novelty in cropping itself. We apply an established method and show that it can also help failure detection. It was not our intention to present it as a methodological contribution. It is an empirical finding. In the camera-ready version we will cite these papers and make this clear.

R3. [Label consistency and binary labels] - "Different data sources employ different success criteria, annotation rules, and temporal granularities, and directly unifying them into binary labels may introduce semantic inconsistencies. Moreover, binary classification discards fine-grained information such as failure types and task progress, limiting the benchmark's diagnostic value."

Ans. Consistency. See Q2.

Granularity. We absolutely agree, and we state this in our Limitations and Future Work sections. Even so, our results show that even this simplest form of the task (binary classification) is not solved.

Questions

Q1. - "What unique evaluation capabilities or scientific insights does FailBench provide beyond existing benchmarks such as Guardian and PRIMO Bench? Could the authors provide comparative analyses or source ablations ...?"

Ans. See R1.

Q2. - "How do the authors ensure label consistency when converting datasets ...? How sensitive are the evaluation results to these conversions and the exclusion of ambiguous or near-completion samples?"

Ans. The sources were not designed for failure benchmarking, but many of them give a direct binary label. 11 of the 14 sources label each sample as success or failure, and we use that label with no conversion. The other 3 (RH20T, RoboReward, RoboArena) give a grade or a progress value, and one fixed rule per source maps it to success or failure. The prompts for our evaluations state the original source's success/failure definition and the task being judged.

We exclude ambiguous labels, for example score 4 ("Near Completion") in RoboReward, the "suboptimal" tier of ARMNetBench and RoboMeter sim, and the "partial progress" clips of RoboMeter. We do not know how sensitive the results are to this exclusion, because these samples are ambiguous even for a human annotator. They can be useful for other tasks such as progress estimation, but they are not suitable for a success/failure benchmark.

Q3. - "What evidence supports the conclusion that fine-grained physical contact understanding is a major bottleneck ...? Could the observed performance differences instead be explained by dataset-specific factors ...?"

Ans. We agree. The statement that contact is a major bottleneck is a hypothesis, not a conclusion backed by quantitative data, and the paper should have said so clearly. We will state it as a hypothesis that needs more testing. A new result points to perception. GPT-6 Astra scores 0.905 on REASSEMBLE, where the models in the paper score 0.52 on average and 0.60 at best. This suggests that the visual perception of the model plays a large role. We will test this and add the results in the camera-ready version.

Reviewer_ZkixOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose FailBench for testing if models can determine failures. The proposed FailBench consists of diverse real world and simulated data, and the benchmark demonstrates current models cannot reliably detect failures. The authors test a variety of both open and closed source models.

Critical strengths
  • The FailBench is useful to determine when failures occur, and is genuinely useful compared to what currently exists
  • The FailBench has a mixture of real-world and simulated data which is useful for measuring generalization.
  • The insight from results was interesting and will be useful for developing future robotics foundation models (given current robotics foundation models still struggle, see weaknesses).
Critical weaknesses
  • The models used are pretty outdated, so it's unclear whether results will generalize to newer models (e.g., does astra saturate this benchmark)? I see this as the biggest flaw of this paper, as it makes it unclear whether it's still useful given how much current models have progressed. Even just measuring one more recent model from openai and anthropic and seeing how it does would make me significantly more confident in the benchmark. Without this, it is hard to advocate strongly for the paper. If this were fixed significantly, I would be a lot more confident in this work.
  • The claim that model-level interventions don't help much isn't supported that well.
  • The benchmark is quite small in terms of the number of samples, so the variance may be a bit higher.
Questions for authors

Why not use newer models for evaluations? Does Astra still fail at many of failure detection tasks? Does it become saturated? Can we scale FailBench to even more data, if not why not? How high is the variance in most cases?

1 comment

We thank the reviewer for the constructive feedback, particularly regarding evaluation on newer models, statistical uncertainty, and the scope of our conclusions.

Reasons

R1. [Newer models] - "The models used are pretty outdated, so it's unclear whether results will generalize to newer models ... Even just measuring one more recent model from openai and anthropic and seeing how it does would make me significantly more confident in the benchmark."

Ans. Our initial evaluation included frontier models available during manuscript preparation. The requested newer models were released close to the submission deadline, after our original experimental evaluation had been finalized. Nevertheless, we agree that evaluating them is important to establish FailBench's continued relevance as model capabilities advance. We have therefore evaluated three additional models, GPT-6 Astra (OpenAI), Claude Opus 5.5 (Anthropic) and Gemini 3.8 Flash (Google).

ModelMean balanced accuracy (BA)95% CI
GPT-6 Astra0.8660.848 to 0.882
Claude Opus 5.50.8340.815 to 0.851
Gemini 3.8 Flash0.8210.802 to 0.839
Gemini 3 Flash (best in paper)0.7660.747 to 0.783
Random answer0.500

The newer models substantially improve performance. However, FailBench is not saturated. The strongest model still reaches only 0.866 mean BA, and no evaluated model exceeds 0.81 on some of the challenging subsets. These results confirm that FailBench remains challenging even for newer frontier models. We will incorporate the new models' results.

R2. [Interventions] - "The claim that model-level interventions don't help much isn't supported that well."

Ans. We agree that our original claim was too broad and will narrow it to the interventions evaluated in this work:

Model ensembling: Majority voting across three models improves BA by 1.3 percentage points over Gemini 3 Flash alone. However, the 95% CI [−0.4, +3.0] includes zero, so the improvement is not statistically conclusive.

Task-specific fine-tuning: All failure-detection specialists underperform their base models on the cross-source evaluation, with statistically supported decreases for three of them.

Given the paper's length constraints, we included only selected findings in the submitted version. We will provide more detailed results and supporting analyses in the camera-ready version and refine our conclusions to reflect the scope of the evidence.

R3. [Variance] - "The benchmark is quite small in terms of the number of samples, so the variance may be a bit higher."

Ans. FailBench comprises 2,197 episodes (1,176 failures and 1,021 successes) across 14 independent sources. Under independent Bernoulli sampling, the conservative theoretical 95% margin of error for pooled BA is approximately ±2.1 percentage points. More importantly, empirical 95% bootstrap confidence interval margins for benchmark-level mean BA range from ±0.9 to ±2.1 percentage points across evaluated models. These results indicate relatively low sampling uncertainty in the aggregate estimates. We agree, however, that variance can be a concern for smaller subsets, which exhibit wider confidence intervals. We will therefore report confidence intervals alongside individual subset scores and interpret subset-level differences cautiously.

For context, comparable evaluation sizes have been used in multiple benchmarks that are commonly used, including OmniSpatial (ICLR26), with a human-annotated test set of 1.5K question-answer pairs, and CV-Bench (NeurIPS24), with 2,638 examples. FailBench size is therefore comparable in scale, while additionally covering 14 independent sources with diverse robotic tasks and failure types.

Questions

Q1. - "Why not use newer models for evaluations?"

Ans. We evaluated frontier models available when our original experiments were finalized. Nevertheless, to ensure FailBench's continued relevance as model capabilities advance, we have now evaluated three additional newer models (see R1).

Q2. - "Does Astra still fail at many of failure detection tasks? Does it become saturated?"

Ans. Yes, it still fails, and no, it does not saturate. GPT-6 Astra is the best model (0.86), but it scores 0.79 or lower on some of the challenging subsets. See R1.

Q3. - "Can we scale FailBench to even more data, if not why not?"

Ans. Yes. FailBench is designed to be extensible. The current benchmark prioritizes cross-source diversity, balanced outcomes where feasible, heterogeneous failure types and verified labels rather than maximizing the number of highly correlated examples from a few environments. We can expand FailBench by sampling additional eligible episodes and incorporating new sources, embodiments and tasks as stated in the future work.

Q4. - "How high is the variance in most cases?"

Ans. ±0.9 to ±2.1 points on the benchmark mean (R3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.