- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark for evaluating failure detection in vision-language models (VLMs), comprising 2,197 samples collected from 14 public sources. The results show that general-purpose VLMs outperform models specifically fine-tuned for robotic failure detection. The authors also propose an input-level intervention method that further improves failure-detection accuracy.
Critical strengths
- Failure detection in vision-language models (VLMs) is an important problem, particularly for reliable robotic deployment.
- The benchmark covers diverse data sources and includes three distinct types of failure data: organic, planned, and synthetic failures.
Critical weaknesses
-
The authors are encouraged to improve the overall writing quality and logical flow of the paper. In the current version, many sentences and paragraphs appear disconnected, making the paper difficult to follow. The Related Work section is also overly long and lacks clear organization. In addition, the first figure does not appear until page 5. More figures or illustrative examples could be included earlier to better communicate the motivation, methodology, and main contributions.
-
The benchmark contains fewer than 3,000 samples, which is relatively small for a representative evaluation benchmark. In addition, the coverage of robotic tasks is unclear, and the embodied platforms appear to be dominated by parallel-jaw robotic arms. The limited scale, task diversity, and embodiment diversity may reduce the representativeness and generalizability of the benchmark.
-
The proposed input-intervention method is relatively simple, essentially cropping the task-relevant region of the input image. Moreover, it achieves only a 2.3% improvement, which is fairly limited. The effectiveness of this method is also demonstrated only on the Gemini 3 Flash model, making it difficult to assess whether the improvement generalizes across different VLMs.
-
The benchmark evaluates only binary failure-detection accuracy, i.e., classifying each sample as either failure or success. This evaluation setting may provide limited insight into the capabilities required for real-world robotic deployment, where identifying the failure type, understanding its cause, localizing the failure, and suggesting appropriate recovery actions are also important.
Questions for authors
-
What tasks and robot embodiments are covered by the benchmark? Are the selected tasks equally fair for evaluating both general-purpose VLMs and models specifically fine-tuned on robotic failure-detection datasets?
-
Could the authors evaluate more recent models, such as GPT-5, GPT-6, Gemini 3.1 Pro, or Gemini 3.8 Flash? In particular, would models such as GPT-6 Astra, which may have stronger embodied reasoning capabilities, achieve better failure-detection performance?
-
Does the proposed image-cropping intervention generalize to other VLMs, or is the improvement specific to the evaluated model?
Thank you for the detailed review.
Reasons
R1. [Writing] - "many sentences and paragraphs appear disconnected ... The Related Work section is also overly long ... the first figure does not appear until page 5."
Ans. We agree that the presentation could be improved and will improve it in the camera-ready version.
R2. [Scale, tasks, embodiments] - "The benchmark contains fewer than 3,000 samples ... the coverage of robotic tasks is unclear, ... dominated by parallel-jaw ..."
Ans. We appreciate this concern and provide more details below. The submitted FailBench covers more than 15 robot models and 10 task categories across 14 independent data sources, providing diverse settings for evaluating cross-source generalization. The tasks include pick/place, opening and closing, assembly, tool-use, multi-step tasks, deformable-object handling, pouring etc. The platforms include Franka Panda, SO-101, Google Robot, UR5, Franka FR3, Flexiv Rizon, WidowX and others. We acknowledge that the original benchmark primarily focuses on tabletop manipulation with parallel-jaw grippers. Dexterous hands and humanoid robots were outside its initial scope. FailBench was designed to be extensible, and expanding its embodiment coverage is among our planned next steps. We are already actively working on this expansion as part of the ongoing extension of FailBench. We will provide a more detailed breakdown in the revised appendix.
R3. [Crop intervention] - "The proposed input-intervention method is relatively simple ... it achieves only a 2.3% improvement..."
Ans. With the evidence-localization pipeline, our intention was not to introduce a novel methodological contribution or claim generalizability across VLMs, but rather to investigate whether an existing technique could also benefit robotic failure detection.
The observed 2.3% improvement demonstrates that this approach can be effective in the evaluated setting. We present this as an empirical insight rather than a methodological contribution and do not claim that the improvement necessarily extends to other models. We will clarify this distinction in the camera-ready version.
R4. [Binary labels only] - "identifying the failure type, understanding its cause ... actions are also important."
Ans. We fully agree that the suggested dimensions are important. Our focus in FailBench is specifically on binary success/failure detection. Our results demonstrate that even this fundamental task remains challenging for current VLMs. Certain capabilities, such as failure-type classification, could be incorporated as extensions of FailBench as mentioned in future work, while others, such as recovery planning, may become dedicated benchmarks with different evaluation protocols.
Questions
Q1. - "What tasks and robot embodiments are covered by the benchmark? Are the selected tasks equally fair for ... general-purpose VLMs and models specifically fine-tuned ..."
Ans. The submitted FailBench covers more than 15 robot models and 10 task categories across 14 independent sources. More details are in R2. To ensure fairness, each judge runs with its own published prompt and inference settings. A specialist is not at a disadvantage. On its own sources (UR5-Fail and BDV2-Fail), Guardian ranks 2nd of 14 models, and 7th on all 13 subsets. The drop comes from the sources it was not trained on.
Q2. - "Could the authors evaluate more recent models, such as GPT-5, GPT-6, Gemini 3.1 Pro, or Gemini 3.8 Flash? In particular, would models such as GPT-6 Astra ... achieve better failure-detection performance?"
Ans. Although our original experiments were completed before these newer models could be incorporated, we have now extended the evaluation to GPT-6 Astra, Claude Opus 5.5, and Gemini 3.8 Flash from 3 different providers.
GPT-6 Astra achieves the strongest performance. Nevertheless, substantial challenges remain: its overall score is 0.86, and performance remains at or below 0.79 on several challenging subsets. Thus, while newer frontier models demonstrate clear progress in robotic failure detection, they do not saturate FailBench. We will extend the results section with these evaluations.
Q3. - "Does the proposed image-cropping intervention generalize to other VLMs, or is the improvement specific to the evaluated model?"
Ans. We have not evaluated the intervention across other VLMs, as our objective was to demonstrate its potential effectiveness rather than establish a methodological contribution. As clarified in R3, we present this result as an empirical finding specific to the evaluated model and do not claim that the improvement generalizes to other VLMs.