- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DART, a proactive runtime failure detection framework designed to detect both sudden and cumulative failures. Specifically, its short-horizon module uses the semantic and action latent states of the VLA model to identify inconsistencies in robot behavior, while its long-horizon module leverages a video world model to predict potential future failures. The final failure decision is made by combining the scores from both modules. Experiments demonstrate that DART outperforms existing failure detection baselines.
Critical strengths
-
Failure detection for VLA models is a timely and important problem in embodied AI.
-
The paper is well written and easy to follow.
-
The experimental results on both simulation and real-world deployment demonstrate that the proposed method consistently outperforms existing failure detection baselines.
Critical weaknesses
-
The short-horizon score is derived from the latent representations of the VLA model, while the long-horizon prediction similarly relies on the latent space of a video world model. This raises concerns about robustness under distribution shift. In particular, it is unclear whether the proposed pipeline can remain effective when the task, embodiment, or environment differs from the data used to train these underlying models.
-
A central limitation is that the proposed framework only predicts a binary success/failure outcome for robotic manipulation. It does not provide richer information such as failure reasoning, failure categorization, or recovery actions, which may limit its practical utility for autonomous recovery.
-
The experimental evaluation lacks comparisons with strong general-purpose VLMs, such as GPT-6 or Gemini-3.1-Pro, making it difficult to assess whether the proposed framework provides a meaningful advantage over modern general-purpose models.
Questions for authors
-
How does DART compare with general-purpose VLMs such as GPT-6 or Gemini-3.1-Pro for failure detection?
-
How well does the proposed pipeline generalize across different VLA models, tasks, and environments? Does DART require retraining or recalibration each time it is deployed with a new VLA model or in a new environment?
-
The computation of the long-horizon score using a video world model is unclear. Since world models typically require an action sequence to predict future states, what actions are provided to the world model in this setting?
-
What are the practical applications of the proposed framework if it only produces a binary failure-detection result without providing additional information such as failure causes, reasoning, or recovery actions?
-
What is the runtime latency and computational overhead of DART, particularly when both the short-horizon module and the video world model are used during online deployment?
I might increase the score if my concern is properly addressed.
3 comments
Response to Reviewer 5Mbf: Generalization Across VLA Models, Tasks, and Environments
Generalization Across VLA Models, Tasks, and Environments
We thank the reviewer for raising this question. DART is designed as an adaptable failure detection framework rather than a detector restricted to a specific VLA model or environment.
First, adaptation to different VLA architectures. The Short-Horizon module requires identifying suitable failure-discriminative internal features from the target VLA. Our experiments on and demonstrate the effectiveness of extracting Vision-Language (VL) states and action-expert hidden states. Our additional feature ablation on -unseen further supports this:
| Features | Recall | Precision | Bal. Acc. | DT | AT |
|---|---|---|---|---|---|
| Hidden-only | 0.8447 | 0.7733 | 0.8239 | 0.0887 | 0.1946 |
| VL-only | 0.9029 | 0.7915 | 0.8569 | 0.0626 | 0.2114 |
| VL+Hidden (Short) | 0.8835 | 0.7948 | 0.8510 | 0.0527 | 0.2001 |
VL-only achieves the highest Recall (0.9029) and Balanced Accuracy (0.8569), only slightly underperforming VL+Hidden in temporal responsiveness (DT: 0.0626 vs. 0.0527). This demonstrates that readily accessible VL backbone features alone provide strong failure-discriminative information. Combining VL and hidden features achieves the highest Precision (0.7948) and lowest DT (0.0527), further demonstrating their complementarity.
Moreover, prior work such as SAFE has explored informative internal representations across different VLA architectures, supporting the feasibility of identifying suitable features beyond our evaluated policies. Therefore, adapting Short to another VLA requires selecting architecture-appropriate features rather than redesigning the entire framework.
Second, adaptation to new environments. The Long-Horizon module requires environment-specific fine-tuning of the Wan video world model. We have already established a reusable fine-tuning pipeline using execution videos and corresponding task instructions. Thus, adapting to a new environment primarily involves collecting a moderate amount of video-text data and fine-tuning the existing model, rather than training a world model from scratch.
Finally, adaptation of the overall framework. We acknowledge that deployment on a new VLA or environment generally requires suitable feature extraction, world-model fine-tuning, retraining the detection networks, and recalibrating the detection threshold. Nevertheless, the complete pipeline remains reusable, including automated timestep-level annotation, Short-Horizon feature-based detection, Long-Horizon future-state prediction, horizon-aligned supervision, and dual-horizon fusion.
Our evaluation across four LIBERO suites, two VLA policies ( and ), and four real-world manipulation tasks supports the framework's applicability across different tasks and settings. While DART does not claim zero-shot transfer to arbitrary VLA architectures or environments, its modular design enables adaptation without changing the core methodology.
Response to Reviewer 5Mbf: World Model Conditioning, Practical Utility, and Runtime Efficiency
World Model Conditioning, Practical Utility, and Runtime Efficiency
1. World Model Conditioning. We thank the reviewer for pointing out this ambiguity. DART employs Wan, a powerful video-generation world model, fine-tuned on execution videos from the target manipulation environment to predict future visual dynamics. Unlike conventional action-conditioned world models, Wan does not require explicit future robot action sequences in our implementation. Instead, it takes the current visual observation, task instruction, and normalized execution progress as conditioning inputs. The predicted future latent representations are temporally aggregated and used to compute the Long-Horizon failure score, without video decoding during inference. We will clarify this mechanism in the revised manuscript.
2. Practical Utility of Binary Failure Detection. DART is designed for runtime failure monitoring, rather than failure explanation or recovery planning. Its primary purpose is to provide timely risk warnings that can trigger downstream safety mechanisms, such as emergency stopping, human intervention, or switching to an independent recovery policy.
Importantly, a binary warning can be useful without identifying the precise failure cause, provided that it is sufficiently accurate, timely, and computationally feasible. We therefore evaluate detection reliability using Recall, Precision, and Balanced Accuracy, and temporal responsiveness using DT, AT, and alarm timing distributions. In particular, the Long-Horizon module enables proactive warnings before failures become explicitly observable, potentially providing additional time for intervention.
We acknowledge that early detection does not necessarily guarantee successful intervention. The effectiveness of stopping or recovery also depends on the available intervention window and the downstream controller. Evaluating closed-loop intervention and recovery remains an important direction for future work.
3. Runtime Latency and Computational Overhead. As reported in Section 4.7, Short-Horizon scoring adds approximately 13 ms of overhead, while Long-Horizon inference takes approximately 1.197 s on an H100 GPU. The Long-Horizon module runs asynchronously alongside robot execution, without blocking the VLA control loop. Thus, the full framework does not require synchronous world-model inference at every control step. Short provides rapid current-state monitoring, while Long supplies predictive risk information once its asynchronous inference completes.
These results demonstrate the feasibility of integrating both modules into a runtime monitoring framework while preserving the normal execution of the VLA policy.
Response to Reviewer 5Mbf: Comparison with General-Purpose VLMs
Comparison with General-Purpose VLMs
We thank the reviewer for this suggestion. To compare DART with direct online VLM-based failure detection, we conducted additional experiments using Qwen3-VL-8B-Instruct. Due to limited time and computational resources, we randomly sampled 200 trajectories from π0-unseen using seed=42, including 85 failed and 115 successful trajectories. All methods were evaluated on the same trajectories using human annotations as ground truth.
All three DART configurations outperform direct online VLM detection in Recall, Precision, Balanced Accuracy, and DT. In particular, Short improves Balanced Accuracy from 0.7151 to 0.8396 and Recall from 0.6824 to 0.8706, while Full reduces DT from 0.1769 to 0.0021, demonstrating stronger detection accuracy and temporal responsiveness.
To further explain these improvements, we conducted two additional comparisons.
First, online VLM detection vs. offline multi-scale annotation. On the same 85 failed trajectories, offline annotation improves Recall from 0.6824 to 0.8235 and reduces DT from 0.1769 to 0.0693. This demonstrates the advantages of offline multi-scale analysis in producing effective failure supervision.
Second, offline VLM labels vs. DART. On all 206 failed trajectories from π0-unseen, directly thresholding offline VLM labels achieves Recall of 0.8204 and DT of 0.0656, whereas DART Short achieves 0.8835 and 0.0539, and Full achieves 0.8495 and 0.0131, respectively. This indicates that DART can learn effective failure representations from automatically generated supervision and further improve detection performance.
Together, these experiments reveal a clear progression: direct online VLM detection → offline multi-scale VLM annotation → DART trained with automated supervision. Rather than deploying the VLM directly for runtime detection, our approach leverages its reasoning capability offline to generate fine-grained supervision, allowing dedicated detection modules to achieve stronger accuracy and temporal responsiveness.
We expect this principle to extend to more powerful general-purpose VLMs, such as GPT-6 and Gemini-3.1-Pro. Specifically, stronger VLMs may produce higher-quality offline annotations, enabling DART to learn more informative failure representations. Therefore, we hypothesize that training DART with annotations generated by these stronger models could outperform directly deploying the same VLMs for online failure detection, following the same progression observed with Qwen3-VL-8B-Instruct. While this expectation remains to be empirically validated, it highlights the broader applicability of our offline-supervision and online-detection framework.
Computational efficiency. Direct online Qwen3-VL-8B-Instruct detection requires an average of 4.39 s per inference, compared with approximately 13 ms for DART Short and 1.197 s for DART Long, which runs asynchronously without blocking robot execution (Section 4.7). This further demonstrates the computational advantages of using VLMs for offline supervision rather than direct online detection.