acceptodds
Desk rejected at ICLR 2027

DART: Proactive Runtime Failure Detection for Vision-Language-Action Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in Vision-Language-Action (VLA) models have enabled strong generalization across diverse robotic manipulation tasks, yet their real-world deployment remains limited by the lack of reliable runtime safety mechanisms. Existing failure detection methods typically rely on coarse trajectory-level supervision or reactive failure scoring, limiting their ability to identify subtle early-stage deviations and rapidly emerging failures. We propose DART, a proactive runtime failure detection framework for VLA policies that jointly addresses sudden failures and cumulative long-horizon deviations through a dual-horizon architecture: a short-horizon module leverages internal semantic and action representations from the VLA to capture abrupt behavioral inconsistencies, while a long-horizon module utilizes a video generation world model to anticipate future state drift before failures fully manifest. To provide fine-grained supervision, we further introduce an automated VLM-based annotation pipeline that generates timestep-level failure labels from execution trajectories while reducing the need for dense manual annotation. Extensive experiments across simulated and real-world manipulation environments on multiple VLA policies demonstrate that DART achieves a favorable trade-off between detection accuracy and temporal responsiveness, particularly under unseen tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Runtime Failure Detection, Robot Safety, Embodied AI, Uncertainty Estimation, World Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5MbfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DART, a proactive runtime failure detection framework designed to detect both sudden and cumulative failures. Specifically, its short-horizon module uses the semantic and action latent states of the VLA model to identify inconsistencies in robot behavior, while its long-horizon module leverages a video world model to predict potential future failures. The final failure decision is made by combining the scores from both modules. Experiments demonstrate that DART outperforms existing failure detection baselines.

Critical strengths
  1. Failure detection for VLA models is a timely and important problem in embodied AI.

  2. The paper is well written and easy to follow.

  3. The experimental results on both simulation and real-world deployment demonstrate that the proposed method consistently outperforms existing failure detection baselines.

Critical weaknesses
  1. The short-horizon score is derived from the latent representations of the VLA model, while the long-horizon prediction similarly relies on the latent space of a video world model. This raises concerns about robustness under distribution shift. In particular, it is unclear whether the proposed pipeline can remain effective when the task, embodiment, or environment differs from the data used to train these underlying models.

  2. A central limitation is that the proposed framework only predicts a binary success/failure outcome for robotic manipulation. It does not provide richer information such as failure reasoning, failure categorization, or recovery actions, which may limit its practical utility for autonomous recovery.

  3. The experimental evaluation lacks comparisons with strong general-purpose VLMs, such as GPT-6 or Gemini-3.1-Pro, making it difficult to assess whether the proposed framework provides a meaningful advantage over modern general-purpose models.

Questions for authors
  1. How does DART compare with general-purpose VLMs such as GPT-6 or Gemini-3.1-Pro for failure detection?

  2. How well does the proposed pipeline generalize across different VLA models, tasks, and environments? Does DART require retraining or recalibration each time it is deployed with a new VLA model or in a new environment?

  3. The computation of the long-horizon score using a video world model is unclear. Since world models typically require an action sequence to predict future states, what actions are provided to the world model in this setting?

  4. What are the practical applications of the proposed framework if it only produces a binary failure-detection result without providing additional information such as failure causes, reasoning, or recovery actions?

  5. What is the runtime latency and computational overhead of DART, particularly when both the short-horizon module and the video world model are used during online deployment?

I might increase the score if my concern is properly addressed.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5Mbf: Comparison with General-Purpose VLMs

Comparison with General-Purpose VLMs

We thank the reviewer for this suggestion. To compare DART with direct online VLM-based failure detection, we conducted additional experiments using Qwen3-VL-8B-Instruct. Due to limited time and computational resources, we randomly sampled 200 trajectories from π0\pi_0-unseen using seed=42, including 85 failed and 115 successful trajectories. All methods were evaluated on the same trajectories using human annotations as ground truth.

MethodRecallPrecisionBal. Acc.DTAT
Online Qwen3-VL-8B0.68240.66670.71510.17690.3186
DART Short0.87060.77080.83960.03170.1849
DART Long0.74120.85140.82280.00000.3757
DART Full0.82350.79550.83350.00210.2965

All three DART configurations outperform direct online VLM detection in Recall, Precision, Balanced Accuracy, and DT. In particular, Short improves Balanced Accuracy from 0.7151 to 0.8396 and Recall from 0.6824 to 0.8706, while Full reduces DT from 0.1769 to 0.0021, demonstrating stronger detection accuracy and temporal responsiveness.

To further explain these improvements, we conducted two additional comparisons.

First, online VLM detection vs. offline multi-scale annotation. On the same 85 failed trajectories, offline annotation improves Recall from 0.6824 to 0.8235 and reduces DT from 0.1769 to 0.0693. This demonstrates the advantages of offline multi-scale analysis in producing effective failure supervision.

Second, offline VLM labels vs. DART. On all 206 failed trajectories from π0\pi_0-unseen, directly thresholding offline VLM labels achieves Recall of 0.8204 and DT of 0.0656, whereas DART Short achieves 0.8835 and 0.0539, and Full achieves 0.8495 and 0.0131, respectively. This indicates that DART can learn effective failure representations from automatically generated supervision and further improve detection performance.

Together, these experiments reveal a clear progression: direct online VLM detection → offline multi-scale VLM annotation → DART trained with automated supervision. Rather than deploying the VLM directly for runtime detection, our approach leverages its reasoning capability offline to generate fine-grained supervision, allowing dedicated detection modules to achieve stronger accuracy and temporal responsiveness.

We expect this principle to extend to more powerful general-purpose VLMs, such as GPT-6 and Gemini-3.1-Pro. Specifically, stronger VLMs may produce higher-quality offline annotations, enabling DART to learn more informative failure representations. Therefore, we hypothesize that training DART with annotations generated by these stronger models could outperform directly deploying the same VLMs for online failure detection, following the same progression observed with Qwen3-VL-8B-Instruct. While this expectation remains to be empirically validated, it highlights the broader applicability of our offline-supervision and online-detection framework.

Computational efficiency. Direct online Qwen3-VL-8B-Instruct detection requires an average of 4.39 s per inference, compared with approximately 13 ms for DART Short and 1.197 s for DART Long, which runs asynchronously without blocking robot execution (Section 4.7). This further demonstrates the computational advantages of using VLMs for offline supervision rather than direct online detection.

AuthorsOpenReview ↗

Response to Reviewer 5Mbf: Generalization Across VLA Models, Tasks, and Environments

Generalization Across VLA Models, Tasks, and Environments

We thank the reviewer for raising this question. DART is designed as an adaptable failure detection framework rather than a detector restricted to a specific VLA model or environment.

First, adaptation to different VLA architectures. The Short-Horizon module requires identifying suitable failure-discriminative internal features from the target VLA. Our experiments on π0\pi_0 and π0.5\pi_{0.5} demonstrate the effectiveness of extracting Vision-Language (VL) states and action-expert hidden states. Our additional feature ablation on π0\pi_0-unseen further supports this:

FeaturesRecallPrecisionBal. Acc.DTAT
Hidden-only0.84470.77330.82390.08870.1946
VL-only0.90290.79150.85690.06260.2114
VL+Hidden (Short)0.88350.79480.85100.05270.2001

VL-only achieves the highest Recall (0.9029) and Balanced Accuracy (0.8569), only slightly underperforming VL+Hidden in temporal responsiveness (DT: 0.0626 vs. 0.0527). This demonstrates that readily accessible VL backbone features alone provide strong failure-discriminative information. Combining VL and hidden features achieves the highest Precision (0.7948) and lowest DT (0.0527), further demonstrating their complementarity.

Moreover, prior work such as SAFE has explored informative internal representations across different VLA architectures, supporting the feasibility of identifying suitable features beyond our evaluated policies. Therefore, adapting Short to another VLA requires selecting architecture-appropriate features rather than redesigning the entire framework.

Second, adaptation to new environments. The Long-Horizon module requires environment-specific fine-tuning of the Wan video world model. We have already established a reusable fine-tuning pipeline using execution videos and corresponding task instructions. Thus, adapting to a new environment primarily involves collecting a moderate amount of video-text data and fine-tuning the existing model, rather than training a world model from scratch.

Finally, adaptation of the overall framework. We acknowledge that deployment on a new VLA or environment generally requires suitable feature extraction, world-model fine-tuning, retraining the detection networks, and recalibrating the detection threshold. Nevertheless, the complete pipeline remains reusable, including automated timestep-level annotation, Short-Horizon feature-based detection, Long-Horizon future-state prediction, horizon-aligned supervision, and dual-horizon fusion.

Our evaluation across four LIBERO suites, two VLA policies (π0\pi_0 and π0.5\pi_{0.5}), and four real-world manipulation tasks supports the framework's applicability across different tasks and settings. While DART does not claim zero-shot transfer to arbitrary VLA architectures or environments, its modular design enables adaptation without changing the core methodology.

AuthorsOpenReview ↗

Response to Reviewer 5Mbf: World Model Conditioning, Practical Utility, and Runtime Efficiency

World Model Conditioning, Practical Utility, and Runtime Efficiency

1. World Model Conditioning. We thank the reviewer for pointing out this ambiguity. DART employs Wan, a powerful video-generation world model, fine-tuned on execution videos from the target manipulation environment to predict future visual dynamics. Unlike conventional action-conditioned world models, Wan does not require explicit future robot action sequences in our implementation. Instead, it takes the current visual observation, task instruction, and normalized execution progress as conditioning inputs. The predicted future latent representations are temporally aggregated and used to compute the Long-Horizon failure score, without video decoding during inference. We will clarify this mechanism in the revised manuscript.

2. Practical Utility of Binary Failure Detection. DART is designed for runtime failure monitoring, rather than failure explanation or recovery planning. Its primary purpose is to provide timely risk warnings that can trigger downstream safety mechanisms, such as emergency stopping, human intervention, or switching to an independent recovery policy.

Importantly, a binary warning can be useful without identifying the precise failure cause, provided that it is sufficiently accurate, timely, and computationally feasible. We therefore evaluate detection reliability using Recall, Precision, and Balanced Accuracy, and temporal responsiveness using DT, AT, and alarm timing distributions. In particular, the Long-Horizon module enables proactive warnings before failures become explicitly observable, potentially providing additional time for intervention.

We acknowledge that early detection does not necessarily guarantee successful intervention. The effectiveness of stopping or recovery also depends on the available intervention window and the downstream controller. Evaluating closed-loop intervention and recovery remains an important direction for future work.

3. Runtime Latency and Computational Overhead. As reported in Section 4.7, Short-Horizon scoring adds approximately 13 ms of overhead, while Long-Horizon inference takes approximately 1.197 s on an H100 GPU. The Long-Horizon module runs asynchronously alongside robot execution, without blocking the VLA control loop. Thus, the full framework does not require synchronous world-model inference at every control step. Short provides rapid current-state monitoring, while Long supplies predictive risk information once its asynchronous inference completes.

These results demonstrate the feasibility of integrating both modules into a runtime monitoring framework while preserving the normal execution of the VLA policy.

Reviewer_aJrgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Background: The lack of reliable runtime safety mechanisms remains a major obstacle to deploying vision-language-action (VLA) models in real-world environments. Existing methods often rely on coarse trajectory-level supervision or post hoc failure scoring, limiting their ability to promptly detect subtle deviations in the early stages of execution or sudden failures.

Proposed Method: The paper introduces DART, a proactive runtime failure detection framework for VLA policies. DART employs a dual-horizon architecture to detect both abrupt failures and deviations that accumulate over extended periods. It also introduces an automated VLM-based annotation pipeline that generates timestep-level failure labels from execution trajectories.

Validation: DART achieves a favorable trade-off between failure detection accuracy and temporal responsiveness, demonstrating particularly strong performance on tasks unseen during training.

Critical strengths
  1. The paper evaluates the proposed method on real-world robotic manipulation tasks, providing evidence of its practical applicability.
  2. The paper combines failure signals from two complementary models and empirically investigates the benefits of this combination.
Critical weaknesses
  1. Presentation and Writing Quality

The paper has several presentation and formatting issues that detract from its professionalism and make it appear below the standard expected at a top-tier venue.

The caption of Figure 1 is excessively long.

A citation is awkwardly split across a page break at the beginning of page 3.

Figures 4, 5, and 9 use excessively small font sizes, making them difficult to read and, in some cases, occupying space inefficiently.

The abstract ends with a single word on the final line, suggesting suboptimal typesetting. Table 2 also wastes considerable horizontal space.

Figure 3 is awkwardly positioned in the middle of the main text.

  1. Limited Novelty and Insufficient Differentiation from Prior Work

First, the paper's contributions should be more clearly distinguished from those of Focus on the Moment: Real-Time Semantic Failure Detection in Vision-Language-Action Models. (https://openreview.net/forum?id=fWN2jSqJkW)

Throughout the abstract and related work sections, the authors emphasize that existing methods operate at a single temporal scale, positioning their approach as a novel solution that operates across multiple temporal scales. However, Focus on the Moment has already explored failure detection across different temporal scales and proposed an approach similar in spirit to the automated VLM-based annotation pipeline presented in this paper.

Consequently, the novelty of these claims is unclear. In particular, two of the three contributions outlined in the introduction appear to overlap substantially with those claimed in Focus on the Moment. The authors should explicitly clarify the differences between the two approaches and identify what is genuinely novel in the present work.

Second, the short-horizon module appears to share key structural ideas with SAFE (Gu et al., 2025b). It is unclear how the proposed module differs from SAFE, or whether the primary contribution should instead be understood as integrating a short-horizon detector with a long-horizon world-model-based detector. The authors should clarify the methodological differences from SAFE and explain which aspects of the proposed approach constitute novel contributions.

Questions for authors
  1. Novelty relative to prior work

As discussed in Weakness 2, could the authors clarify the contributions of this work relative to Focus on the Moment and SAFE? My current understanding is that the main distinguishing contribution is the integration of two failure signals—one derived from the VLA model's internal representations and the other from a world model's predictions—and the empirical demonstration that these signals provide complementary information. If this is indeed the primary contribution, the authors should articulate it more explicitly and explain how it advances beyond the existing approaches.

  1. Direct VLM-based runtime detection

If a VLM can generate timestep-level failure labels from execution videos, why not use the VLM directly for runtime failure detection instead of training separate short- and long-horizon detection modules? What advantages does the proposed approach offer in terms of detection accuracy, temporal responsiveness, and computational efficiency?

5 comments
AuthorsOpenReview ↗

Response to Reviewer aJrg: Novelty and Distinctions from Focus on the Moment and SAFE

We thank the reviewer for the suggestion. We will address all formatting and presentation issues in the revised manuscript. We agree that DART's main contribution is not simply multi-timescale failure detection, but the integration of current-state perception from VLA internal representations and future-risk prediction from a world model, supported by horizon-aligned supervision.

Comparison with Focus on the Moment. We acknowledge that Focus on the Moment has explored real-time failure detection and fine-grained failure supervision, sharing certain research objectives with DART.

Although both SemGen and our annotation pipeline aim to reduce the cost of fine-grained supervision, their mechanisms differ. SemGen constructs semantic failure examples and identifies failure frames using gripper signals, whereas DART employs offline multi-scale VLM analysis, combining predictions from video segments of different durations to generate continuous timestep-level risk scores from execution trajectories.

More importantly, their detection mechanisms differ. FailMoment primarily relies on current and recent historical information for real-time semantic failure detection. In contrast, DART explicitly combines two complementary sources of information: Short-Horizon uses internal vision-language and control representations from the VLA to perceive the current execution state, while Long-Horizon uses the Wan world model to predict future states and anticipate potential failure risks.

Furthermore, DART employs horizon-aligned supervision: Short-Horizon learns the current timestep label YtY_t, while Long-Horizon learns the future label Yt+nY_{t+n}. Table 3 in the original submission demonstrates the effectiveness of this design. Therefore, our dual-horizon framework is not merely based on historical features of different lengths, but combines current-state failure detection and future-risk anticipation through distinct representations and supervision objectives.

Comparison with SAFE. We acknowledge that SAFE already leverages VLA internal hidden control states for failure detection. DART Short further incorporates Vision-Language (VL) states to combine semantic and control information.

To clarify their respective contributions, we conducted additional feature ablations on π0\pi_0-unseen:

FeaturesRecallPrecisionBal. Acc.DTAT
Hidden-only0.84470.77330.82390.08870.1946
VL-only0.90290.79150.85690.06260.2114
VL+Hidden (Short)0.88350.79480.85100.05270.2001

VL-only achieves stronger detection performance than Hidden-only. Combining both representations further reduces DT to 0.0527 and achieves the highest Precision, demonstrating their complementarity in detection performance and temporal responsiveness.

We emphasize that the Short-Horizon feature design is not DART's only contribution. More importantly, DART integrates Short-Horizon detection with Long-Horizon future prediction through complementary supervision objectives. Tables 1–2 and Fig. 4 already demonstrate their complementary advantages. In particular, Full achieves a 91.45% detection rate on cumulative failures, outperforming both individual modules.

We will revise the Introduction and Related Work to clarify these distinctions and highlight DART's three main contributions: dual-horizon failure detection combining VLA internal representations and world-model predictions, horizon-aligned supervision, and multi-scale VLM annotation for cost-effective fine-grained risk supervision.

AuthorsOpenReview ↗

Response to Reviewer aJrg: Direct VLM Detection and the Effectiveness of Automated Supervision

To address why we do not directly use a VLM for online failure detection, we conducted additional experiments with Qwen3-VL-8B-Instruct, comparing direct online detection, offline multi-scale annotation, and DART trained using the generated supervision.

1. Direct Online VLM Detection vs. Offline Multi-Scale Annotation

Due to limited time and computational resources, we randomly sampled 200 trajectories from π0\pi_0-unseen using seed=42, including 85 failed trajectories. Both methods were evaluated on the same 85 failed trajectories.

Online detection simulates a robot observing its execution in real time. The VLM is queried every 2 seconds, with an additional query at the final frame if necessary. Each query contains only the execution history available up to that moment and the task instruction, without future frames or final success/failure information. The model classifies execution as normal (category 1) or abnormal (categories 2–5). Once an abnormality is detected, the alarm remains active.

In contrast, our offline annotation pipeline analyzes video segments at five temporal scales (2, 2.5, 3, 3.5, and 4 seconds), aggregates their predictions, and applies temporal smoothing. The resulting anomaly scores are thresholded at 0.5 for direct failure detection.

MethodDetected/FailedRecallDTAT
Online VLM detection58/850.68240.17690.3186
Offline multi-scale annotation70/850.82350.06930.2832

The offline multi-scale annotation pipeline achieves higher trajectory-level Recall (0.8235 vs. 0.6824) and lower DT (0.0693 vs. 0.1769). We attribute this improvement to the overall offline annotation procedure, which integrates observations across multiple temporal scales to produce more stable and continuous risk estimates.

Importantly, offline annotation is not intended to serve as a deployable online detector, since it can access complete execution videos. Instead, it provides effective fine-grained supervision for training DART's online detection modules. We next examine whether DART can further improve upon the detection performance of these offline-generated labels.

2. Offline VLM Labels vs. DART Detection

We further compared direct failure detection using offline VLM-generated labels against the three DART configurations trained with these labels. This evaluation uses all 206 failed trajectories from the π0\pi_0-unseen set.

MethodDetected/FailedRecallDTAT
Offline VLM labels (0.5)169/2060.82040.06560.3002
DART Short182/2060.88350.05390.1977
DART Long152/2060.73790.00020.3783
DART Full175/2060.84950.01310.2839

Both DART Short and Full improve trajectory-level Recall and temporal responsiveness over direct detection using offline VLM labels. Short increases Recall from 0.8204 to 0.8835 while reducing DT from 0.0656 to 0.0539. Full achieves Recall of 0.8495 and further reduces DT to 0.0131. Although Long has lower Recall, its DT is nearly zero, indicating stronger early-warning capability through world-model-based future prediction. Its larger AT may reflect earlier alarms rather than delayed detection.

These results suggest that, despite discrepancies between VLM-generated and human annotations, DART can learn meaningful failure-discriminative information from automated supervision and achieve better detection performance than directly thresholding the original VLM labels. Together with the first comparison, these findings support our approach of using the VLM for offline supervision rather than relying on direct VLM inference for runtime detection.

Computational Efficiency. Direct online Qwen3-VL-8B-Instruct detection requires an average of 4.39 s per inference, compared with approximately 13 ms for DART Short and 1.197 s for DART Long, which runs asynchronously without blocking robot execution (Section 4.7). Overall, these comparisons demonstrate the advantages of DART's offline-supervision and online-detection design in detection accuracy, temporal responsiveness, and computational efficiency.

Evaluation scope. These two new comparisons include only failed trajectories because successful trajectories receive normal labels by default in the offline annotation pipeline. Consequently, Precision and Balanced Accuracy cannot be meaningfully evaluated in this setting. We therefore focus on Recall, DT, and AT. The online and offline VLM methods also differ in temporal access and inference frequency, so this comparison evaluates their overall detection procedures rather than isolating the effect of multi-scale aggregation alone.

Thank you for the detailed response and the additional experiments. Some of my concerns remain.

  1. Novelty vs. Focus on the Moment. The distinction is argued only descriptively. There is no direct comparison on the same benchmark and metrics, so it is unclear which approach performs better. Please include it as a baseline, or explain why this is not feasible.
  2. Novelty vs. SAFE. The new ablation weakens the complementarity claim. VL-only has higher Recall (0.9029 vs. 0.8835) and Bal. Acc. (0.8569 vs. 0.8510) than VL+Hidden, and the combined gains (DT, Precision +0.003) are small and reported without seeds or variance. Please also clarify whether Hidden-only faithfully reproduces SAFE (architecture, calibration).
  3. Direct VLM detection. The relevant comparison is online VLM vs. DART on the same trajectories, but the two new comparisons use different subsets (85 vs. 206 failed trajectories). Offline annotation, which sees future frames, is not a runtime alternative, so the first comparison does not answer my question.
Reviewer_dcDoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a method for proactive runtime failure detection for vision-language-action models. The paper proposes a dual-horizon approach to cover both sudden failures (via vision-language and control representations) and longer-term compounding failures (through mean-pooled features from world modeling). An automated annotation pipeline is also proposed to generate dense supervision in a low-cost manner. Failure detection results are shown on LIBERO and through real-world robot experiments.

Critical strengths
  • Failure detection is an important topic for robot safety and recovery.

  • The paper is overall well-written and contains sufficient description of the method.

  • The paper considers compute/run-time which is important for robotics.

Critical weaknesses
  • The paper essentially disentangles two aspects (sudden failures and long-term failures), proposes new methods for both along with new supervision, and then combines them. However, this makes it difficult to understand the contribution. For example, is the method for sudden failure detection (which is pretty simple in terms of features it uses) actually better than prior methods? Or does the new type of supervision/data skew results in favor of the proposed method? Since three things are combined, without ablating state of art within each, it's difficult to tell.

  • Related, this paper proposes automatic dense annotation but has very little information about it (even in the appendix). Even Figure 12 in the appendix shows some biases, and their propagation/correlation to the reported metrics are not analyzed. For example, it could be that the bias disadvantages prior methods that were used as baselines.

  • The overall novelty is very low (the paper uses very simple models on typical features) but the empirical contribution is not strong either, given that LIBERO (which is an easy/saturated benchmark) is used along with real robot experiments (which are hard to reproduce). The method should be demonstrated on significantly more complex benchmarks such as RoboCasa 360, etc.

  • The method is only evaluated with respect to failure detection. The usefulness of such a signal for anything (stopping, intervention, etc.) is unclear given that they may have specific characteristics to be useful.

Questions for authors
  1. What is the novelty and contribution of the method with respect to prior work, and are all proposed components necessary or can prior methods be used as subcomponents (e.g. for sudden failure detection) just as well as the proposed one?

  2. What are the details of the annotation pipeline and do the demonstrated biases affect the conclusions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer dcDo: Novelty and Component Ablations

We thank the reviewer for the constructive feedback. Our response to the raised concerns is as follows:

1. Novelty and Component Ablations

We have already reported the individual ablation results of the Short-Horizon and Long-Horizon modules in Table 1. Existing baseline methods primarily focus on failure detection based on historical or current information, whereas Table 1 and Fig. 4 show that our Short-Horizon module outperforms the baselines in both trajectory-level detection accuracy and temporal responsiveness. Furthermore, the Long-Horizon module utilizes a world model for future-state prediction, demonstrating stronger early-warning capability than conventional failure detection methods. As shown in Table 1, Fig. 4, and our real-world experiments, combining the Short-Horizon and Long-Horizon modules maximizes their complementary advantages.

Additionally, we conducted a feature ablation study for the Short-Horizon module (evaluated on π0\pi_0-unseen), comparing the use of action-expert hidden features alone (Hidden-only), vision-language features alone (VL-only), and their combination (VL+Hidden). The results show that the vision-language (VL) features introduced by DART provide stronger detection performance than the hidden features used alone in SAFE. Combining these two feature types further reduces detection delay (DT) to 0.0527 and achieves the highest Precision, demonstrating their complementarity in detection performance and temporal responsiveness.

FeaturesRecallPrecisionBal. Acc.DTAT
Hidden-only0.84470.77330.82390.08870.1946
VL-only0.90290.79150.85690.06260.2114
VL+Hidden (Short)0.88350.79480.85100.05270.2001

We emphasize that the main contribution of this work lies not only in proposing a new lightweight classifier, but more importantly in integrating multidimensional VLA (Vision-Language-Action) state perception, world-model-based future-state prediction, and horizon-specific supervision mechanisms to achieve complementary failure detection across two temporal horizons.

AuthorsOpenReview ↗

Response to Reviewer dcDo: Annotation Pipeline Details and Supervision Bias Analysis

Automated Annotation Pipeline and Potential Temporal Bias

First, regarding the automated annotation pipeline, the overall procedure and prompt design are described in Section 3.4, Fig. 3, and Appendix A.7. We provide the key implementation details below.

We use Qwen3-VL-8B-Instruct (FP16) to analyze failed trajectories at five temporal scales (2, 2.5, 3, 3.5, and 4 seconds). Each segment is classified into five categories: normal execution, object overturning, gripper stagnation, semantic errors, and grasping failures. Normal execution is assigned label 1, while all failure categories are assigned label 0. Successful trajectories are directly assigned normal labels.

Predictions are aggregated across five scales at 0.5-second intervals and transformed into continuous anomaly scores using a sigmoid function (scale parameter α=10\alpha=10, midpoint 0.5). The scores are then expanded to individual frames and smoothed with a Gaussian filter (σ=max⁡(1,⌊0.1N⌋)\sigma=\max(1,\lfloor 0.1N\rfloor), kernel size 6σ+16\sigma+1, edge padding), where NN is the total number of frames.

Second, regarding potential temporal bias, VLM-generated labels are used exclusively for training, while evaluation relies on independently human-annotated failure manifestation times (Appendix A.3).

To investigate whether VLM-generated temporal biases unfairly favor DART or disadvantage baseline methods, we retrained Fail-Detect, Short, Long, and Full using human-annotated timestep-level labels and evaluated them on the same π0\pi_0-unseen benchmark.

LabelsMethodRecall ↑Precision ↑Bal. Acc. ↑DT ↓AT
VLMFail-Detect0.93200.73280.83090.05300.2251
VLMShort0.88350.79480.85100.05270.2001
VLMLong0.73790.84920.81680.00020.3832
VLMFull0.84950.80650.84370.01250.2876
HumanFail-Detect0.78640.82230.82560.09270.1521
HumanShort0.78640.91010.86230.13360.1585
HumanLong0.81070.83080.83970.00400.3832
HumanFull0.81070.84770.84740.04170.2324

Under both supervision settings, Short consistently outperforms Fail-Detect in overall trajectory-level detection accuracy and temporal responsiveness, while DART (Full) further improves upon Short in overall detection performance and early-warning capability. This demonstrates the complementary benefits of the two horizons.

These consistent advantages under both VLM-generated and human-annotated supervision suggest that DART's improvements are not primarily driven by temporal biases in VLM-generated labels, but also stem from its detection architecture and dual-horizon design.

Furthermore, VLM-supervised models achieve performance comparable to those trained with human annotations, suggesting that the detectors can learn reliable failure patterns despite imperfections in automatically generated labels, rather than simply reproducing their temporal biases. Since VLM annotation also eliminates the need for labor-intensive manual timestep labeling, it offers an effective and scalable alternative to human supervision.

AuthorsOpenReview ↗

Response to Reviewer dcDo: Benchmark Generalization and Practical Utility of Failure Detection

Benchmark Generalization and Practical Utility of Failure Detection

We thank the reviewer for these valuable comments. We address the concerns regarding benchmark complexity, generalization, and practical utility below.

1. Benchmark Complexity and Generalization. Our evaluation covers four LIBERO suites (LIBERO-10, LIBERO-90, Goal, and Spatial), two VLA policies (π0\pi_0 and π0.5\pi_{0.5}), and both seen and unseen tasks. These settings involve diverse manipulation objectives, spatial configurations, and task variations, providing evidence of DART's effectiveness across different tasks and policies.

Additionally, we evaluate DART on four multi-stage manipulation tasks using a physical Franka robot, further demonstrating its applicability beyond simulation and under real-world execution conditions. Together, these experiments provide complementary evidence of task-level generalization and practical applicability.

We agree that evaluation on more challenging and reproducible benchmarks such as RoboCasa would further strengthen the evidence for broader generalization. While our current experiments cover multiple benchmark suites, VLA policies, unseen tasks, and real-world manipulation scenarios, we acknowledge that generalization to more complex and open-world environments warrants further investigation.

2. Practical Utility of Failure Detection. DART is designed for runtime failure monitoring, rather than directly performing intervention or recovery. Its primary purpose is to provide timely risk warnings that can trigger downstream safety mechanisms, such as emergency stopping, human intervention, or switching to an independent recovery policy. Detection and intervention are two related but distinct research problems.

We evaluate detection reliability through Recall, Precision, and Balanced Accuracy, and temporal responsiveness through DT, AT, and alarm timing distributions. Section 4.7 further demonstrates computational feasibility: Short-Horizon scoring adds approximately 13 ms of overhead, while Long-Horizon inference takes approximately 1.197 s on an H100 GPU and runs asynchronously without blocking the VLA control loop.

These results support DART's potential as a practical warning component for downstream safety mechanisms. Nevertheless, we acknowledge that early detection does not guarantee successful intervention, which also depends on the available response window and downstream controller. Evaluating closed-loop intervention and recovery remains an important direction for future work.

Thank you for the rebuttal and additional results. The rebuttal partially addresses some of my concerns e.g. the effects of the bias. Note that I already saw the details in the section 3.4/appendix, but my point was that this was not sufficiently detailed for a complex annotation process given its importance.

In terms of the novelty and component ablation, again yes I saw the ablation. My point was that this paper combines these different types of failures but does not independently test (e.g. for sudden failures) against the large body of such methods; in other words, for the sudden failure part why couldn't you have just used prior works (as mentioned by the other reviewers, there are also ones that go beyond binary classification)? The combination of the modules itself is not very complex/interwoven (via weighted fusion) so just because you combine these two aspect doesn't mean you don't have to compare to the state of the art for each part (otherwise there isn't a justification for the individual module compared to just using what exists already).

In terms of the environments tested, I realize there are multiple LIBERO suites but they are saturated and not sufficient to demonstrate generality of results. Further, real-robot results are great to show that the method works but are typically hard to reproduce.

In addition, some of the other reviewer's points either align with my comments or provide other ones that I agree with (environment-specific tuning, complexity, etc.) As a result, unfortunately I am left even more negative on the paper as-is than before, and recommend thorough revisions for a future venue.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank the reviewers for their constructive feedback. We have uploaded a revised manuscript incorporating the following updates:

  1. Clarified novelty and contributions. We refined DART's contributions, emphasizing VLA-internal failure detection, world-model-based future-risk prediction, and horizon-aligned supervision. We also expanded the related work discussion to distinguish DART from FailMoment and SAFE.

  2. Added direct online VLM comparisons (Section 4.6). We evaluated online VLM detection, offline multi-scale annotation, and DART on the same complete set of 465 trajectories, including 206 failures. The results demonstrate the advantages of DART in detection accuracy and temporal responsiveness.

  3. Expanded ablation studies (Appendix). We added three-seed feature ablations comparing VL-only, Hidden-only, and VL+Hidden, as well as controlled experiments comparing VLM-generated and human-annotated timestep-level supervision. The comparable performance under both supervision settings suggests that temporal biases in VLM-generated labels do not substantially affect DART's overall effectiveness.

  4. Improved methodological clarity and limitations. We clarified implementation and evaluation details, including the distinction between SAFE and our Hidden-only variant, and expanded the discussion of benchmark generalization and practical intervention limitations.

We hope these revisions address the reviewers' concerns and further clarify the contributions and empirical evidence supporting DART. We appreciate the reviewers' time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.