acceptodds
Withdrawn from ICLR 2027

DoOver: Detect Failure from Demos, Do It Over On Demand

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Current robot policies often continue acting obliviously after execution goes wrong. We introduce DoOver, which couples lightweight, policy-native failure detection with on-demand recovery mechanism. With a M-parameter head that learns jointly with the policy from timestamps in expert demonstrations and reuses policy's visual features, DoOver tracks changes in predicted progress to detect stalls and regressions. It requires no failure data, pretrained foundation models, separate detector training, or runtime action resampling. Across simulated tasks with flow-matching and diffusion backbones and detection baselines per backbone, DoOver leads the eligible baselines in aggregate average precision across task-calibrated detection, alarm-threshold sweeps, and threshold generalization, with lower failure-detection compute than the sampling-based detectors STAC and ACE. Beyond detection, we establish a systematic benchmark of failure-triggered recovery for frozen policies, comparing seven operators spanning five inference-time intervention families. Among them, failure-triggered feedback world model improves recovery rates over plain replanning by / pp and over normal execution by / pp for flow-matching/diffusion policies, while avoiding the high cost of guided sampling during normal execution.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Robot Manipulation, Failure Detection, Failure Monitoring, Robot Learning, Failure Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GrGmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents an intuitive and practically motivated framework that combines lightweight runtime failure monitoring with on-demand recovery. Learning a progress signal from existing demonstration timestamps while reusing policy features is appealing. The extensive detection evaluation across two policy families, recovery comparisons, and real-robot demonstrations are strengths.

Critical strengths

The paper has an intuitive and practically relevant motivation. Runtime failure monitoring should be connected to recovery so that detecting a problem leads to corrective action. Reusing policy features for lightweight monitoring and activating additional computation when needed is an appealing design.

Critical weaknesses

1. The writing and organization could be improved.

The manuscript places considerable emphasis on numerical results and comparisons, while the core method and its design rationale receive less clear explanation. A more coherent presentation of the motivation, proposed mechanism, and relationship between components would make the paper easier to follow and its contribution easier to assess. For example,

  1. The training objectives need a more precise mathematical specification. Section 3.2 describes LpolicyL_{\text{policy}} as an action-prediction loss and LprogressL_{\text{progress}} as a mean squared error, but does not explicitly formulate these objectives.

  2. Section 3.4 mixes the proposed method with the experimental comparison. Much of this section enumerates existing recovery operators and their evaluation settings, which would fit better in the experimental setup. The Methods section should focus on the proposed alarm-to-recovery controller and clearly distinguish the authors’ contributions from reused components.

2. Several consequential hyperparameter choices lack sufficient justification or sensitivity analysis.

For example, fixed-threshold PS uses a window of two policy decisions and raises an alarm when predicted progress gain falls below 0.001. The rule is defined, but the basis for choosing these values and their robustness across tasks and execution speeds remain unclear.

3. The methodological novelty of the recovery component appears limited.

The recovery operators largely build on existing methods, particularly FWM, so the contribution appears to lie primarily in progress-triggered activation and integration with the policy. In my view, this is an incremental extension of prior work on progress-aware heads [1,2].

[1] Ma et al., "CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decoding," arXiv.2601.02295

[2] Physical Intelligence, "π0.6 : a VLA That Learns From Experience," arXiv:2511.14759

Questions for authors

I am very interested in failure detection and recovery, and I appreciate the paper’s practical motivation. However, the manuscript requires substantial improvements in its presentation, justification of methodological choices, and demonstration of novelty, which currently fall below the ICLR bar. Due to these concerns, I am assigning a score of 1. I encourage the authors to address the weaknesses detailed above, particularly with respect to the writing and overall presentation.

Reviewer_ie7EOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies runtime failure detection and recovery for generative visuomotor policies (flow matching and diffusion). It attaches a small progress head (0.526M parameters) to the policy, trained jointly with it on the policy’s visual features to regress each demonstration frame’s normalized timestamp. Four detectors turn predicted progress into alarms. Two flag stalled or decreasing progress over one or several time windows (PS, MS-PS), and two flag progress falling short of the head’s own forecast (FE, MS-FE). None requires failure data, an external foundation model, or action resampling. Across 40 LIBERO tasks and 13 baselines per backbone, the detectors lead the baselines that do not use failure data in aggregate AP, at far lower compute than sampling-based detectors. The paper also benchmarks seven recovery strategies that activate only after a PS alarm, with the policy frozen. The strongest is FWM, an existing world-model guidance method that the authors extend to flow matching and enable only on demand. It improves the recovery rate over plain replanning by 6.8/5.1 pp (FM/DDPM). Two real-robot tasks complement the simulation results.

Critical strengths

The empirical evaluation is thorough. The benchmark covers 40 LIBERO tasks across four suites, flow-matching and diffusion policies, 13 baselines per backbone, and three seeds. All detectors score the same recorded rollouts, using disjoint fitting, normalization, calibration, and test splits. Every detector observes the same per-task horizon, which removes episode length as a cue that would otherwise reveal the outcome. Failure-trained baselines are reported separately rather than omitted. Beyond task-calibrated detection, the authors test alarm-threshold sweeps, threshold transfer across tasks and suites, and cross-policy monitoring. In each setting the detectors lead the baselines that use no failure data. They roughly match the failure-trained baselines. The authors also report parameters, storage, fitting time, and per-decision latency for every method.

Combining a light weight failure detection mechanism with targeted recovery strategies presents an insightful direction for the field. Leveraging shared representations from the base policy allows the progress module to track performance with minimal computational overhead, ensuring that computationally heavy intervention steps are triggered only upon detecting a failure.

Critical weaknesses

Weakness 1: The detector has a narrow, implicit notion of failure, inherited from defining progress as normalized time on successful demonstrations. The head is trained only on expert demonstrations with labels t/(T−1). All four detectors fire only when predicted progress advances less than expected. A failure is detected only if the head’s guess on an unseen failure state happens to look like a stall or regression. Failures where the estimate keeps rising, or is already high, are undetectable by design: pursuing the wrong goal, manipulating the wrong object, or stopping just short of the goal.

For example, the head appears not to see the task instruction. Its input is image features, while the policy’s input also includes a language instruction. On LIBERO-Goal, where all tasks share one scene, it is unclear how the head distinguishes progress on the commanded task from progress on another.

The numbers in Table 2 and 3 in the appendix suggest that the proposed method never triggers on half or more failures. To confirm, it would be useful if the authors directly report TPR (the percentage of correctly detected failures). While the reported baselines do not seem to do better, this paper’s central claim is that failures appear as stalled or falling progress. The roughly half of failures that never trigger an alarm are the most direct test of that claim, since they may be failures that do not look like stagnation at all. Currently, the paper does not report which kinds of failures are caught and which are missed.

The timestamp labels also assume demonstrations progress at a similar pace. Diverse teleoperation data violates this: operators differ in speed and strategy, so the same visual state can receive very different labels. MS-PS would be affected most, since it normalizes scores with per-decision statistics of successful calibration rollouts and so assumes they are aligned in time. The paper labels its main results in-distribution, and no shifts in viewpoint, initial state, or object configuration are tested, so it is unclear how well the signal generalizes.

Weakness 2: Limited novelty, and the progress estimator is never compared against alternatives. Learning progress from the temporal order of demonstrations is an established methodology. TimeRewarder [1] learns progress from temporal distances between frames. The paper itself cites RISE, which fits normalized temporal progress, and the Self-Monitoring agent (Ma et al., 2019a), which trains an auxiliary progress head jointly with its policy. The new contribution is the combination: a jointly trained auxiliary head in manipulation, used as a failure trigger.

That makes the progress estimator the key component, yet no alternative estimator is tested. Robometer [2] is discussed but excluded, as is Sentinel’s VLM progress component (only STAC is evaluated). Inference cost doesn’t justify this. All detectors score identical recorded rollouts, so Robometer’s per-frame progress could be computed offline on the same videos and passed through the same PS/MS-PS rules. Keeping the PS/MS-PS rules fixed and swapping only the progress estimator would show whether DoOver’s gains come from the policy-native head or from the stagnation rule itself, which could work with any reasonable progress signal. It would also show what policy-native features buy compared with a large general model trained with failed trajectories, which relates to Weakness 1.

Questions for authors

Q1: Which failures are missed? Could the authors categorize failed test rollouts (missed grasp, dropped object, wrong object or goal, failure near the goal, timeout while still progressing) and report detection rates per category?

Q2: Does the progress head see the language instruction? If not, how does it handle LIBERO-Goal’s shared scene?

Q3: Could the authors feed Robometer’s progress (or another off-the-shelf progress model’s) through PS/MS-PS on the same recorded rollouts? If the argument against it is cost, please show the accuracy–cost trade-off explicitly.

Q5: How would the method perform in contact-rich tasks that require physical exploration, where progress may not be visible in the image?

Q6: The interplay between progress predictions, action chunks, and decision steps is hard to follow. A more detailed version of Fig. 1(a,b) would help.

Reviewer_YgN7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes the idea of using a progress model for failure monitoring and using this failure monitoring to do adaptive planning. The main idea is to take the visual features of the policy and train a separate progress score head on it. In the paper’s implementation the progress score is a windowed prediction over a time horizon. Once this progress model is learned it is used for policy monitoring. The simplest policy monitoring is to look at the progress scores within a given window and if the policy does not show sufficient progress within this window, failure detection or stalling gets triggered. There is another multi-scale approach with a similar idea. Once the policy stalls, the paper proposes to spend more compute budget to get a better action output from the policy. Multiple different methods such as best-of-n, using an external model etc. are proposed for this. Experiments are performed in Libero, which shows the advantage of the proposed approach. Real-world experiments are also performed.

Critical strengths

The paper is mostly well written and focuses on an important problem. Policy monitoring and understanding failure modes to generate better actions is an important problem especially in the context of deployment.

Critical weaknesses

I am a bit skeptical about the main argument the paper is making against using external vision models for progress monitoring. The fundamental problem with positive demonstrations only training of progress models is that the models will never see what failure looks like. This is a very significant problem which is almost impossible to alleviate given that the learning data is insufficient. Specifically, such positive demo only progress models exhibit strong semantic or task-identity blindness. This is because given any state they have always seen progress and since there is no explicit contrastive signal teaching the head that "picking up Y when asked to pick up X" should yield 00 progress is impossible.

The paper talks about such a failure mode and they strip away proprioceptive information from the progress head. However, having trained exactly similar models, this approach is insufficient in practice. It may work for toy/simple benchmarks (as in this paper) but fails for real-world tasks of moderate complexity.

Also, looking at Section 3.2 line 162, it is not clear if any of the language conditioning is being used in the progress model. This would be odd since then the policy head has no way of knowing what task is being done.

But this is the exact failure mode that is avoided by using large external models. Since these models have good visual grounding we can use them directly off the shelf for monitoring failure. Unfortunately, there aren’t that many comparisons with such approaches in complex real-world settings, such as having multiple objects in a cluttered scene and the policy being steered.

To address the semantic and task-identity blindness without requiring actual failure rollouts or heavy external models at runtime, the authors could condition the progress head on the task's language embedding (which is already computed in the policy context cc) and use negative relabeling during training. Specifically, simply concatenating the language embedding to the visual features hkh_k and training with MSE on positive demos is insufficient, as the MLP can ignore the language token when hkh_k alone predicts the timestamp t/(T−1)t/(T-1). However, if the paper pairs the visual trajectory of a negative task (Y) with the language instruction of task X as synthetic negative examples and supervise the head to predict 00 (or stall) progress the lightweight head would be forced to attend to task identity while adding almost zero inference overhead. I think it would be important to add this to the approach and then also compare with external VLM based approaches.

Questions for authors

see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.