acceptodds
Under review as a conference paper at ICLR 2027

DRFR:Deviation Representation and Failure Risk Learning for Weakly Supervised Continual Monitoring of Robot Task Semantics

Abstract

Robots executing language-conditioned tasks may produce physically plausible actions that still violate the intended semantics. Continuously detecting such deviations and their associated failure risk is essential for reliable autonomous execution. Yet successful executions vary in strategy and timing, and local deviations can often be corrected by subsequent actions. This makes single-reference or instantaneous detection unreliable. We present a weakly supervised framework for continuous semantic failure detection that uses only task-level success/failure labels to learn process-level risk, while dynamically expanding its reference space with accumulated reliable successes. Our framework comprises three components. First, we introduce Progress-Aware Deviation Representation with Success Memory (PADR). PADR retrieves task-relevant successful trajectories using task-conditioned vision-language representations. It then aligns the current execution prefix with multiple candidate successes through open-ended monotonic alignment, producing task-relevant relative deviation signals. Second, we propose Failure Risk Learning from Execution Deviations (FRLED). By leveraging non-negative evidence injection and a learnable recovery gate, FRLED tracks the accumulation and resolution of deviation evidence and estimates failure risk during execution under task-level success/failure supervision. Finally, we introduce Continual Success Memory Expansion via Reliability and Novelty Gating (CSME). CSME accepts, buffers, or rejects candidate experiences according to task outcome, behavioral reliability, and experience novelty. Newly confirmed success patterns are then incorporated into the reference memory. Experimental results demonstrate that the proposed framework can identify semantic task failures, while further improves detection performance on subsequent tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Robotic Failure Detection, Execution Deviation, Weakly Supervised Learning, Vision-Language Models, Robot Manipulation, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bPQFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an approach for failure detection in robotics. The authors motivate that continuously detecting deviations of robot behavior from the intended task semantics is important. They first propose a retrival-based method that retrieves task-relevant successful trajectories to compare with the robot's execution and measure task deviation. They then propose a mechanism to continuously measure the recoverability of the robot during the deviation of execution. Finally, they propse a mechansim to incorporate the evaluated trajecotry into the memory for future retrieval.

Overall, the idea of using memory retrieval to monitor robotic execution is interesting. The proposed continual monitoring of progress is important for real-time failure detection. However, I have major concerns about how the experiment results provided in the paper do not support the authors' claims.

Additionally, there are major formatting issues in the paper. The texts are not in the standard ICLR format, figures have very small texts that are hard to read, and the AI use statement is missing.

Critical strengths

Using a retrieval-based approach for robotic failure detection is a promising area of research. Continually monitoring progress to detect failures in real-time is important for safe and reliable deployment. The authors make the motivation of their work very clear.

Critical weaknesses

Experiment results do not validate the authors' central claims. The authors claim their method can monitor an ongoing execution and distinguish recoverable deviations from deviations leading to failure. The experimental setting is outcome failure classification, which does not test the method's ability to continuously monitor failure or measure recoverability.

Experiment results have very little significance. The authors report published baseline numbers and zero-shot baselines without training, whereas their method is trained on the in-domain dataset. Further, baselines are evalauted on 1 dataset whereas their method is evaluated on 3 additional datasets without comparison.

The ablation results do not show that the proposed components are needed. Removing retrieval, alignment, or recovery changes AUROC by less than one percentage point, so these results do not justify the method's design.

Questions for authors

Can the authors compare against existing approaches that leverage fine-tuning on their existing datasets, e.g. AHA[1]?

Can the authors compare against a simple failure classifier trained on the same VLM features (see an example in ARMOR[2])?

Given the ablation study results showing less than 1% improvement by removing retrieval, alignment, or recovery, what justifies their existence in the method?


References

[1] AHA: A Vision-Language Model for Detecting and Reasoning over Failures in Robotic Manipulation, Duan et al., 2025

[2] Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning, Qi et al., 2026

Reviewer_Nk77OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to monitor semantic deviations and estimate failure risk during robot task execution. The proposed framework, DRFR, compares the observed execution with multiple successful trajectories aligned to the current task stage. It then uses a temporal model to estimate failure risk, with a learned gate that discounts previously accumulated failure evidence. After execution, verified successes are screened for inclusion in the reference memory. The evaluation consists of semantic-misalignment classification on a CALVIN-based dataset, component ablations, and transfer tests on AHA/RLBench, SMF-DROID, and a real-robot dataset. The approach is reasonably motivated, but the experiments do not adequately validate the claimed semantic deviation representation, recovery mechanism, or benefits of success-memory expansion.

Critical strengths
  • The paper focuses on a relevant problem: physically plausible robot behavior can violate task instructions, while temporary deviations do not necessarily lead to failure.
  • The framework has a coherent design. Multiple success references and progress alignment account for variation in strategy and timing, while the gated evidence update provides a mechanism for discounting past deviations.
  • Table 3 provides initial empirical support for the proposed components. Using multiple references improves failure F1 from 89.42% to 93.53%, and removing progress alignment or explicit recovery also reduces classification performance.
Critical weaknesses
  • The task-level metrics used in the main experiments are insufficient to validate the reference matching and semantic deviation modeling motivated by Q1 and Q2.

    Q1 concerns selecting success references that match the current task and execution stage despite differences in strategy and timing. Q2 concerns representing deviations that violate the task instruction. However, the evaluation only reports task-level success/failure classification results, without directly assessing reference-matching quality or whether the deviation features distinguish valid behavioral variation from semantic errors. The ablations in Table 3 (Section 4.4) show that multiple references and progress alignment improve final classification. These gains provide indirect support, but do not establish that PADR captures the distinctions motivating its design. A more direct evaluation would test whether PADR distinguishes successful executions with different strategies or speeds from executions that violate the instruction. A classifier using the same adapted VLM features would further help determine whether the proposed deviation representation improves on direct classification.

  • The data and experiments do not adequately support the recovery-aware design motivated by Q3.

    Q3 asks whether a deviation is corrected by subsequent actions or remains relevant to failure. FRLED addresses this through a learned gate that discounts accumulated evidence. However, Section 4.1 constructs failures by pairing trajectories with mismatched instructions, which does not itself create cases where a robot deviates and then corrects its behavior. The paper does not report how many such cases occur in the training and evaluation sets, or whether the four sampled stages capture both the deviation and the correction.

    The 2.30-point drop in F1 when explicit recovery is removed (Section 4.4, Table 3) shows that the gate helps final classification, but does not establish that it distinguishes corrected deviations from persistent ones. Since Eq. (12) supervises only the final prediction, the gain could also reflect a general benefit of forgetting past information. The authors should characterize cases involving corrective behavior and evaluate whether the model appropriately discounts earlier deviations after correction while retaining evidence of unresolved deviations. This would directly test the design motivated by Q3.

  • The principal baselines are not evaluated under a matched protocol.

    Table 1 uses the published I-FailSense result rather than a new evaluation under DRFR’s protocol, as explicitly stated in Section 4.2. However, the manuscript does not establish whether the published result and DRFR’s validation result use the same evaluation samples and protocol. The reported accuracy difference therefore does not establish a performance advantage over I-FailSense.

    Additionally, the reported I-FailSense accuracy of 91.18% differs from the 88.18% reported for the full two-view model in Table I of the cited paper [1]. Please provide the source of the 91.18% result or correct the entry. A comparison with I-FailSense under the same evaluation protocol is needed to assess relative performance; component ablations cannot substitute for this comparison.

  • Transfer results lack both dataset definitions and comparative baselines.

    The manuscript names AHA/RLBench, SMF-DROID, and “a real-robot dataset” without adequately specifying the evaluated subsets, sample counts, class composition, or construction procedures. Figure 6 provides scene examples rather than an evaluation protocol. In particular, the successful examples used to compute AHA/RLBench AUROC are unspecified.

    Table 2 also contains no baselines. Its scores therefore do not establish an advantage over existing detectors or over the adapted backbone alone. The datasets should be defined and evaluated against matched alternatives without target-domain adaptation. The reported AUROC range must also be corrected: Section 4.3 states 82.71%–83.48%, whereas Table 2 gives 82.71%, 81.48%, and 77.15%.

  • The claimed benefits of success-memory expansion are not evaluated. CSME is a principal contribution, and the abstract claims improved detection on subsequent tasks. Yet Section 4.4 acknowledges that its benefit remains unquantified; the transfer experiments keep memory fixed. A sequential evaluation comparing a fixed memory, a memory that admits all verified successes, and CSME under comparable memory budgets is needed. This should also examine whether the reliability filter rejects unfamiliar but genuinely successful strategies. Without such evidence, neither continual improvement nor the benefit of the admission rule is established.

Additional Suggestions for Improvement

-- Define the ablation settings precisely. Explain how “No task-conditioned retrieval,” “Fixed-stage matching,” and “No explicit recovery” are implemented and which components remain unchanged. For example, does removing recovery mean fixing (q_t=0) or replacing the entire evidence recurrence? These choices affect what can be concluded from the reported performance differences.

-- Provide the essential implementation details that are missing. Specify the temporal model architecture, the VLM adaptation objective, and how the two camera views are combined. Briefly explain SetEnc in the main text and refer readers to the appendix for details.

-- Resolve inconsistencies between Figure 2 and the equations. The injection and recovery head labels appear reversed relative to Eq. (9): evidence injection corresponds to (a_t) with softplus, whereas the recovery gate corresponds to (q_t) with sigmoid. The CSME branches also appear inconsistent with the convention that (y_n=1) denotes failure and with the acceptance and buffering conditions in Eq. (14). Please check the figure against both the equations and the implementation.

-- Thoroughly verify the reported results and their provenance. The discrepancy in the cited I-FailSense accuracy and the mismatch between the transfer AUROC range in Section 4.3 and Table 2 warrant a careful check of all reported numbers. Verify each result against its original source or evaluation output, confirm the associated dataset, configuration, and metric definition, and ensure consistency across the text, tables, and figures. The conclusions should also be checked against what each evaluation actually supports; correcting numerical discrepancies alone does not resolve gaps in experimental validation.

[1] Grislain et al., I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models, arXiv:2509.16072, Table I.

Questions for authors

Please refer to the “Critical Weakness” section for details.

Reviewer_VJDyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DRFR, a weakly supervised framework for monitoring semantic failures in language-conditioned robotic tasks. The core idea is to assess failure risk from how task-relevant deviations evolve over time, rather than from a single observation. The framework consists of three components: PADR compares the observed execution prefix with multiple progress-aligned successful trajectories; FRLED models deviation evidence using injection and recovery gates with task-level supervision; and CSME expands the reference memory with outcome-verified executions.

Critical strengths
  1. This paper proposes a method that compares an execution with multiple progress-aligned successful trajectories and models deviation evidence that accumulates or diminishes over time, providing a mechanism to handle temporary mistakes that may later be corrected.
  2. This paper clearly specifies the memory-update procedure, strictly separates prediction from outcome verification and requires multiple confirmations before admitting unfamiliar success patterns.
  3. The ablations support two central design choices: using a single reference reduces F1 from 93.53% to 89.42%, while removing explicit recovery reduces it to 91.23%.
Critical weaknesses
  1. The authors explicitly acknowledge in Section 4.4 that a fixed-versus-updated memory comparison is missing, and the transfer experiments leave the memory unchanged. The claim that memory expansion improves subsequent monitoring seems to lack sequential evidence.
  2. Comparing results across different protocols and splits is unfair. Although the "without retrieval" ablation is helpful, the paper lacks a temporal classifier baseline using the same backbone but no success-reference alignment.
  3. The presentation needs significant improvement. Figure 2 is overly dense with overlapping elements (e.g., the "..." symbols obscure other parts). A thumbnail version seems better here, since the details of each module are already illustrated later. Additionally, pushing the calculations for A_{n,j}^t, \omega_{n,j}^t and \Delta_{n,j}^t to the appendix makes the main text disjointed. There is an inconsistency in the reported results: Table 2 reports 82.71%, 81.48%, and 77.15%, but the text mentions 83.48%.
Questions for authors
  1. How does the method compare with a temporal classifier using the same adapted backbone but without success-reference alignment? Also, for Table 1, can the authors clarify the protocol differences and confirm if a shared-split comparison is available?
  2. Since failures are constructed by pairing valid trajectories with mismatched instructions, does the evaluation include cases where a semantic deviation from the instruction is later corrected? If not, what evidence supports the claim that the recovery gate distinguishes temporary deviations from persistent failures?
  3. Section 4.4 states that fixed and updated memory are not compared, and the transfer experiments keep the memory unchanged. Was CSME actually updated in any reported experiment? If so, which result isolates its effect on subsequent monitoring?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.