-
The task-level metrics used in the main experiments are insufficient to validate the reference matching and semantic deviation modeling motivated by Q1 and Q2.
Q1 concerns selecting success references that match the current task and execution stage despite differences in strategy and timing. Q2 concerns representing deviations that violate the task instruction. However, the evaluation only reports task-level success/failure classification results, without directly assessing reference-matching quality or whether the deviation features distinguish valid behavioral variation from semantic errors.
The ablations in Table 3 (Section 4.4) show that multiple references and progress alignment improve final classification. These gains provide indirect support, but do not establish that PADR captures the distinctions motivating its design. A more direct evaluation would test whether PADR distinguishes successful executions with different strategies or speeds from executions that violate the instruction. A classifier using the same adapted VLM features would further help determine whether the proposed deviation representation improves on direct classification.
-
The data and experiments do not adequately support the recovery-aware design motivated by Q3.
Q3 asks whether a deviation is corrected by subsequent actions or remains relevant to failure. FRLED addresses this through a learned gate that discounts accumulated evidence. However, Section 4.1 constructs failures by pairing trajectories with mismatched instructions, which does not itself create cases where a robot deviates and then corrects its behavior. The paper does not report how many such cases occur in the training and evaluation sets, or whether the four sampled stages capture both the deviation and the correction.
The 2.30-point drop in F1 when explicit recovery is removed (Section 4.4, Table 3) shows that the gate helps final classification, but does not establish that it distinguishes corrected deviations from persistent ones. Since Eq. (12) supervises only the final prediction, the gain could also reflect a general benefit of forgetting past information. The authors should characterize cases involving corrective behavior and evaluate whether the model appropriately discounts earlier deviations after correction while retaining evidence of unresolved deviations. This would directly test the design motivated by Q3.
-
The principal baselines are not evaluated under a matched protocol.
Table 1 uses the published I-FailSense result rather than a new evaluation under DRFR’s protocol, as explicitly stated in Section 4.2. However, the manuscript does not establish whether the published result and DRFR’s validation result use the same evaluation samples and protocol. The reported accuracy difference therefore does not establish a performance advantage over I-FailSense.
Additionally, the reported I-FailSense accuracy of 91.18% differs from the 88.18% reported for the full two-view model in Table I of the cited paper [1]. Please provide the source of the 91.18% result or correct the entry. A comparison with I-FailSense under the same evaluation protocol is needed to assess relative performance; component ablations cannot substitute for this comparison.
-
Transfer results lack both dataset definitions and comparative baselines.
The manuscript names AHA/RLBench, SMF-DROID, and “a real-robot dataset” without adequately specifying the evaluated subsets, sample counts, class composition, or construction procedures. Figure 6 provides scene examples rather than an evaluation protocol. In particular, the successful examples used to compute AHA/RLBench AUROC are unspecified.
Table 2 also contains no baselines. Its scores therefore do not establish an advantage over existing detectors or over the adapted backbone alone. The datasets should be defined and evaluated against matched alternatives without target-domain adaptation. The reported AUROC range must also be corrected: Section 4.3 states 82.71%–83.48%, whereas Table 2 gives 82.71%, 81.48%, and 77.15%.
-
The claimed benefits of success-memory expansion are not evaluated.
CSME is a principal contribution, and the abstract claims improved detection on subsequent tasks. Yet Section 4.4 acknowledges that its benefit remains unquantified; the transfer experiments keep memory fixed.
A sequential evaluation comparing a fixed memory, a memory that admits all verified successes, and CSME under comparable memory budgets is needed. This should also examine whether the reliability filter rejects unfamiliar but genuinely successful strategies. Without such evidence, neither continual improvement nor the benefit of the admission rule is established.
Additional Suggestions for Improvement
-- Define the ablation settings precisely. Explain how “No task-conditioned retrieval,” “Fixed-stage matching,” and “No explicit recovery” are implemented and which components remain unchanged. For example, does removing recovery mean fixing (q_t=0) or replacing the entire evidence recurrence? These choices affect what can be concluded from the reported performance differences.
-- Provide the essential implementation details that are missing. Specify the temporal model architecture, the VLM adaptation objective, and how the two camera views are combined. Briefly explain SetEnc in the main text and refer readers to the appendix for details.
-- Resolve inconsistencies between Figure 2 and the equations. The injection and recovery head labels appear reversed relative to Eq. (9): evidence injection corresponds to (a_t) with softplus, whereas the recovery gate corresponds to (q_t) with sigmoid. The CSME branches also appear inconsistent with the convention that (y_n=1) denotes failure and with the acceptance and buffering conditions in Eq. (14). Please check the figure against both the equations and the implementation.
-- Thoroughly verify the reported results and their provenance. The discrepancy in the cited I-FailSense accuracy and the mismatch between the transfer AUROC range in Section 4.3 and Table 2 warrant a careful check of all reported numbers. Verify each result against its original source or evaluation output, confirm the associated dataset, configuration, and metric definition, and ensure consistency across the text, tables, and figures. The conclusions should also be checked against what each evaluation actually supports; correcting numerical discrepancies alone does not resolve gaps in experimental validation.
[1] Grislain et al., I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models, arXiv:2509.16072, Table I.