acceptodds
Under review as a conference paper at ICLR 2027

ReAP: From Visual Failure Recognition to Action-Conditioned Risk Prediction for Runtime Failure Monitoring

Abstract

Despite rapid progress, generalist robot policies remain brittle during execution. Reliable deployment therefore requires runtime monitoring that provides timely warnings of failure. Existing monitors commonly estimate risk from current observations or policy signals without explicitly predicting the future consequences of proposed actions. To anticipate such risks before action execution, we introduce ReAP, a future-aware runtime monitoring framework that combines visual failure recognition with action-conditioned failure prediction. Specifically, ReAP first trains a visual detector on successful and failed rollouts to learn a failure-relevant representation space and estimate current risk. An action-conditioned predictor then forecasts the resulting future representation in this space to estimate future failure risk. At runtime, the detector and predictor provide complementary signals for current-state failure recognition and future-aware risk assessment. Experiments across diverse robot policies in simulation and the real world demonstrate that ReAP effectively balances failure-detection accuracy and warning timeliness.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Runtime Failure Detection, Robotic Manipulation, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HbrgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReAP, a runtime failure monitor for robot policies that looks at the future instead of only the current state. A visual detector is trained on successful and failed rollouts. An action-conditioned predictor then forecasts the detector's latent representation after a proposed action chunk. This gives a future risk score before the action runs. It is tested on LIBERO-10 and RoboTwin with four policies and on three real-robot tasks.

Critical strengths
  • Predicting the consequences of a proposed action before it executes is well motivated and fills a real gap in current-state monitors.
  • The method is clean, and the paper is easy to follow. Two stages, one shared latent space, and one frozen detector head keep the risk semantics consistent.
  • The evaluation is broad and reliable. It covers four policy-benchmark settings, three seeds, five calibration levels, and comparisons with baselines from three families.
Critical weaknesses
  1. Incremental novelty. Action-conditioned prediction of future latents for failure detection is already established (Liu et al., 2024/2025; ForeSight; ContactGuard). ReAP's specific contribution is to predict in a failure-supervised representation space with a frozen head. Table 3a supports this choice in-distribution (0.683 vs 0.644 for generic PCA), but the generalization benefit is not shown: seen-to-unseen transfer (Table 9) is on par with baselines (0.568 vs 0.556).
  2. The gain from the action-conditioned predictor is small. Across the four settings in Table 1, the predictor improves bTWA over the detector by only 0.005-0.014 (computed from the Detector and Predictor rows).
  3. The comparison with baselines is not fully fair. ReAP uses failed-rollout labels and two alarm branches, while several baselines use neither. The OR rule is also not matched to the baselines' false-alarm rate: each branch is calibrated separately, so the combined alarm budget exceeds that of a single calibrated detector. ReAP does not win bACC or T-det consistently.
  4. The real-world evaluation is small (40 rollouts per task) with no variance reported. Several differences are within noise, and baselines are run post hoc.
  5. The paper does not test whether warnings improve task success. Robustness is only checked through cross-policy transfer, where performance drops and ReAP only ties RoboMeter when transferring to OpenVLA. There is no test under environment or task shift, and component analysis is on one setting only.
Questions for authors
  1. How does ReAP differ from Liu et al. (2024/2025) beyond the choice of a failure-supervised prediction space, and can you show that this space generalizes better than a video-pretrained or self-supervised world model under task or policy shift?
  2. Can you isolate the contribution of genuine latent-dynamics prediction from direct future-risk supervision, for example by reporting the risk-only and latent-only variants per setting?
  3. Can you report all methods at a matched false-alarm rate or alarm budget, including a fair calibration of the OR combination?
  4. Can you report real-robot variance and add more rollouts per task?
  5. Does using the warning to trigger a fallback improve task success?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Hbrg - Part 1/2

We thank the reviewer for recognizing the motivation of evaluating actions before execution, the shared risk head, and the breadth of our evaluation. We address each concern below and refer to the other responses for shared experimental results.

1. Failure-supervised representations [W1; Q1]

We appreciate the reviewer’s pointers to these closely related works. ReAP uses failure supervision to shape the representation space before learning action-conditioned transitions, making the prediction target directly relevant to failure monitoring.

Liu et al. (2024) [1] jointly train a policy and latent dynamics model, then train a failure classifier on the frozen latent space using pre-intervention states as failure labels. Sirius-Fleet [2] pretrains a visual world model through image reconstruction and future-embedding prediction, then builds task-specific failure classifiers and OOD detectors on its frozen representations. Its failure labels also come from human intervention records.

ReAP first trains the detector representation using episode-level success/failure outcomes, then freezes the detector and trains the predictor with latent-transition and future-risk objectives. This design directs prediction toward failure-relevant features and uses the same risk head to assess observed and predicted states. It enables monitoring of an existing policy using rollout outcomes, without requiring intervention annotations or joint policy training. The learned/random/PCA comparison in Table 3(a) supports the value of this failure-supervised prediction space.

Foresight learns predictions in a video-pretrained representation space and trains a downstream temporal failure detector. Section 2 and Appendix A discuss these distinctions. In our additional π0\pi_0/LIBERO comparison, ReAP OR achieves higher bTWA than our Foresight reproduction (0.780 vs. 0.730) and greater before-onset coverage with fewer false alarms. These results support ReAP’s effectiveness in both failure monitoring and advance warning. Full results and reproduction details appear in Item 5, “Predictive monitoring baselines,” of our response to Reviewer JvV7.

[1] Huihan Liu et al. Model-Based Runtime Monitoring with Interactive Imitation Learning. ICRA, 2024. https://arxiv.org/abs/2310.17552

[2] Huihan Liu et al. Multi-Task Interactive Robot Fleet Learning with Visual World Models. CoRL 2024, PMLR 270, 2025. https://proceedings.mlr.press/v270/liu25g.html

2. Predictor contribution [W2]

Our central idea is to combine current-state recognition and action-conditioned prediction to improve the complete monitor. Table 1 shows that ReAP OR achieves higher bTWA and earlier detection than either branch alone across all four settings. For example, on OpenVLA, OR raises bTWA from 0.671 to 0.701 over the detector and reduces T-det from 0.450 to 0.326.

Appendix D.5 explains this complementarity. At the operating point reported in Table 17, the predictor detects 17 OpenVLA failures missed by the detector, while the detector detects four missed by the predictor. OR retains both sets of warnings. Figure 7 illustrates how prediction can anticipate failure before clear visual cues emerge, while recognition captures failures missed by prediction. Table 16 further shows that OR achieves higher bTWA and earlier detection than AND across all four settings. These results demonstrate the benefit of combining the two branches.

3. Training objectives [Q2]

Figure 3(b) compares latent-only, risk-only, and joint training on OpenVLA/LIBERO. We provide the corresponding metrics below and extend the comparison to π0\pi_0/LIBERO.

Training objectivebTWA ↑bACC ↑T-det ↓
Latent-only0.493 ± 0.0090.522 ± 0.0020.794 ± 0.239
Risk-only0.642 ± 0.0320.740 ± 0.0320.529 ± 0.090
Joint0.683 ± 0.0070.764 ± 0.0170.406 ± 0.038

We additionally compared the three objectives for the predictor on π0\pi_0/LIBERO.

Training objectivebTWA ↑bACC ↑T-det ↓
Latent-only0.665 ± 0.0340.808 ± 0.0210.593 ± 0.079
Risk-only0.735 ± 0.0040.820 ± 0.0070.480 ± 0.006
Joint0.755 ± 0.0090.831 ± 0.0100.461 ± 0.017

Joint training achieves the highest bTWA and bACC and the lowest T-det on both policies, supporting the complementary roles of latent-transition and future-risk supervision.

AuthorsOpenReview ↗

Response to Reviewer Hbrg - Part 2/2

4. Matched-FAR comparison [W3; Q3]

We agree that supervision requirements and false-alarm costs are important for a fair comparison. ReAP and SAFE both use successful and failed rollouts. For LogpZO, RND, and FIPER, our implementations also use both successful and failed rollouts to model their respective distributions. Action-uncertainty methods are designed for zero-shot evaluation, and we include them as reference baselines.

We also compare methods at 5%, 10%, and 15% episode-level FAR, accounting for alarms from either branch. On π0\pi_0, OR achieves the highest mean bTWA at 5% FAR and improves over both branches at 5% and 10% FAR. Full tables and the test-curve interpolation procedure appear in Item 6, “Matched-FAR comparison,” of our response to Reviewer JvV7.

5. Real-robot evaluation [W4; Q4]

We appreciate the reviewer’s concern about sample size and uncertainty. We conducted an additional evaluation on 80 Fold Cloth rollouts, obtaining 0.705 bACC and 0.600 bTWA for ReAP OR. These results are close to the 0.713 bACC and 0.603 bTWA reported for 40 rollouts in Table 2. We will include standard deviations for our methods and baselines in the revised paper.

6. Alarm-triggered recovery [W5; Q5]

The connection between failure warnings and successful recovery is important for practical deployment. ReAP focuses on accurate and timely warnings, which can trigger a pause and hand control to a human or recovery policy. Task success also depends on the intervention strategy and recovery capability.

In our preliminary π0\pi_0/LIBERO experiment, the original policy and SAFE- and ReAP-triggered QAM-based steering from RL$^2$-VLA each achieved 16/20 successes. Details and the RL$^2$-VLA reference appear in Item 3, “Alarm-triggered recovery,” of our response to Reviewer JvV7. Developing recovery mechanisms that effectively use these warnings remains a complementary direction for future work.

7. Cross-policy transfer [W5]

We appreciate the reviewer’s comment on cross-policy transfer. This experiment tests a practical advantage of ReAP: it operates on visual observations and proposed actions, independently of policy-internal features or a particular action-generation mechanism. This enables transfer between the autoregressive OpenVLA policy and the flow-based π0\pi_0 policy. Feature-based monitors such as SAFE are tied to the source policy’s internal representations, making such direct transfer more difficult.

As shown in Appendix D.1, Table 10, ReAP outperforms RoboMeter when transferring to π0\pi_0 and achieves comparable performance when transferring to OpenVLA, while using a substantially smaller model. These results support the value of ReAP’s policy-independent monitoring ability across different policy architectures.

We sincerely thank the reviewer for the valuable feedback. With the discussion period drawing to a close, we would appreciate hearing whether our responses and additional experiments have addressed your concerns. We remain happy to clarify any outstanding questions or provide further details if needed.

Dear Reviewer Hbrg,

Please respond to the author soon and engage in the discussion.

Best,

AC

Reviewer_htHFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors present ReAP, a runtime failure-monitoring framework that combines visual failure detection and action-conditioned prediction. The approach is to first learn a failure-relevant visual representation by using episode-level supervision from robot failures and successes during rollout. Then, they freeze the detector and learn a predictor that can predict a potential future representation induced by a proposed action sequence. The authors use the same detector head for the current and predicted representation to flag a failure warning if any of the scores exceeds their respective calibrated thresholds. Experiments are shown in four simulated policy-benchmark settings and three real robot manipulation tasks.

Critical strengths
  1. The testing measures ReAP across various policy structures, two simulation benchmarks and tests on physical robots. Based on the research results, the ReAP model provides the highest balanced time-weighted accuracy for all four main simulation settings, indicating its benefits are probably not restricted to just one class of policy.
  2. An analysis of components appears to be useful. ablation tests suggest what roles representation, action conditioning, prediction losses, prediction horizon and alarm fusion play. There is evidence to suggest that performance is adversely affected when actions are removed or permuted, and that the combination of future-risk supervision and latent prediction may surpass either one on its own.
  3. This method requires only the final success or failure of an episode to be provided for training, removing the requirement for manually labeling the failure-onset time. The stand-alone vision detector as well as the measured overhead of inference time also indicate that the monitoring architecture is feasible.
Critical weaknesses
  1. This training supervision is a heuristic surrogate of risk of failure. In this heuristic, all failed trajectories receive a target that linearly increases with normalized time, regardless of when the failure occurred. Although this assumption may not capture abrupt failures, early irreversible errors, and recoverable deviations, comparisons to binary labels seem to support this assumption empirically. However, it is unclear how well the learned score correlates to the true actionable risk of failure.
  2. A comparative statement of the proposed method versus the related methods of predictive-monitoring has not been fully made. Appendix A describes Foresight and ContactGuard, but this again is a conceptual comparison rather than the comparison with experimental results. Internal ablation studies give some indication of the benefit of the individual components but this comparison would ideally be performed with some controlled experiment comparing the advantages of the proposed training methodology. It is recognized that these methods are described as concurrent work.
  3. The links made between ReAP and related approaches on failure representations and robustness assessment could be more thorough. For example, knowledge built from success and failure data to represent the environment for detecting weak spots (RoboMD [1]) and the systematic evaluation of manipulation policies in disturbed environments (The Colosseum [2]) are relevant. Strengthening these links would probably help inform the scope of the ReAP system and motivate the experiment to test the monitoring system in different conditions: lighting, appearance, distractors, and camera placement.
  4. The usefulness of the warnings, however, is only partially demonstrated. For the real robot experiments, alarms were collected but there was no change in execution. The results show that it is possible to monitor successfully, but it is not clear if the use of alarm points makes effective intervention possible, and the extra analysis using model-generated annotations leaves uncertainty about the recorded lead times.

References [1] Sagar, Som, et al. "Robomd: Uncovering robot vulnerabilities through semantic potential fields." International Conference on Learning Representations. Vol. 2026. 2026. [2] Pumacay, Wilbert, et al. "The colosseum: A benchmark for evaluating generalization for robotic manipulation." arXiv preprint arXiv:2402.08191 (2024).

Questions for authors
  1. How does ReAP do with its individual branches and at matched empirical trajectory-level false-alarm rates? As the branches are fit independently and then merged using an OR rule, the same nominal calibration parameters do not in fact correspond to the same total false-alarm rate.
  2. What is the duration of the lead time, in control steps or seconds, before a human-validated failure onset? This information would be especially useful for OpenVLA, since the prediction horizon is one step. What percentage of rollouts that failed had an alarm precede the onset?
  3. In the unseen-task experiment, thresholds are derived from the successful rollouts of the target tasks. How does this calibration impact results and how does performance change when transferring both model and thresholds?
5 comments
AuthorsOpenReview ↗

Response to Reviewer htHF - Part 1/2

We thank the reviewer for recognizing ReAP’s evaluation across policy architectures, its component ablations, and the practicality of learning from rollout outcomes without manual failure-onset annotations. We address the weaknesses first, followed by the specific questions. Where concerns overlap with Reviewer JvV7, we summarize the relevant findings here and refer to that response for the full tables.

1. Training labels [W1]

The reviewer raises an important question about the supervision scheme and the interpretation of the learned score. As discussed in the paper, our monotonic progress-based labels require only episode-level success/failure outcomes and timestep indices, without additional human or model-generated annotations. They also achieve higher bTWA than binary episode labels for both our detector and SAFE-MLP (Appendix D.2, Table 13).

We compared the original labels with onset-step and local error-interval labels constructed by annotating the full training set with GPT-6 Astra, followed by manual checking and correction. Under the same OpenVLA split and training budget, the three schemes yield similar performance, with the original labels achieving the highest OR bTWA of 0.691, compared with 0.682 and 0.664, respectively. Our focus is effective monitoring using the original episode-level outcomes without additional annotation. The full comparison and label definitions appear in Item 1, “Training labels,” of our response to Reviewer JvV7.

2. Predictive monitoring baselines [W2]

Following the reviewer’s suggestion, we added a comparison with Foresight. Our reproduction of Foresight achieves 0.838 bACC and 0.730 bTWA on π0\pi_0/LIBERO, compared with 0.865 ± 0.015 and 0.780 ± 0.008 for ReAP OR across three seeds. The full comparison appears in Item 5, “Predictive monitoring baselines,” of our response to Reviewer JvV7.

Foresight’s official code is unavailable, so we reproduced it from the paper. ContactGuard’s code is also unavailable, and its direct comparison remains unimplemented. Both are concurrent works whose differences from ReAP are discussed in Section 2 and Appendix A, including failure supervision, prediction space, and monitoring scope.

3. Visual perturbations [W3]

We thank the reviewer for pointing us to RoboMD and The Colosseum. RoboMD searches for policy vulnerabilities under environmental changes using reinforcement learning in a learned vision-language space. The Colosseum systematically evaluates manipulation policies under environmental perturbations. These works motivate evaluating how reliably ReAP monitors execution under visual variation.

ReAP maintains similar bTWA under most tested perturbations. The largest decrease is 0.035 under warm tint on Pick Cubes; all other decreases are at most 0.021. We probed sensitivity by applying RGB-channel scaling and clipping to recorded real-robot images, keeping trajectories, checkpoints, and thresholds fixed. The same transformation is applied to all camera views.

TaskOriginalDimWarm tintBlue tint
Fold Cloth0.6030.5820.6010.586
Stack Plates0.8180.8180.8090.809
Pick Cubes0.7180.7100.6830.710

These results provide an initial probe of sensitivity to synthetic visual variation. We will further evaluate ReAP on data collected under varied lighting, backgrounds, and camera viewpoints to assess monitoring performance under real environmental changes.

4. Alarm-triggered recovery [W4]

The connection between failure warnings and successful recovery is important for practical deployment. An alarm can trigger a pause and hand control to a human operator or a recovery policy. ReAP focuses on accurate and timely warnings, while the resulting task success also depends on the intervention strategy and recovery capability. Our real-robot experiments record alarms during execution to evaluate monitoring performance.

We additionally connected SAFE and ReAP alarms to the same QAM-based steering mechanism from RL$^2$-VLA [1] on π0\pi_0/LIBERO. The original policy and both alarm-triggered variants each achieved 16/20 successes. Experimental details appear in Item 3, “Alarm-triggered recovery,” of our response to Reviewer JvV7. Developing recovery mechanisms that effectively use these warnings is a complementary direction for future work.

[1] Derek Ming Siang Tan et al. RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models. arXiv:2607.26991, 2026. https://arxiv.org/abs/2607.26991

AuthorsOpenReview ↗

Response to Reviewer htHF - Part 2/2

5. Matched-FAR comparison [Q1]

We agree that matching the overall false-alarm rate is essential for assessing the benefit of OR fusion. We compared the detector, predictor, OR fusion, and baselines at 5%, 10%, and 15% episode-level FAR, so the comparison accounts for alarms from either branch. On π0\pi_0, OR achieves 0.798 ± 0.012 bTWA at 5% FAR, the highest mean among the evaluated methods, and improves over both individual branches at 5% and 10% FAR.

The complete results for both policies appear in Item 6, “Matched-FAR comparison,” of our response to Reviewer JvV7. We sweep thresholds on the test set and interpolate between neighboring points to compare methods at the same expected FAR.

6. Warning lead time [W4; Q2]

Onset-relative timing directly assesses whether alarms provide advance warning. Using manually annotated onsets for all 75 OpenVLA and 60 π0\pi_0 failed rollouts, ReAP OR achieves before-onset coverage of 66.5% and 61.9%, with normalized lead times of +0.182 and +0.177, respectively. The full baseline comparison and evaluation definitions appear in Item 2, “Warning lead time,” of our response to Reviewer JvV7.

The prediction horizon specifies how far ahead the predictor models the representation, while warning lead measures how early an alarm precedes failure onset. Even a one-step predictor can warn earlier when the observation and proposed action contain cues of a later failure. Lead is normalized by rollout length and averaged over detected failures.

7. Unseen-task calibration [Q3]

We thank the reviewer for highlighting the role of target-task calibration. To examine transferring both the monitor and its thresholds, we compared source-task and target-task calibration with the monitor and target evaluation set fixed. Since the monitor was trained on all available seen-task training data, we collected 28 new successful rollouts from the seven seen tasks for independent source calibration. Target calibration used 30 independent successful rollouts from the three unseen tasks. Both settings were evaluated on the same 120 target rollouts. This controlled comparison uses one fixed detector–predictor checkpoint pair, whereas Table 9 reports averages across three training seeds.

Calibration dataMonitorSuccess FAR ↓Failure TPR ↑bACC ↑bTWA ↑
Source-task successesDetector59.7%76.0%0.5820.450
Source-task successesPredictor67.7%88.0%0.6020.413
Source-task successesOR70.3%88.0%0.5880.423
Target-task successesDetector22.0%64.3%0.7120.602
Target-task successesPredictor23.3%67.7%0.7220.579
Target-task successesOR27.7%74.7%0.7350.610

Target calibration reduces false alarms and improves bACC and bTWA for all three monitors. To examine why, we compared risk scores on successful rollouts. Entries below are the median of each rollout’s mean score.

Successful rollout setDetector scorePredictor score
Source-task calibration0.0420.066
Target-task calibration0.1270.170
Target-task test0.1410.196

Both monitors assign higher scores to successful target-task executions, helping explain why source-calibrated thresholds trigger more false alarms. Target calibration scores are closer to the target test scores, suggesting that target calibration better accounts for this score-distribution shift.

Our unseen-task results therefore demonstrate transfer of the learned monitor with calibration on successful target-task rollouts. We acknowledge the dependence on target calibration as a limitation and will clarify this requirement in the paper. Improving threshold transfer across tasks remains an important direction for future work.

Thanks for doing the rematch experiments. The answers to the question of predicting monitoring baselines and transfer of thresholds to unseen tasks have resolved my concetsn. My main remaining worry is about what we can do if the alarm is sounded. The manually labelled onsets, no doubt, are convincing as warnings in time. But without second/timestep lead times, it is hard to assess whether this is ample time for the user to intervene. The recovery trigger after alarms did not improve task success, so is still unresolved whether that is enough benefit for recovery. The comparison at equal false alarm rate is nice. But the thresholds were swept on the test set; Ideally want to see them fixed and tested on held-out rollouts, due to these reasons I believe that the current score is justified.

Reviewer_JvV7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReAP, a runtime failure monitoring framework that combines visual failure recognition with action-conditioned risk prediction. The method first trains a visual detector to learn failure-relevant representations from successful and failed rollouts. An action-conditioned predictor is then trained to estimate future representations and their associated failure risks. The two components are combined to provide runtime failure warnings. Experiments on LIBERO, RoboTwin, and real-world manipulation tasks show improvements over several existing failure monitoring methods.

Critical strengths
  1. Clear motivation and practical relevance. The paper addresses an important problem in robotic manipulation. Predicting future failure risks based on proposed actions, rather than relying only on current observations, is a reasonable and practically useful direction.

  2. Meaningful ablation studies. The ablations provide evidence for the effectiveness of the main components. In particular, the comparisons of different representations, prediction losses, and action conditioning help explain why the proposed method improves failure monitoring performance.

Critical weaknesses
  1. Limitations in failure supervision and early warning validation.

    • The paper assigns linearly increasing risk labels to failed trajectories based on execution progress. However, failures do not necessarily develop monotonically over time. A robot may execute several correct actions before making a critical mistake, and a failed trajectory may contain both correct and incorrect intermediate behaviors. Therefore, the proposed labels may not accurately reflect the actual failure risk at each timestep.
    • Although the method aims to predict failures before they occur, the experiments do not directly evaluate warning time relative to the actual failure onset. Furthermore, the real-world experiments only record alarms without triggering interventions. It is therefore unclear whether these predictions can support timely and effective intervention during execution.
  2. Limited experimental coverage and insufficient comparisons.

    • The simulation experiments are limited to LIBERO-10 and policy-specific subsets of eight RoboTwin tasks. Given the diversity of robotic manipulation scenarios, broader task coverage and per-task analysis would help establish the generalizability of the method.
    • The paper also lacks direct experimental comparisons with closely related predictive monitoring methods, such as Foresight and ContactGuard. Additionally, the benefits of combining the detector and predictor through an OR rule should be further examined under matched overall false-alarm rates to ensure a fair comparison.
Questions for authors
  1. Can the authors evaluate how early the model raises alarms relative to the actual failure onset? Have they considered testing whether interventions triggered by these predictions can prevent failures?

  2. What motivates the use of linearly increasing risk labels for failed trajectories? How robust is the method when task progress is non-monotonic or when correct intermediate actions occur in eventually failed trajectories? Would failure-onset-based or phase-aware labels yield different results?

  3. Can the authors provide direct comparisons with closely related predictive monitoring methods, such as Foresight and ContactGuard? Does the OR fusion strategy still provide improvements when the overall false-alarm rate is matched across methods?

  4. How were the eight RoboTwin tasks selected for each policy? Can the authors provide additional task-level results or expand the evaluation to more diverse manipulation tasks to better demonstrate generalization?

5 comments
AuthorsOpenReview ↗

Response to Reviewer JvV7 - Part 1/3

We thank the reviewer for recognizing ReAP’s practical motivation and meaningful component ablations. Our central objective is to combine recognition of current failure cues with action-conditioned prediction of future risk. We address the concerns below in the order raised in the review.

1. Training labels [W1, first bullet; Q2]

The reviewer raises an important question about how the training labels affect monitoring performance. As discussed in the paper, our monotonic progress-based labels require only episode-level success/failure outcomes and timestep indices, without additional human or model-generated annotations. They also achieve higher bTWA than binary episode labels for both our detector and SAFE-MLP (Appendix D.2, Table 13).

To examine alternative supervision, we first used GPT-6 Astra to annotate the full training set, given the scale of the data, and then manually checked and corrected the annotations. We constructed two variants. Onset-step assigns 0 to normal intervals and 1 to error intervals, with all frames from an identified failure onset onward labeled 1. Local error-interval assigns 0 to normal intervals and 1 to error intervals, allowing the label to return to 0 when locally correct behavior resumes.

We kept the OpenVLA data split and training budget fixed. We report results for seed 0.

Training labelsDetector bTWA ↑Predictor bTWA ↑OR bTWA ↑
Progress-based0.6550.6810.691
Onset-step0.6830.6840.682
Local error intervals0.6430.6600.664

The three labeling schemes yield similar overall monitoring performance, with our original monotonic labels achieving the highest OR bTWA of 0.691, compared with 0.682 for onset-step labels and 0.664 for local error-interval labels. However, we acknowledge that the quality of temporal annotations is crucial to evaluating alternative labeling schemes. Our focus here is on learning effective failure monitors from episode-level success/failure outcomes without requiring additional annotations. Obtaining more accurate annotations and using them to improve detector and predictor training would be an interesting direction for future work.

2. Warning lead time [W1, second bullet; Q1]

We appreciate this suggestion, as onset-relative timing directly assesses whether alarms provide advance warning. We manually annotated failure onsets and evaluated all 75 OpenVLA and 60 π0\pi_0 failed rollouts, retaining every trajectory.

Before-onset coverage is the fraction of all failed rollouts receiving an alarm before onset, including missed failures in the denominator. Normalized lead is (onset time − alarm time) / rollout length, averaged over detected failures. Positive lead indicates advance warning.

OpenVLA

MethodDetected failures / 75 ↑Before-onset coverage ↑Normalized lead ↑Success FAR ↓
SAFE-LSTM12.3 ± 10.99.4% ± 5.1%+0.215 ± 0.18910.7% ± 7.0%
SAFE-MLP36.7 ± 2.129.5% ± 2.5%+0.107 ± 0.00411.5% ± 2.7%
FIPER51.3 ± 5.54.9% ± 0.3%−0.374 ± 0.0146.9% ± 0.8%
LogpZO22.7 ± 13.613.2% ± 9.3%+0.036 ± 0.12424.1% ± 15.9%
RND49.9 ± 10.329.5% ± 6.4%−0.044 ± 0.01723.6% ± 4.3%
Detector (ours)52.1 ± 6.151.5% ± 8.1%+0.138 ± 0.02720.8% ± 1.2%
Predictor (ours)56.7 ± 4.358.8% ± 4.5%+0.141 ± 0.01222.9% ± 2.4%
ReAP OR60.9 ± 2.366.5% ± 3.3%+0.182 ± 0.01027.3% ± 2.0%

π0\pi_0

MethodDetected failures / 60 ↑Before-onset coverage ↑Normalized lead ↑Success FAR ↓
STAC56.475.3%+0.17278.4%
ACCEL46.437.7%−0.00222.0%
Diff-DAgger59.482.3%+0.21992.9%
SAFE-LSTM54.0 ± 1.149.3% ± 3.1%+0.047 ± 0.01415.8% ± 0.3%
SAFE-MLP50.6 ± 0.355.4% ± 1.0%+0.107 ± 0.00410.5% ± 0.5%
FIPER50.9 ± 1.535.1% ± 2.3%−0.063 ± 0.01911.6% ± 2.4%
LogpZO50.6 ± 0.348.9% ± 1.3%+0.022 ± 0.00911.9% ± 0.9%
RND50.3 ± 0.851.3% ± 2.0%+0.081 ± 0.0208.9% ± 1.6%
Detector (ours)42.5 ± 2.253.3% ± 2.2%+0.153 ± 0.0273.3% ± 1.0%
Predictor (ours)41.5 ± 1.257.1% ± 1.6%+0.187 ± 0.0072.9% ± 0.3%
ReAP OR46.6 ± 2.361.9% ± 1.6%+0.177 ± 0.0164.6% ± 0.9%

ReAP OR achieves higher before-onset coverage than SAFE, RND, and FIPER on both policies, with fewer false alarms on π0\pi_0. On π0\pi_0, only STAC and Diff-DAgger achieve higher coverage, at substantially higher false-alarm rates. On OpenVLA, ReAP OR also achieves a strong normalized lead time, while its higher FAR reflects a more aggressive alarm behavior.

AuthorsOpenReview ↗

Response to Reviewer JvV7 - Part 2/3

3. Alarm-triggered recovery [W1, second bullet; Q1]

The connection between failure warnings and successful recovery is important for practical deployment. ReAP focuses on accurate and timely failure monitoring, particularly anticipating failure risk before executing proposed actions. Our contribution is to combine recognition of current failure cues with action-conditioned prediction of future risk, and our evaluation measures monitoring accuracy and warning timeliness. Translating these warnings into higher task success additionally requires an effective intervention strategy and a policy capable of recovering from the state where the alarm is raised. Task success therefore reflects the combined performance of the monitor and the recovery system.

To explore this connection, we integrated SAFE and ReAP alarms with the same QAM-based RL steering mechanism from RL$^2$-VLA [1] in the π0\pi_0/LIBERO setting. The original policy, SAFE-triggered steering, and ReAP-triggered steering each achieved 16 successes out of 20 trials. We report this preliminary result as an exploration of downstream integration. Our central research question concerns when failures can be detected and anticipated. Developing recovery mechanisms that effectively use these warnings is a complementary research direction, and we will leave it to future work.

[1] Derek Ming Siang Tan, Shailesh Shailesh, Srikrishna Iyer, William Wei Jie Teo, Yuanliang Ju, Qiao Gu, and Guillaume Sartoretti. RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models. arXiv preprint arXiv:2607.26991, 2026. https://arxiv.org/abs/2607.26991

4. RoboTwin task selection [W2, first bullet; Q4]

As described in Appendix C.1, we selected eight RoboTwin tasks separately for π0.5\pi_{0.5} and Fast-WAM based on their respective policy success rates. We excluded tasks with extremely low or high success rates and selected tasks spanning different manipulation operations and objects. This provides both successful and failed executions for monitoring evaluation while covering behaviors including bimanual lifting, object transfer, placement, stacking, and articulated-object manipulation. Table 7 lists the selected tasks and their success rates, ranging from 20.0% to 72.5% for π0.5\pi_{0.5} and from 52.5% to 82.5% for Fast-WAM.

5. Predictive monitoring baselines [W2, second bullet; Q3]

Following the reviewer’s suggestion, we added a paper-based reproduction of Foresight for a direct comparison with another predictive monitor. On π0\pi_0/LIBERO, the results are as follows:

MethodbACC ↑bTWA ↑
Foresight reproduction, 50 epochs0.8380.730
ReAP OR (ours)0.865 ± 0.0150.780 ± 0.008

ReAP results are from Table 1 of our paper and are reported as mean ± standard deviation across three seeds. Foresight results are from seed 0.

On the full manually annotated failure-onset evaluation set, this reproduction achieves 46.3% before-onset coverage at 8.4% FAR, compared with 61.9% coverage at 4.6% FAR for ReAP OR.

The official code for Foresight has not yet been released. Although we followed the model architecture and training settings described in the paper, our reproduction may still differ from the original implementation. The official code for ContactGuard is also unavailable, and we have not yet implemented it for a direct comparison. Given that Foresight and ContactGuard are concurrent works, we have already discussed their differences from ReAP in Section 2 and Appendix A, including the role of failure supervision in learning the prediction space, the monitoring scope, and the use of temporal context.

AuthorsOpenReview ↗

Response to Reviewer JvV7 - Part 3/3

6. Matched-FAR comparison [W2, second bullet; Q3]

We agree that matching the overall false-alarm rate is essential for assessing the benefit of OR fusion. We added comparisons at 5%, 10%, and 15% episode-level FAR, counting a successful rollout as a false alarm if it receives at least one alarm. Matching FAR controls the false-alarm cost when comparing detection accuracy and timeliness, including for OR fusion.

OpenVLA: bTWA at matched expected FAR

MethodFAR 5%FAR 10%FAR 15%
SAFE-LSTM0.502 ± 0.0030.492 ± 0.0060.490 ± 0.011
SAFE-MLP0.592 ± 0.0180.626 ± 0.0150.639 ± 0.009
FIPER0.563 ± 0.0020.597 ± 0.0140.594 ± 0.015
LogpZO0.493 ± 0.0050.493 ± 0.0110.488 ± 0.007
RND0.572 ± 0.0260.580 ± 0.0180.601 ± 0.011
Detector (ours)0.567 ± 0.0390.619 ± 0.0520.650 ± 0.051
Predictor (ours)0.582 ± 0.0370.636 ± 0.0290.673 ± 0.006
ReAP OR0.621 ± 0.0210.651 ± 0.0130.674 ± 0.020

π0\pi_0: bTWA at matched expected FAR

MethodFAR 5%FAR 10%FAR 15%
SAFE-LSTM0.681 ± 0.0180.717 ± 0.0130.739 ± 0.021
SAFE-MLP0.761 ± 0.0120.809 ± 0.0070.818 ± 0.003
FIPER0.729 ± 0.0100.769 ± 0.0010.770 ± 0.006
LogpZO0.695 ± 0.0130.743 ± 0.0140.763 ± 0.002
RND0.785 ± 0.0220.787 ± 0.0180.790 ± 0.016
Detector (ours)0.774 ± 0.0070.776 ± 0.0130.758 ± 0.017
Predictor (ours)0.785 ± 0.0040.780 ± 0.0070.783 ± 0.001
ReAP OR0.798 ± 0.0120.793 ± 0.0050.780 ± 0.005

We sweep thresholds on the test set and interpolate between neighboring points to compare methods at the same expected FAR. When thresholds yield the same FAR, we retain the highest bTWA.

ReAP OR achieves the highest mean bTWA at all three FAR levels on OpenVLA and at 5% FAR on π0\pi_0. Fusion also improves over both individual branches at all three levels on OpenVLA and at 5% and 10% FAR on π0\pi_0. These results demonstrate the benefit of combining current-state recognition and future-risk prediction under controlled false-alarm costs, particularly at the strict 5% FAR operating point across both policies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.