- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ReAP, a runtime failure monitor for robot policies that looks at the future instead of only the current state. A visual detector is trained on successful and failed rollouts. An action-conditioned predictor then forecasts the detector's latent representation after a proposed action chunk. This gives a future risk score before the action runs. It is tested on LIBERO-10 and RoboTwin with four policies and on three real-robot tasks.
Critical strengths
- Predicting the consequences of a proposed action before it executes is well motivated and fills a real gap in current-state monitors.
- The method is clean, and the paper is easy to follow. Two stages, one shared latent space, and one frozen detector head keep the risk semantics consistent.
- The evaluation is broad and reliable. It covers four policy-benchmark settings, three seeds, five calibration levels, and comparisons with baselines from three families.
Critical weaknesses
- Incremental novelty. Action-conditioned prediction of future latents for failure detection is already established (Liu et al., 2024/2025; ForeSight; ContactGuard). ReAP's specific contribution is to predict in a failure-supervised representation space with a frozen head. Table 3a supports this choice in-distribution (0.683 vs 0.644 for generic PCA), but the generalization benefit is not shown: seen-to-unseen transfer (Table 9) is on par with baselines (0.568 vs 0.556).
- The gain from the action-conditioned predictor is small. Across the four settings in Table 1, the predictor improves bTWA over the detector by only 0.005-0.014 (computed from the Detector and Predictor rows).
- The comparison with baselines is not fully fair. ReAP uses failed-rollout labels and two alarm branches, while several baselines use neither. The OR rule is also not matched to the baselines' false-alarm rate: each branch is calibrated separately, so the combined alarm budget exceeds that of a single calibrated detector. ReAP does not win bACC or T-det consistently.
- The real-world evaluation is small (40 rollouts per task) with no variance reported. Several differences are within noise, and baselines are run post hoc.
- The paper does not test whether warnings improve task success. Robustness is only checked through cross-policy transfer, where performance drops and ReAP only ties RoboMeter when transferring to OpenVLA. There is no test under environment or task shift, and component analysis is on one setting only.
Questions for authors
- How does ReAP differ from Liu et al. (2024/2025) beyond the choice of a failure-supervised prediction space, and can you show that this space generalizes better than a video-pretrained or self-supervised world model under task or policy shift?
- Can you isolate the contribution of genuine latent-dynamics prediction from direct future-risk supervision, for example by reporting the risk-only and latent-only variants per setting?
- Can you report all methods at a matched false-alarm rate or alarm budget, including a fair calibration of the OR combination?
- Can you report real-robot variance and add more rollouts per task?
- Does using the warning to trigger a fallback improve task success?
4 comments
Response to Reviewer Hbrg - Part 2/2
4. Matched-FAR comparison [W3; Q3]
We agree that supervision requirements and false-alarm costs are important for a fair comparison. ReAP and SAFE both use successful and failed rollouts. For LogpZO, RND, and FIPER, our implementations also use both successful and failed rollouts to model their respective distributions. Action-uncertainty methods are designed for zero-shot evaluation, and we include them as reference baselines.
We also compare methods at 5%, 10%, and 15% episode-level FAR, accounting for alarms from either branch. On , OR achieves the highest mean bTWA at 5% FAR and improves over both branches at 5% and 10% FAR. Full tables and the test-curve interpolation procedure appear in Item 6, “Matched-FAR comparison,” of our response to Reviewer JvV7.
5. Real-robot evaluation [W4; Q4]
We appreciate the reviewer’s concern about sample size and uncertainty. We conducted an additional evaluation on 80 Fold Cloth rollouts, obtaining 0.705 bACC and 0.600 bTWA for ReAP OR. These results are close to the 0.713 bACC and 0.603 bTWA reported for 40 rollouts in Table 2. We will include standard deviations for our methods and baselines in the revised paper.
6. Alarm-triggered recovery [W5; Q5]
The connection between failure warnings and successful recovery is important for practical deployment. ReAP focuses on accurate and timely warnings, which can trigger a pause and hand control to a human or recovery policy. Task success also depends on the intervention strategy and recovery capability.
In our preliminary /LIBERO experiment, the original policy and SAFE- and ReAP-triggered QAM-based steering from RL$^2$-VLA each achieved 16/20 successes. Details and the RL$^2$-VLA reference appear in Item 3, “Alarm-triggered recovery,” of our response to Reviewer JvV7. Developing recovery mechanisms that effectively use these warnings remains a complementary direction for future work.
7. Cross-policy transfer [W5]
We appreciate the reviewer’s comment on cross-policy transfer. This experiment tests a practical advantage of ReAP: it operates on visual observations and proposed actions, independently of policy-internal features or a particular action-generation mechanism. This enables transfer between the autoregressive OpenVLA policy and the flow-based policy. Feature-based monitors such as SAFE are tied to the source policy’s internal representations, making such direct transfer more difficult.
As shown in Appendix D.1, Table 10, ReAP outperforms RoboMeter when transferring to and achieves comparable performance when transferring to OpenVLA, while using a substantially smaller model. These results support the value of ReAP’s policy-independent monitoring ability across different policy architectures.
We sincerely thank the reviewer for the valuable feedback. With the discussion period drawing to a close, we would appreciate hearing whether our responses and additional experiments have addressed your concerns. We remain happy to clarify any outstanding questions or provide further details if needed.
Dear Reviewer Hbrg,
Please respond to the author soon and engage in the discussion.
Best,
AC
Response to Reviewer Hbrg - Part 1/2
We thank the reviewer for recognizing the motivation of evaluating actions before execution, the shared risk head, and the breadth of our evaluation. We address each concern below and refer to the other responses for shared experimental results.
1. Failure-supervised representations [W1; Q1]
We appreciate the reviewer’s pointers to these closely related works. ReAP uses failure supervision to shape the representation space before learning action-conditioned transitions, making the prediction target directly relevant to failure monitoring.
Liu et al. (2024) [1] jointly train a policy and latent dynamics model, then train a failure classifier on the frozen latent space using pre-intervention states as failure labels. Sirius-Fleet [2] pretrains a visual world model through image reconstruction and future-embedding prediction, then builds task-specific failure classifiers and OOD detectors on its frozen representations. Its failure labels also come from human intervention records.
ReAP first trains the detector representation using episode-level success/failure outcomes, then freezes the detector and trains the predictor with latent-transition and future-risk objectives. This design directs prediction toward failure-relevant features and uses the same risk head to assess observed and predicted states. It enables monitoring of an existing policy using rollout outcomes, without requiring intervention annotations or joint policy training. The learned/random/PCA comparison in Table 3(a) supports the value of this failure-supervised prediction space.
Foresight learns predictions in a video-pretrained representation space and trains a downstream temporal failure detector. Section 2 and Appendix A discuss these distinctions. In our additional π0/LIBERO comparison, ReAP OR achieves higher bTWA than our Foresight reproduction (0.780 vs. 0.730) and greater before-onset coverage with fewer false alarms. These results support ReAP’s effectiveness in both failure monitoring and advance warning. Full results and reproduction details appear in Item 5, “Predictive monitoring baselines,” of our response to Reviewer JvV7.
[1] Huihan Liu et al. Model-Based Runtime Monitoring with Interactive Imitation Learning. ICRA, 2024. https://arxiv.org/abs/2310.17552
[2] Huihan Liu et al. Multi-Task Interactive Robot Fleet Learning with Visual World Models. CoRL 2024, PMLR 270, 2025. https://proceedings.mlr.press/v270/liu25g.html
2. Predictor contribution [W2]
Our central idea is to combine current-state recognition and action-conditioned prediction to improve the complete monitor. Table 1 shows that ReAP OR achieves higher bTWA and earlier detection than either branch alone across all four settings. For example, on OpenVLA, OR raises bTWA from 0.671 to 0.701 over the detector and reduces T-det from 0.450 to 0.326.
Appendix D.5 explains this complementarity. At the operating point reported in Table 17, the predictor detects 17 OpenVLA failures missed by the detector, while the detector detects four missed by the predictor. OR retains both sets of warnings. Figure 7 illustrates how prediction can anticipate failure before clear visual cues emerge, while recognition captures failures missed by prediction. Table 16 further shows that OR achieves higher bTWA and earlier detection than AND across all four settings. These results demonstrate the benefit of combining the two branches.
3. Training objectives [Q2]
Figure 3(b) compares latent-only, risk-only, and joint training on OpenVLA/LIBERO. We provide the corresponding metrics below and extend the comparison to π0/LIBERO.
We additionally compared the three objectives for the predictor on π0/LIBERO.
Joint training achieves the highest bTWA and bACC and the lowest T-det on both policies, supporting the complementary roles of latent-transition and future-risk supervision.