acceptodds
Withdrawn from ICLR 2027

Rollout-Level Calibration for Failure Monitoring in Generative Robot Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Failure monitors for generative robot policies must raise timely alarms while limiting false positives on executions that would otherwise succeed. We introduce RoLCA (Rollout-Level Conformal Alerting), a modular layer that aggregates progress-matched score evidence over time and across channels, then split-conformally calibrates its maximum over eligible monitoring times using successful rollouts only. Under exchangeability of successful rollouts, RoLCA provides finite-sample rollout-level false-positive control for causal scoring, aggregation, fusion, and monitoring rules fixed independently of threshold calibration. The guarantee is marginal over the calibration draw and the next successful rollout, without requiring within-rollout independence. Across 30,502 rollouts in six task–backbone cells, calibrated monitors operate near the nominal false-positive rate in two sealed confirmations, but detection gains depend on the monitoring convention. On the second confirmation, a post-hoc comparison yields a 7.6-percentage-point paired recall gain over matched running-average OR under fixed-horizon monitoring; this advantage is not established after stop-at-success recalibration. Fusion improves early recall over a rollout-trained single channel but does not consistently outperform execution consistency alone. A clock-only baseline detects all timeout failures in this population under stop-at-success evaluation, motivating assessment of early warning beyond eventual coverage. These results separate false-alarm validity from detection power and underscore the need for explicit monitoring intervals, matched comparators, and timing baselines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot failure monitoring, conformal prediction, anytime-valid detection, generative robot policies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zrUJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies failure monitoring for generative robot policies when thresholds are calibrated from successful rollouts only. It proposes RoLCA, which applies split conformal calibration to the maximum of a monitoring statistic over eligible times. This gives finite-sample rollout-level false-positive control, with no need for within-rollout independence. On 30,502 robomimic rollouts, calibrated monitors reach the nominal false-positive rate. The paper also shows that detection gains depend on the monitoring convention, and that a clock-only baseline catches all timeout failures.

Critical strengths
  • The guarantee is clearly stated, with its assumptions and limits. Experiments confirm FPR control across six cells and two sealed confirmations.
  • The experimental protocol is rigorous. Candidates were frozen before opening seeds, post-hoc analyses are labeled, and non-replicating results (flow-inversion) are reported openly.
  • The paper reveals evaluation pitfalls that matter for the field. The clock-only baseline shows eventual recall can be misleading, and the (E)/(S) comparison shows that monitoring conventions change detector rankings.
  • Limitations are discussed in detail, which makes the claims easy to trust.
Critical weaknesses
  1. Limited novelty. Rollout-maximum split-conformal calibration is established in FAIL-Detect and related work, and Proposition 1 is the standard order-statistic argument. The remaining contribution is statistical design, whose benefit is not established.
  2. Detection gains are not robust. Flow-inversion does not replicate, three-channel fusion fails its per-cell criterion, and the advantage over matched OR (+0.076) appears only under (E) and is post hoc. Under (S), there is no established advantage over execution alone.
  3. The scope is narrow. Only three robomimic tasks, one actor family, and two backbones are used; all failures are timeouts, and no real-robot data is used. Early-recall results do not show detection before true failure onset.
  4. The presentation is hard to follow. Many conventions, comparators, and post-hoc labels are mixed in the main text, and the key takeaways for practitioners are unclear.
Questions for authors
  1. Beyond the rollout-maximum split-conformal calibration inherited from FAIL-Detect, what is the concrete new contribution, and can you isolate its benefit against a FAIL-Detect-style baseline (one statistic, same calibration)?
  2. The +0.076 gain over matched OR is post hoc and holds only under fixed-horizon (E); under stop-at-success (S) no advantage over execution alone or matched OR is established. Is there a confirmatory, or (S)-convention, comparison in which the proposed statistic beats these baselines?
  3. All evaluated failures are timeouts, so early recall measures warning before timeout, not before validated failure onset, and the study covers three robomimic tasks in one actor family. Do the findings hold for non-timeout failures, on real-robot data or another policy family, and before independently validated failure onset?
  4. For a practitioner, which single monitor and monitoring convention should be deployed, and in brief, when does channel fusion help over execution consistency alone?
Reviewer_jCK2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes RoLCA, a modular layer for runtime failure monitoring of generative robot policies. RoLCA converts per-cycle monitoring scores into rank-based evidence, aggregates this evidence over time and across score channels, and calibrates the maximum of the resulting statistic over a rollout with split conformal prediction on successful rollouts only. Under exchangeability of successful rollouts, this gives finite-sample rollout-level false-positive control. The method is evaluated on 30,502 simulated rollouts of rectified-flow policies on three robomimic tasks. The calibrated monitors operate near the nominal false-positive rate, while detection gains depend on whether monitoring stops at success.

Critical strengths
  • The empirical protocol is careful and transparent. For both confirmation studies, the candidate monitors and decision criteria were fixed before the evaluation seeds were opened; the study comprises 30,502 rollouts evaluated over 50 shared splits, and negative and post-hoc results are reported openly, e.g., that the flow-inversion gain from the discovery study does not replicate (Section 4.5). This gives an honest picture of what does and does not work.
  • The rollout-level false-alarm control works as intended. The calibrated monitors operate near the nominal FPR of 0.10, with macro FPRs between 0.089 and 0.093 on the first confirmation, whereas the uncalibrated product/Ville diagnostic reaches about 0.26. This is a good result because reliable control of false alarms on successful rollouts is a central requirement for using such monitors in practice.
Critical weaknesses
  • The experimental evaluation is narrow. The main study only covers three robomimic tasks (lift, can, and square), and all six actors are comparatively small rectified-flow chunk policies of a single family that differ only in the task and in their visual-feature backbone (BC-RNN or Diffusion Policy). There is no other simulation benchmark, no hardware experiment, and no larger state-of-the-art policy such as a VLA or a world-action model (WAM), although the introduction motivates the work with flow-based VLAs like π0 (Black et al., 2025a) and π0.5 (Black et al., 2025b). The authors acknowledge this in Section 5, but because all claims about detection power are empirical, it remains unclear whether the findings generalize to the policies motivating the paper.
  • The structure and the writing make the paper very hard to follow. The introduction already contains very specific implementation details and numbers, for example, the FPR of approximately 0.26 for a product of per-cycle evidence with a Ville threshold, the +0.076 paired gain over the matched running-average OR under (E), and the FPR of 0.072 of an S-calibrated clock (lines 060-073). This is before RND-OE, the joint mean, the matched OR, or the clock are even explained in Sections 3.4, 4.1, and 4.3. Moreover, Section 4.5 concludes that the flow-inversion gain does not replicate because both prespecified G-C1 criteria fail, but these criteria and the underlying results are only given in Appendix D.1. The comparison with published monitors likewise gets only a few sentences in Section 4.5, with setup and results in Appendix E. Beyond the structure, the writing is very dense and hard to follow for a human reader. Section 4 involves four rollout populations and two monitoring conventions, Figure 2 and Table 1 alone compare eleven monitor variants, and almost every result comes with several qualifications (post hoc, pointwise, conditional on the success pool, not independent replications). As a result, the main message of the paper is difficult to extract.

References:

  • K. Black, N. Brown, D. Driess, et al., "π0: A vision-language-action flow model for general robot control", Robotics: Science and Systems, 2025.
  • K. Black, N. Brown, J. Darpinian, et al., "π0.5: A vision-language-action model with open-world generalization", Conference on Robot Learning, 2025.
Questions for authors

Clarification requests:

  • What are the main contributions of the paper? The introduction does not state them explicitly. It does state, though, that rollout-maximum calibration is already established in FAIL-Detect (Xu et al., 2025) and that RoLCA uses the same split-conformal principle, and Section 3.2 describes the proof of Proposition 1 as the standard split-conformal order-statistic argument. The empirical contribution is also unclear to me. Under stop-at-success monitoring, a clock-only baseline that uses only elapsed time reaches recall 1.000 at FPR 0.072, because all 676 failures in the second confirmation are timeouts (Section 4.3, Table 1). The benefit of the learned monitors in this population thus lies only in earlier warnings, and the joint mean detects 0.413 of the failures before the clock alarm, fewer than execution alone with 0.438. Could the authors clarify what is methodologically new compared to FAIL-Detect, and which empirical findings they consider the main takeaways?
  • Appendix E lists several deviations from the original implementations of the published monitors. For FIPER (Römer et al., 2025), the AND/OR rules use window length one and a demonstration-trained RND-OE, which the authors describe as choices of their evaluation, whereas the original method aggregates both scores over a sliding window and trains RND-OE on successful policy rollouts. The evaluated FIPER AND rule has an FPR of 0.000 and a recall difference of +0.459 compared to execution-inversion fusion; i.e., it detects almost no failures (Table 14b). In Table 13, the rollout-trained RND-OE reaches much higher recall than the demonstration-trained one (0.532 vs. 0.206). Could the authors explain these choices and report results for the original configuration, so that the comparison reflects the published method?

Suggestion for future work:

  • Evaluating RoLCA on another benchmark, on hardware, and with a larger pretrained policy such as a VLA or a WAM would show whether the findings hold beyond robomimic.
  • Restructuring the introduction around the problem, the main idea, and the contributions, moving key results such as the replication study into the main text, and reducing the number of monitor variants and qualifications there would greatly improve readability.

References:

  • R. Römer, A. Kobras, L. Worbis, and A. P. Schoellig, "Failure prediction at runtime for generative robot policies", Advances in Neural Information Processing Systems, 2025.
  • C. Xu, T. K. Nguyen, E. Dixon, C. Rodriguez, P. Miller, R. Lee, P. Shah, R. Ambrus, H. Nishimura, and M. Itkina, "Can we detect failures without failure data? Uncertainty-aware runtime failure detection for imitation learning policies", Robotics: Science and Systems, 2025.
Reviewer_bcfzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies rollout-level false-alarm control for generative robot policies. RoLCA combines progress-matched scores, temporal aggregation, and channel fusion, then calibrates the maximum monitoring statistic using successful rollouts. Under exchangeability, this provides a finite-sample false-positive guarantee over the specified monitoring interval. Experiments distinguish fixed-horizon from stop-at-success monitoring and show that valid false-alarm control does not necessarily translate into improved failure detection.

Critical strengths
  • The false-alarm guarantee is clearly stated and appropriately qualified. Calibrating the complete rollout statistic avoids assuming independence between observations within a trajectory, and the paper carefully distinguishes marginal validity from calibration-conditional control.

  • The analysis of monitoring conventions is useful. Showing that detector rankings change when monitoring stops at success, together with the clock-only baseline, highlights evaluation choices that can materially affect conclusions about failure monitors.

Critical weaknesses
  • The methodological advance over existing rollout-maximum conformal monitors appears limited. The main guarantee follows the standard split-conformal argument, while the added value of progress matching and the rank-to-evidence transform is not cleanly isolated.

  • The proposed fusion does not show a consistent advantage over simpler monitors. Under stop-at-success monitoring, its full-horizon gains over execution consistency and matched OR are not established, and execution consistency performs better by step 100. This makes the practical benefit of the additional channels unclear.

  • The main early-warning evidence concerns timeout failures without independently validated failure onset. Alarming before the time budget expires does not establish that an intervention would still be useful. The timing analyses are also post hoc, so the practical early-warning claim would benefit from independent validation.

Questions for authors
  1. What do progress matching and the rank-to-evidence transform add when score channels, temporal aggregation, and rollout calibration are held fixed?

  2. Under stop-at-success monitoring, how do fusion and simpler monitors compare across recall–FPR operating points when their alarm opportunities are aligned?

  3. Can the early-warning results be evaluated against independently annotated failure onset or an actionable intervention deadline, to establish how much useful warning the monitors provide?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.