- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies failure monitoring for generative robot policies when thresholds are calibrated from successful rollouts only. It proposes RoLCA, which applies split conformal calibration to the maximum of a monitoring statistic over eligible times. This gives finite-sample rollout-level false-positive control, with no need for within-rollout independence. On 30,502 robomimic rollouts, calibrated monitors reach the nominal false-positive rate. The paper also shows that detection gains depend on the monitoring convention, and that a clock-only baseline catches all timeout failures.
Critical strengths
- The guarantee is clearly stated, with its assumptions and limits. Experiments confirm FPR control across six cells and two sealed confirmations.
- The experimental protocol is rigorous. Candidates were frozen before opening seeds, post-hoc analyses are labeled, and non-replicating results (flow-inversion) are reported openly.
- The paper reveals evaluation pitfalls that matter for the field. The clock-only baseline shows eventual recall can be misleading, and the (E)/(S) comparison shows that monitoring conventions change detector rankings.
- Limitations are discussed in detail, which makes the claims easy to trust.
Critical weaknesses
- Limited novelty. Rollout-maximum split-conformal calibration is established in FAIL-Detect and related work, and Proposition 1 is the standard order-statistic argument. The remaining contribution is statistical design, whose benefit is not established.
- Detection gains are not robust. Flow-inversion does not replicate, three-channel fusion fails its per-cell criterion, and the advantage over matched OR (+0.076) appears only under (E) and is post hoc. Under (S), there is no established advantage over execution alone.
- The scope is narrow. Only three robomimic tasks, one actor family, and two backbones are used; all failures are timeouts, and no real-robot data is used. Early-recall results do not show detection before true failure onset.
- The presentation is hard to follow. Many conventions, comparators, and post-hoc labels are mixed in the main text, and the key takeaways for practitioners are unclear.
Questions for authors
- Beyond the rollout-maximum split-conformal calibration inherited from FAIL-Detect, what is the concrete new contribution, and can you isolate its benefit against a FAIL-Detect-style baseline (one statistic, same calibration)?
- The +0.076 gain over matched OR is post hoc and holds only under fixed-horizon (E); under stop-at-success (S) no advantage over execution alone or matched OR is established. Is there a confirmatory, or (S)-convention, comparison in which the proposed statistic beats these baselines?
- All evaluated failures are timeouts, so early recall measures warning before timeout, not before validated failure onset, and the study covers three robomimic tasks in one actor family. Do the findings hold for non-timeout failures, on real-robot data or another policy family, and before independently validated failure onset?
- For a practitioner, which single monitor and monitoring convention should be deployed, and in brief, when does channel fusion help over execution consistency alone?