- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors analyze the robustness of runtime failure detectors for VLA policies under distribution shifts. They evaluate failure detectors on two VLA policies, OpenVLA-OFT and π0, under seven types of environmental and instruction perturbations, with the main evaluation being a protocol that compares clean successes, perturbed successes, and perturbed failures to differentiate failure discrimination, threshold validity, and sensitivity to novelty. Results indicate that some detectors retain strong ranking performance but generate too many false positives, while others become less sensitive and miss failures. The authors further analyze temporal score accumulation, generalization to unseen tasks, and recalibration.
Critical strengths
- The paper asks that if robot policy becomes unreliable across those same changes, do failure detectors also become unreliable? Which I believe is an important and well throughout question Testing the monitor itself across distribution shifts is a relevant concern for deploying this system.
- Helpful difference of evaluation dimensions i.e comparing clean successes, shifted successes and shifted failures makes it possible to separate the sensitivity to the unfamiliar from the sensitivity to the failure.
Critical weaknesses
- The main insight that ranking performance could stay solid while a fixed threshold would become unstable under distribution shift isn't necessarily a big leap conceptually. Furthermore, the paper could better communicate what insights it contributes more generally through its three-distribution protocol, rather than just through its systematic empirical investigation of existing detectors.
- These experiments use independently calibrated timestep-wise quantiles rather than the form of the simultaneous functional calibration inspired by SAFE. These pointwise thresholds hint that we do not get control over false-alarms at the entire trajectory level, and some setups seem miscalibrated even in the absence of interference. Using measures of degradation relative to clean performance may help to identify further degradation, but does not demonstrate that the original calibration scheme would likely perform the same way.
- The bulk of evidence is from applying each policy to a simulated benchmark family with multiple severity cells but sparse outcome support. Using alarms offline without a stopping or recovery controller calls into question whether the simulated results can settle the question of whether the alarm behavior is likely to raise or lower physical risk. Additionally, repairs are selected ex-post based on coverage error, making the experiment's achievements diagnostic rather than representative of a deployment procedure that does not use outcome labels. 4.The related work could better cover studies on robot failure diagnosis and robustness under environmental changes. For example, The Colosseum [1] evaluates manipulation policies under a range of environmental perturbations, RoboMD [2] investigates conditions that make robot policies fail, and Colosseum V2 [3] examines VLA generalization across tasks and environments. Although these works focus on policy robustness rather than runtime detector calibration, discussing them would help place the paper’s findings in context and clarify its contribution.
References
- Pumacay, Wilbert, et al. "The colosseum: A benchmark for evaluating generalization for robotic manipulation." arXiv preprint arXiv:2402.08191 (2024).
- Sagar, Som, et al. "Robomd: Uncovering robot vulnerabilities through semantic potential fields." International Conference on Learning Representations. Vol. 2026. 2026.
- Morgan, Jeremy, et al. "Colosseum v2: Benchmarking generalization for vision language action models." arXiv preprint arXiv:2605.27759 (2026).
Questions for authors
- How would the outcomes differ if the original functional calibration described in SAFE were used rather than timestep-wise quantiles? Would the finding that thresholds become unreliable while detectors maintain strong ranking performance still hold?
- If the repair hyperparameters were chosen using a separate validation set rather than by minimizing coverage error on the evaluation set, would the observed trade-off between calibration and failure detection still hold?
- The primary analysis centers on OpenVLA-OFT and π0. Would the supplementary policy rollouts already collected be sufficient to demonstrate whether the same behavior of excessive false alarms and missed failures is exhibited by other policies?