- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CALM, a runtime monitor for detecting instruction–execution mismatch in frozen VLA policies. It uses a Bayesian spatial readout to predict execution endpoints and applies conformal calibration to decide when to request assistance. Experiments on LIBERO-10 cover π0.5, π0, and OpenVLA.
Critical strengths
- An important problem.
The paper considers a meaningful failure mode: a VLA may confidently execute a familiar action that does not follow the current instruction. I agree that action uncertainty alone may not detect this type of failure, and runtime monitoring is worth studying.
- Extensive comparisons within the evaluated setting.
The paper compares against many existing monitoring methods and reports strong results on LIBERO-10. The matched-state instruction analysis and episode-level false-alarm calibration are also useful parts of the evaluation.
Critical weaknesses
- I am not convinced that the experiments establish genuine instruction–execution mismatch detection.
CALM relies on the concentration of a predicted spatial distribution, but this is not necessarily a measure of whether the robot's behavior matches the instruction. A policy could confidently execute the wrong goal while still producing a concentrated spatial prediction.
The matched-state goal-replacement experiment shows that CALM responds to changed instructions, which is useful. However, it does not rule out sensitivity to unfamiliar instructions as an alternative explanation. For example, how would the monitor behave when given a novel but valid instruction that the VLA follows correctly? This distinction matters because the paper's main claim is about detecting incorrect execution, not simply unfamiliar inputs.
The evaluation also combines ordinary execution failures, wrong-goal execution, absent targets, and ambiguous instructions into its overall detection metric. These are all reasonable reasons to request help, but they do not necessarily test the same failure mechanism.
- The evaluation is too narrow for the claimed applicability.
All experiments are limited to LIBERO-10. Testing three VLA backbones provides some evidence across policy architectures, but it does not establish generalization to new tasks, environments, or instruction distributions.
I am particularly concerned by the lack of real-robot experiments. For a runtime intervention method, it is important to understand how observation noise, execution delays, and deployment-time distribution shifts affect both detection and calibration. None of this is evaluated in a physical setting.
I also question how much the full spatial scoring mechanism contributes. The abstention-only ablation performs nearly as well as CALM, with slightly higher mean time-weighted accuracy. The paper would benefit from showing where the spatial concentration score provides a clear advantage.
Questions for authors
-
How can the authors distinguish instruction–execution mismatch from unfamiliarity with the instruction? Could they test novel but correctly executed instructions or semantically equivalent paraphrases?
-
How does CALM perform beyond LIBERO-10? Is there evidence of generalization to unseen tasks or physical robotic environments?
-
In which failure cases does the spatial concentration score provide a clear advantage over the abstention-only score?
5 comments
Gentle Follow-up to Reviewer ifKD on Our Rebuttal
Dear Reviewer ifKD,
Thank you again for your review. As the discussion period closes on November 18, we would like to briefly summarize our response and ask whether it addresses your concerns.
- Mismatch versus instruction novelty. CALM detects confident wrong-goal execution in the matched-state experiments. We additionally evaluated both novel but correctly executed instructions and semantically equivalent paraphrases, with alarm rates remaining within the false-positive budget. CALM continues to detect goal replacements, absent targets, and ambiguous instructions, distinguishing execution mismatch from instruction novelty.
- Generalization beyond LIBERO-10. Our leave-one-task-out experiments demonstrate zero-shot detection on unseen tasks, with further improvements from recalibration using a small number of successful target-task rollouts (full results in our response to Reviewer W3fq, W1).
- Spatial concentration versus abstention only. The spatial concentration score improves execution-failure detection over abstention alone and substantially reduces false alarms on correctly executed novel and paraphrased instructions. The detailed comparisons are provided in our response to W2 + Q3.
- Real-robot evaluation. We further evaluated CALM on a physical robot, where it remains within the false-positive budget and achieves the highest TPR, balanced accuracy, and time-weighted accuracy, with the earliest detection among all compared methods. The full results are provided in our November 11 follow-up, "Rebuttal to ifKD (W2 + Q2): Evaluation in a Physical Robotic Environment."
We hope these results address your central concerns about genuine instruction–execution mismatch detection, generalization beyond the original setting, and the contribution of spatial concentration. We would appreciate your feedback on whether these concerns have been resolved or if any further clarification would be helpful. If our responses have addressed your concerns, we would be grateful if you could reconsider your rating.
Thank you for your detailed results! I decide to raise my rating to 2.
Rebuttal to ifKD (W2 + Q2): Evaluation in a Physical Robotic Environment
As promised in our response to W2 + Q2, we report results in a physical robotic environment. The table reports mean ± standard deviation over three seeds at α=0.15.
Setup. We deploy CALM on a real myPalletizer 260 arm with a single third-person camera. The monitored policy is π0.5 fine-tuned on demonstrations collected on this robot, and the task is to reach the instructed object among several objects on the table. We collect 270 real-robot rollouts spanning the same four conditions as in the paper: original instructions, goal replacements, absent targets, and ambiguous instructions. All methods are trained as in Appendix E and calibrated with CALM's rule (Eq. 10) on the same 20 successful real-robot rollouts.
Results.
With real observation noise and execution delays, CALM stays within the false-positive budget and achieves the highest TPR, balanced accuracy, and time-weighted accuracy, with the earliest detection among all methods.
We will add these results to the main text of the revised paper, with the full setup and photos of the platform in the appendix and a demo video in the supplementary material.