acceptodds
Under review as a conference paper at ICLR 2027

Calibrated Monitoring of Instruction–Execution Mismatch in Vision-Language-Action Policies

Abstract

Vision–language–action (VLA) policies can execute familiar motions while failing to follow the current instruction. Such instruction–execution mismatch can provide weak signals for monitors based on action uncertainty or novelty. To address this, we introduce **CALM**, a **CAL**ibrated **M**onitor that decodes instruction-conditioned spatial support from frozen VLA representations. Specifically, CALM constructs a Bayesian last layer (BLL) model to predict a distribution over end-effector locations together with an abstention output. After scalable variational training on successful execution endpoints and absent-target examples, we accumulate spatial support deficits from dispersed or abstaining predictions based on the trained BLL model at runtime and request assistance when the cumulative score exceeds a calibrated threshold. To obtain this threshold, we rely on split conformal prediction (CP), which entails held-out calibration episodes. Under episode-level exchangeability, split CP provides a finite-sample bound on the marginal probability of interrupting an otherwise successful episode. On LIBERO-10 with , , and OpenVLA, CALM achieves the highest detection rate, balanced accuracy, and time-weighted accuracy among evaluated methods within the empirical false-positive-rate budget.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-language-action models, Robot manipulation, Runtime monitoring, Instruction-execution mismatch, Uncertainty quantification, Conformal prediction, Bayesian prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ifKDOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CALM, a runtime monitor for detecting instruction–execution mismatch in frozen VLA policies. It uses a Bayesian spatial readout to predict execution endpoints and applies conformal calibration to decide when to request assistance. Experiments on LIBERO-10 cover π0.5, π0, and OpenVLA.

Critical strengths
  1. An important problem.

The paper considers a meaningful failure mode: a VLA may confidently execute a familiar action that does not follow the current instruction. I agree that action uncertainty alone may not detect this type of failure, and runtime monitoring is worth studying.

  1. Extensive comparisons within the evaluated setting.

The paper compares against many existing monitoring methods and reports strong results on LIBERO-10. The matched-state instruction analysis and episode-level false-alarm calibration are also useful parts of the evaluation.

Critical weaknesses
  1. I am not convinced that the experiments establish genuine instruction–execution mismatch detection.

CALM relies on the concentration of a predicted spatial distribution, but this is not necessarily a measure of whether the robot's behavior matches the instruction. A policy could confidently execute the wrong goal while still producing a concentrated spatial prediction.

The matched-state goal-replacement experiment shows that CALM responds to changed instructions, which is useful. However, it does not rule out sensitivity to unfamiliar instructions as an alternative explanation. For example, how would the monitor behave when given a novel but valid instruction that the VLA follows correctly? This distinction matters because the paper's main claim is about detecting incorrect execution, not simply unfamiliar inputs.

The evaluation also combines ordinary execution failures, wrong-goal execution, absent targets, and ambiguous instructions into its overall detection metric. These are all reasonable reasons to request help, but they do not necessarily test the same failure mechanism.

  1. The evaluation is too narrow for the claimed applicability.

All experiments are limited to LIBERO-10. Testing three VLA backbones provides some evidence across policy architectures, but it does not establish generalization to new tasks, environments, or instruction distributions.

I am particularly concerned by the lack of real-robot experiments. For a runtime intervention method, it is important to understand how observation noise, execution delays, and deployment-time distribution shifts affect both detection and calibration. None of this is evaluated in a physical setting.

I also question how much the full spatial scoring mechanism contributes. The abstention-only ablation performs nearly as well as CALM, with slightly higher mean time-weighted accuracy. The paper would benefit from showing where the spatial concentration score provides a clear advantage.

Questions for authors
  1. How can the authors distinguish instruction–execution mismatch from unfamiliarity with the instruction? Could they test novel but correctly executed instructions or semantically equivalent paraphrases?

  2. How does CALM perform beyond LIBERO-10? Is there evidence of generalization to unseen tasks or physical robotic environments?

  3. In which failure cases does the spatial concentration score provide a clear advantage over the abstention-only score?

5 comments
AuthorsOpenReview ↗

Rebuttal to ifKD (W2 + Q2): Evaluation in a Physical Robotic Environment

As promised in our response to W2 + Q2, we report results in a physical robotic environment. The table reports mean ± standard deviation over three seeds at α=0.15\alpha = 0.15.

Setup. We deploy CALM on a real myPalletizer 260 arm with a single third-person camera. The monitored policy is π0.5\pi_{0.5} fine-tuned on demonstrations collected on this robot, and the task is to reach the instructed object among several objects on the table. We collect 270 real-robot rollouts spanning the same four conditions as in the paper: original instructions, goal replacements, absent targets, and ambiguous instructions. All methods are trained as in Appendix E and calibrated with CALM's rule (Eq. 10) on the same 20 successful real-robot rollouts.

Results.

MethodFPR ↓TPR ↑Bal. Acc. ↑TWA ↑Det. Time ↓
Within FPR budget (≤ 0.15)
CALM0.061 ± 0.0140.976 ± 0.0120.957 ± 0.0010.947 ± 0.0050.021 ± 0.013
Rel. Mahalanobis0.044 ± 0.0310.917 ± 0.0480.936 ± 0.0390.650 ± 0.0680.629 ± 0.099
RND-OE0.139 ± 0.0290.969 ± 0.0000.915 ± 0.0150.744 ± 0.0250.353 ± 0.024
Similarity0.140 ± 0.1000.927 ± 0.0370.893 ± 0.0320.708 ± 0.0380.399 ± 0.007
SAFE0.096 ± 0.0410.819 ± 0.0510.861 ± 0.0420.815 ± 0.0490.111 ± 0.099
Hide-and-Seek0.139 ± 0.0800.711 ± 0.2160.786 ± 0.0690.689 ± 0.0480.278 ± 0.042
KnowNo0.078 ± 0.0460.439 ± 0.1840.680 ± 0.0710.656 ± 0.0860.145 ± 0.146
CLIP0.138 ± 0.1020.411 ± 0.0490.637 ± 0.0310.431 ± 0.051--
UPS0.078 ± 0.0460.332 ± 0.1420.627 ± 0.0510.604 ± 0.0610.176 ± 0.158
CoVer0.096 ± 0.0410.232 ± 0.0520.568 ± 0.0240.507 ± 0.0190.517 ± 0.055
DiffDAgger0.113 ± 0.0320.233 ± 0.0740.560 ± 0.0220.530 ± 0.0140.255 ± 0.015
I-FailSense0.026 ± 0.0000.063 ± 0.0370.518 ± 0.0190.510 ± 0.0120.207 ± 0.183
GPLA0.113 ± 0.0930.104 ± 0.1020.496 ± 0.0090.484 ± 0.0100.235 ± 0.031
TC0.148 ± 0.1000.125 ± 0.0920.488 ± 0.0060.434 ± 0.0440.869 ± 0.013
RND-A0.148 ± 0.1070.119 ± 0.0370.485 ± 0.0660.439 ± 0.0490.758 ± 0.138
SEAL0.078 ± 0.0700.035 ± 0.0370.478 ± 0.0180.471 ± 0.0250.410 ± 0.004
FOREWARN0.113 ± 0.0850.010 ± 0.0000.449 ± 0.0430.447 ± 0.0420.296 ± 0.064
Above FPR budget (> 0.15)
Deep k-NN0.175 ± 0.0860.955 ± 0.0120.890 ± 0.0490.556 ± 0.0550.700 ± 0.028
FIPER0.192 ± 0.1310.948 ± 0.0270.878 ± 0.0560.700 ± 0.0340.376 ± 0.053
ACE0.174 ± 0.0920.460 ± 0.1630.643 ± 0.0380.563 ± 0.0160.351 ± 0.019
FM-accel0.165 ± 0.0160.394 ± 0.0240.614 ± 0.0060.588 ± 0.0070.134 ± 0.013
logpZO0.192 ± 0.0420.372 ± 0.2300.590 ± 0.0950.510 ± 0.0520.460 ± 0.078
VLA-FAIL0.225 ± 0.1660.384 ± 0.1970.580 ± 0.0460.533 ± 0.0430.231 ± 0.048
ActProbe0.165 ± 0.0370.139 ± 0.0060.487 ± 0.0210.437 ± 0.0180.717 ± 0.043

With real observation noise and execution delays, CALM stays within the false-positive budget and achieves the highest TPR, balanced accuracy, and time-weighted accuracy, with the earliest detection among all methods.

We will add these results to the main text of the revised paper, with the full setup and photos of the platform in the appendix and a demo video in the supplementary material.

AuthorsOpenReview ↗

Gentle Follow-up to Reviewer ifKD on Our Rebuttal

Dear Reviewer ifKD,

Thank you again for your review. As the discussion period closes on November 18, we would like to briefly summarize our response and ask whether it addresses your concerns.

  • Mismatch versus instruction novelty. CALM detects confident wrong-goal execution in the matched-state experiments. We additionally evaluated both novel but correctly executed instructions and semantically equivalent paraphrases, with alarm rates remaining within the false-positive budget. CALM continues to detect goal replacements, absent targets, and ambiguous instructions, distinguishing execution mismatch from instruction novelty.
  • Generalization beyond LIBERO-10. Our leave-one-task-out experiments demonstrate zero-shot detection on unseen tasks, with further improvements from recalibration using a small number of successful target-task rollouts (full results in our response to Reviewer W3fq, W1).
  • Spatial concentration versus abstention only. The spatial concentration score improves execution-failure detection over abstention alone and substantially reduces false alarms on correctly executed novel and paraphrased instructions. The detailed comparisons are provided in our response to W2 + Q3.
  • Real-robot evaluation. We further evaluated CALM on a physical robot, where it remains within the false-positive budget and achieves the highest TPR, balanced accuracy, and time-weighted accuracy, with the earliest detection among all compared methods. The full results are provided in our November 11 follow-up, "Rebuttal to ifKD (W2 + Q2): Evaluation in a Physical Robotic Environment."

We hope these results address your central concerns about genuine instruction–execution mismatch detection, generalization beyond the original setting, and the contribution of spatial concentration. We would appreciate your feedback on whether these concerns have been resolved or if any further clarification would be helpful. If our responses have addressed your concerns, we would be grateful if you could reconsider your rating.

Thank you for your detailed results! I decide to raise my rating to 2.

Reviewer_Mbc4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses failures where a robot performs familiar actions but does not follow its current instruction. CALM reads frozen VLA features through a Bayesian spatial predictor with an abstention output. Successful endpoints and absent-target examples supervise the predictor. A cumulative spatial score triggers assistance after crossing a threshold calibrated on successful episodes. The main strength is combining an instruction-sensitive signal with control of false interruptions over a complete episode. This guarantee is marginal and assumes exchangeability. The monitor also adds little measured runtime cost. However, false alarms on successfully followed new instructions remain untested, and baseline adaptations need clearer documentation. On LIBERO-10 with π0.5, CALM achieves a 0.970 detection rate and a 0.086 false-positive rate. Results also cover π0 and OpenVLA. These findings support the approach within the evaluated setting, while broader instruction generalization remains uncertain.

Critical strengths
  1. A relevant monitoring signal. Matched-state instruction changes affect spatial support more than the action-uncertainty proxy. This supports using existing VLA features to assess instruction compliance. It is a good idea.

  2. Theory aligned with deployment. The guarantee controls interruption anywhere in a successful episode. Its assumptions, marginal scope, and finite-calibration-set variation are explicitly addressed.

  3. Strong results with low measured overhead. Experiments cover three policies and several component ablations. Monitoring takes 1.80 ms alongside 84.1 ms of policy inference in the reported setup. The performance is ok.

Critical weaknesses
  1. The new-instruction control is incomplete. Replacement-instruction diagnostics use success labels for the original instruction. Separate false-alarm rates for correctly followed new instructions are needed to distinguish compliance monitoring from sensitivity to instruction changes.

  2. Baseline adaptations are underdocumented. Method-specific inputs, supervision, adaptation settings, and threshold rules are missing. Shared test episodes alone do not establish comparable monitoring conditions.

  3. Calibration terminology is too broad. The guarantee concerns episode-level alarms. Calling spatial predictions calibrated needs separate probability-calibration evidence, or narrower wording. This does not invalidate the alarm theorem.

Questions for authors

I still have three questions:

  1. What are the alarm rates for verified successful and failed executions of comparable new instructions?

  2. How were baseline inputs, supervision, adaptation, and thresholds configured? How do continuous-score methods compare at a common calibrated false-positive target?

  3. Does the calibration claim concern only alarms, or is predictive probability calibration also supported?

3 comments
AuthorsOpenReview ↗

Rebuttal to Mbc4: Real-Robot Results Beyond the Evaluated Setting

Beyond the simulation setting evaluated in the paper, we have also deployed CALM on a real robot. The table reports mean ± standard deviation over three seeds at α=0.15\alpha = 0.15.

Setup. We deploy CALM on a real myPalletizer 260 arm with a single third-person camera. The monitored policy is π0.5\pi_{0.5} fine-tuned on demonstrations collected on this robot, and the task is to reach the instructed object among several objects on the table. We collect 270 real-robot rollouts spanning the same four conditions as in the paper: original instructions, goal replacements, absent targets, and ambiguous instructions. All methods are trained as in Appendix E and calibrated with CALM's rule (Eq. 10) on the same 20 successful real-robot rollouts.

Results.

MethodFPR ↓TPR ↑Bal. Acc. ↑TWA ↑Det. Time ↓
Within FPR budget (≤ 0.15)
CALM0.061 ± 0.0140.976 ± 0.0120.957 ± 0.0010.947 ± 0.0050.021 ± 0.013
Rel. Mahalanobis0.044 ± 0.0310.917 ± 0.0480.936 ± 0.0390.650 ± 0.0680.629 ± 0.099
RND-OE0.139 ± 0.0290.969 ± 0.0000.915 ± 0.0150.744 ± 0.0250.353 ± 0.024
Similarity0.140 ± 0.1000.927 ± 0.0370.893 ± 0.0320.708 ± 0.0380.399 ± 0.007
SAFE0.096 ± 0.0410.819 ± 0.0510.861 ± 0.0420.815 ± 0.0490.111 ± 0.099
Hide-and-Seek0.139 ± 0.0800.711 ± 0.2160.786 ± 0.0690.689 ± 0.0480.278 ± 0.042
KnowNo0.078 ± 0.0460.439 ± 0.1840.680 ± 0.0710.656 ± 0.0860.145 ± 0.146
CLIP0.138 ± 0.1020.411 ± 0.0490.637 ± 0.0310.431 ± 0.051--
UPS0.078 ± 0.0460.332 ± 0.1420.627 ± 0.0510.604 ± 0.0610.176 ± 0.158
CoVer0.096 ± 0.0410.232 ± 0.0520.568 ± 0.0240.507 ± 0.0190.517 ± 0.055
DiffDAgger0.113 ± 0.0320.233 ± 0.0740.560 ± 0.0220.530 ± 0.0140.255 ± 0.015
I-FailSense0.026 ± 0.0000.063 ± 0.0370.518 ± 0.0190.510 ± 0.0120.207 ± 0.183
GPLA0.113 ± 0.0930.104 ± 0.1020.496 ± 0.0090.484 ± 0.0100.235 ± 0.031
TC0.148 ± 0.1000.125 ± 0.0920.488 ± 0.0060.434 ± 0.0440.869 ± 0.013
RND-A0.148 ± 0.1070.119 ± 0.0370.485 ± 0.0660.439 ± 0.0490.758 ± 0.138
SEAL0.078 ± 0.0700.035 ± 0.0370.478 ± 0.0180.471 ± 0.0250.410 ± 0.004
FOREWARN0.113 ± 0.0850.010 ± 0.0000.449 ± 0.0430.447 ± 0.0420.296 ± 0.064
Above FPR budget (> 0.15)
Deep k-NN0.175 ± 0.0860.955 ± 0.0120.890 ± 0.0490.556 ± 0.0550.700 ± 0.028
FIPER0.192 ± 0.1310.948 ± 0.0270.878 ± 0.0560.700 ± 0.0340.376 ± 0.053
ACE0.174 ± 0.0920.460 ± 0.1630.643 ± 0.0380.563 ± 0.0160.351 ± 0.019
FM-accel0.165 ± 0.0160.394 ± 0.0240.614 ± 0.0060.588 ± 0.0070.134 ± 0.013
logpZO0.192 ± 0.0420.372 ± 0.2300.590 ± 0.0950.510 ± 0.0520.460 ± 0.078
VLA-FAIL0.225 ± 0.1660.384 ± 0.1970.580 ± 0.0460.533 ± 0.0430.231 ± 0.048
ActProbe0.165 ± 0.0370.139 ± 0.0060.487 ± 0.0210.437 ± 0.0180.717 ± 0.043

On the real robot, CALM stays within the false-positive budget and achieves the highest TPR, balanced accuracy, and time-weighted accuracy, with the earliest detection among all methods.

We will add these results to the main text of the revised paper, with the full setup and photos of the platform in the appendix and a demo video in the supplementary material.

Thank you for the detailed response. I will keep my original positive rating.

AuthorsOpenReview ↗

Thank You for Your Feedback (To Reviewer Mbc4)

Dear Reviewer Mbc4,

Thank you for your positive assessment and for taking the time to consider our rebuttal and additional real-robot results. We appreciate your feedback and support.

Reviewer_W3fqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • CALM learns a Bayesian spatial readout from frozen VLA features and accumulates spatial uncertainty and abstention evidence. Episode-level conformal calibration controls marginal false interruptions on exchangeable successful rollouts.
Critical strengths
  • Detecting instruction-following failures during otherwise confident execution is an important deployment problem. Extracting this signal from existing VLA features is a useful direction for follow-up research.

  • Results across three backbones are promising, with extensive baseline comparisons and only 1.8 ms additional monitoring latency.

Critical weaknesses
  • Generalization to unseen tasks remains unclear. Training and evaluation cover the same ten tasks, so transfer to unseen instruction conditions does not establish cross-task generalization. A held-out-task evaluation should distinguish zero-shot transfer, recalibration using successful target-task rollouts, and readout retraining, while reporting their data requirements.
  • Mismatch detection is not sufficiently separated from instruction novelty. The score measures spatial concentration rather than explicitly comparing intended and executed behavior. Successful goal replacements and benign paraphrases are essential negative controls: the matched-state experiment establishes instruction sensitivity, but not whether CALM accepts unfamiliar instructions that the policy follows correctly.
  • The contribution of each component remains unclear. Which component contributes most to the performance gains?
  • Experimental validation is limited to simulation. Evaluation covers only LIBERO-10, with no real-robot experiments to establish robustness under realistic perception noise, calibration errors, and execution variability.
Questions for authors

See section above.

3 comments
AuthorsOpenReview ↗

Rebuttal to W3fq (W4): Real-Robot Evaluation

As promised in our response to W4, we report results on a real robot. The table reports mean ± standard deviation over three seeds at α=0.15\alpha = 0.15.

Setup. We deploy CALM on a real myPalletizer 260 arm with a single third-person camera. The monitored policy is π0.5\pi_{0.5} fine-tuned on demonstrations collected on this robot, and the task is to reach the instructed object among several objects on the table. We collect 270 real-robot rollouts spanning the same four conditions as in the paper: original instructions, goal replacements, absent targets, and ambiguous instructions. All methods are trained as in Appendix E and calibrated with CALM's rule (Eq. 10) on the same 20 successful real-robot rollouts.

Results.

MethodFPR ↓TPR ↑Bal. Acc. ↑TWA ↑Det. Time ↓
Within FPR budget (≤ 0.15)
CALM0.061 ± 0.0140.976 ± 0.0120.957 ± 0.0010.947 ± 0.0050.021 ± 0.013
Rel. Mahalanobis0.044 ± 0.0310.917 ± 0.0480.936 ± 0.0390.650 ± 0.0680.629 ± 0.099
RND-OE0.139 ± 0.0290.969 ± 0.0000.915 ± 0.0150.744 ± 0.0250.353 ± 0.024
Similarity0.140 ± 0.1000.927 ± 0.0370.893 ± 0.0320.708 ± 0.0380.399 ± 0.007
SAFE0.096 ± 0.0410.819 ± 0.0510.861 ± 0.0420.815 ± 0.0490.111 ± 0.099
Hide-and-Seek0.139 ± 0.0800.711 ± 0.2160.786 ± 0.0690.689 ± 0.0480.278 ± 0.042
KnowNo0.078 ± 0.0460.439 ± 0.1840.680 ± 0.0710.656 ± 0.0860.145 ± 0.146
CLIP0.138 ± 0.1020.411 ± 0.0490.637 ± 0.0310.431 ± 0.051--
UPS0.078 ± 0.0460.332 ± 0.1420.627 ± 0.0510.604 ± 0.0610.176 ± 0.158
CoVer0.096 ± 0.0410.232 ± 0.0520.568 ± 0.0240.507 ± 0.0190.517 ± 0.055
DiffDAgger0.113 ± 0.0320.233 ± 0.0740.560 ± 0.0220.530 ± 0.0140.255 ± 0.015
I-FailSense0.026 ± 0.0000.063 ± 0.0370.518 ± 0.0190.510 ± 0.0120.207 ± 0.183
GPLA0.113 ± 0.0930.104 ± 0.1020.496 ± 0.0090.484 ± 0.0100.235 ± 0.031
TC0.148 ± 0.1000.125 ± 0.0920.488 ± 0.0060.434 ± 0.0440.869 ± 0.013
RND-A0.148 ± 0.1070.119 ± 0.0370.485 ± 0.0660.439 ± 0.0490.758 ± 0.138
SEAL0.078 ± 0.0700.035 ± 0.0370.478 ± 0.0180.471 ± 0.0250.410 ± 0.004
FOREWARN0.113 ± 0.0850.010 ± 0.0000.449 ± 0.0430.447 ± 0.0420.296 ± 0.064
Above FPR budget (> 0.15)
Deep k-NN0.175 ± 0.0860.955 ± 0.0120.890 ± 0.0490.556 ± 0.0550.700 ± 0.028
FIPER0.192 ± 0.1310.948 ± 0.0270.878 ± 0.0560.700 ± 0.0340.376 ± 0.053
ACE0.174 ± 0.0920.460 ± 0.1630.643 ± 0.0380.563 ± 0.0160.351 ± 0.019
FM-accel0.165 ± 0.0160.394 ± 0.0240.614 ± 0.0060.588 ± 0.0070.134 ± 0.013
logpZO0.192 ± 0.0420.372 ± 0.2300.590 ± 0.0950.510 ± 0.0520.460 ± 0.078
VLA-FAIL0.225 ± 0.1660.384 ± 0.1970.580 ± 0.0460.533 ± 0.0430.231 ± 0.048
ActProbe0.165 ± 0.0370.139 ± 0.0060.487 ± 0.0210.437 ± 0.0180.717 ± 0.043

Under real perception noise and execution variability, CALM stays within the false-positive budget and achieves the highest TPR, balanced accuracy, and time-weighted accuracy, with the earliest detection among all methods.

We will add these results to the main text of the revised paper, with the full setup and photos of the platform in the appendix and a demo video in the supplementary material.

AuthorsOpenReview ↗

Gentle Follow-up to Reviewer W3fq on Our Rebuttal

Dear Reviewer W3fq,

Thank you again for your review. As the discussion period closes on November 18, we would like to briefly summarize our response and ask whether it addresses your concerns.

  • Generalization to unseen tasks. Our leave-one-task-out evaluation distinguishes zero-shot transfer, recalibration, and readout retraining, together with their data requirements. CALM detects most episodes requiring assistance without any target-task data, and recalibration using only 15 successful target-task rollouts improves every metric without retraining or failure labels.
  • Mismatch versus instruction novelty. We evaluated both benign paraphrases and novel but correctly executed instructions. Their alarm rates remain within the false-positive budget, while CALM continues to detect goal replacements, absent targets, and ambiguous instructions. These results demonstrate that CALM responds to instruction–execution mismatch rather than instruction novelty.
  • Component contributions. The ablations in Table 7 identify the patch grid and abstention output as the largest contributors. The Bayesian last layer further improves all metrics and stabilizes the false-positive rate across seeds.
  • Real-robot evaluation. We additionally evaluated CALM on a physical robot, where it remains within the false-positive budget and achieves the highest TPR, balanced accuracy, and time-weighted accuracy, with the earliest detection among all compared methods. The complete setup and comparison table are provided in our November 11 follow-up, "Rebuttal to W3fq (W4): Real-Robot Evaluation."

We hope these additional experiments address your concerns about generalization, instruction novelty, component contributions, and real-world validation. We would greatly appreciate any remaining questions or feedback. If these results have resolved your concerns, we would be grateful if you could reconsider your rating before the discussion period ends.

AuthorsOpenReview ↗

Final Reminder Before the Discussion Period Closes

Dear Reviewer W3fq,

As the discussion period closes today, we would like to kindly follow up once more. Each point in your review is addressed with new results:

  • Unseen tasks. The leave-one-task-out evaluation separates zero-shot transfer, recalibration, and readout retraining, together with their data requirements.
  • Instruction novelty. On paraphrased and novel but correctly executed instructions, the alarm rate stays within the false-positive budget, while CALM still detects every goal replacement, absent target, and ambiguous instruction.
  • Component contributions. The ablation in Table 7 identifies the patch grid and the abstention output as the main contributors.
  • Real robot. On a physical robot, CALM stays within the false-positive budget and achieves the best detection performance among all methods.

We would be grateful if you could take these results into account in your final rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

AC Rebuttal Summary: Reviewer ifKD Raised the Rating after the Real-Robot Results, and Reviewer Mbc4 Kept the Positive Rating

Dear Area Chair,

We thank you and the reviewers for the constructive discussion. We summarize the discussion with each reviewer below.

Reviewer Mbc4 (Rating 3: Weak Acceptance)

Reviewer Mbc4 finds that "the main strength is combining an instruction-sensitive signal with control of false interruptions over a complete episode", describes the spatial signal as "a relevant monitoring signal" and "a good idea", and recognizes "theory aligned with deployment" and "strong results with low measured overhead." After reading our response and the real-robot results, the reviewer wrote: "Thank you for the detailed response. I will keep my original positive rating."

Alarm rates on new instructions (W1 + Q1). On correctly executed new instructions, CALM keeps the alarm rate within the false-positive budget while flagging nearly all failed executions. See our comment "Rebuttal to Mbc4", W1 + Q1.

Baselines at a common calibrated target (W2 + Q2). We documented the inputs, supervision, adaptation, and thresholds of every baseline. When every continuous-score method is calibrated with CALM's rule, CALM still achieves the highest TPR, balanced accuracy, and time-weighted accuracy. See our comment "Rebuttal to Mbc4", W2 + Q2.

Reviewer ifKD (Rating 2: Weak Rejection, Raised from 1)

Reviewer ifKD finds that the paper "considers a meaningful failure mode" and "compares against many existing monitoring methods and reports strong results on LIBERO-10". The reviewer writes "I agree that action uncertainty alone may not detect this type of failure, and runtime monitoring is worth studying", and notes that "the matched-state instruction analysis and episode-level false-alarm calibration are also useful parts of the evaluation." After our real-robot results, the reviewer wrote: "Thank you for your detailed results! I decide to raise my rating to 2."

Mismatch versus instruction novelty (W1 + Q1). CALM detects confident wrong-goal execution. On novel but correctly executed instructions and on paraphrases, its alarm rate stays within the false-positive budget, while CALM still detects every goal replacement, absent target, and ambiguous instruction. See our comment "Rebuttal to ifKD", W1 + Q1.

Generalization and physical environments (W2 + Q2). On held-out tasks, CALM detects most episodes requiring assistance without target-task data, and recalibration on a few successful rollouts improves every metric. On a real robot, CALM stays within the false-positive budget and achieves the best detection performance among all methods. See our comment "Rebuttal to ifKD (W2 + Q2): Evaluation in a Physical Robotic Environment".

Spatial concentration versus abstention only (W2 + Q3). The spatial score detects substantially more execution failures and keeps the alarm rate within the budget on correctly executed unfamiliar instructions, where the abstention-only score exceeds it. See our comment "Rebuttal to ifKD", W2 + Q3.

Reviewer W3fq (Rating 2: Weak Rejection)

Reviewer W3fq finds that "detecting instruction-following failures during otherwise confident execution is an important deployment problem", that "extracting this signal from existing VLA features is a useful direction for follow-up research", and that "results across three backbones are promising, with extensive baseline comparisons and only 1.8 ms additional monitoring latency." We have done our best to engage with the reviewer, addressing every point with new results and following up twice to invite the reviewer's feedback.

Unseen tasks (W1). A leave-one-task-out evaluation separates zero-shot transfer, recalibration, and readout retraining, with their data requirements. Without target-task data, CALM detects most episodes requiring assistance, and recalibration on a few successful rollouts, without failure labels or retraining, improves every metric. See our comment "Rebuttal to W3fq", W1.

Mismatch versus instruction novelty (W2). On both requested negative controls, benign paraphrases and novel but correctly executed instructions, the alarm rate stays within the false-positive budget, while every mismatch is still detected. See our comment "Rebuttal to W3fq", W2.

Component contributions (W3). The patch grid and the abstention output contribute most, and the Bayesian last layer improves every metric (Table 7). See our comment "Rebuttal to W3fq", W3.

Real-robot evaluation (W4). See our comment "Rebuttal to W3fq (W4): Real-Robot Evaluation".

We have revised the PDF accordingly, with all changes highlighted in blue, and added a demo video of the real-robot experiment to the supplementary material. To keep the table numbers cited in the discussion unchanged, each new table is numbered after the table it extends, e.g., Table 1 (Real robot).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.