- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors proposed calibration in the loop method for monitoring VLA failures, and their framework computes the calibration rule that set the deployed threshold inside the objective. Their experiments showed relatively high performance on LIBERO-Plus, LIBERO-Spatial and other datasets.
Critical strengths
Thanks the authors for submitting to ICLR 2027! I think monitoring VLA’s failure, alerting and improving this loop is important. The authors also contributed a concrete step to this problem. They proposed a calibration in the loop method that computes the calibration rule that set the deployed threshold inside the objective.
- Well-motivated problem and well-organized calibration in the loop method.
- Extensive experiments that show the effectiveness of their calibration in the loop method.
Critical weaknesses
I did not see critical weakness. There are a few minor comments that I think the authors could take to improve their paper.
(Minor) The VLA operation figure in the Figure 2 should be clearer.
The authors need to explain tests A and B in L346 (and also appeared in L362, etc.). I found the descriptions of tests A and B in the appendix but it is currently confusing in the main text.
I think the authors need to briefly clarify what statistic the false alarm’s definition is depending on L316-317, “a success is a false alarm the first time its statistics crosses the threshold”
I think the authors at least need to report the numbers in the main text and their conclusions. Most of the results of the current 5.2 seems to be pointing to the appendix, which is difficult to read.
Questions for authors
NA. See the weaknesses session and consider improve these minor points.
2 comments
Dear Reviewer oWBT,
We hope we’ve addressed all your concerns. Since the discussion period ends in <24 hours, please let us know if you need any further clarification. If no further clarifications are needed, may we kindly request that you revise your scores upwards?
Thank you, The Authors
We thank Reviewer oWBT for the encouraging assessment and the careful reading. We appreciate the reviewer’s recognition that the problem is well motivated and that calibration in the loop is a concrete step toward reliable VLA failure monitoring.
W1 — VLA panel of Figure 2
Reply: Panel (a) will be redrawn in the revision. It will show the frozen VLA, the context vector xt read from its internal state at each policy call, the per-call surprise st, its running mean sˉt, and the crossing of the deployed threshold τ that triggers the reset.
W2 — tests A and B
Reply: Tests A and B are two held-out test sets of π0.5 on LIBERO-Spatial, separate from the main test set used in Table 1, run at initial states 2–3 while the thresholds are calibrated at initial states 0–1. Test B is used for the recovery experiments of Sec. 5.4: on top of the LIBERO-Plus condition, each episode receives one transient disturbance that we add (the target object is pushed by 4–7 cm, or the gripper is forced open). Test A holds the same episodes without this disturbance, which tells how many of the failures in test B are rescuable by a reset. Sec. 5.1 of the revision will define both.
W3 — the statistic behind a false alarm
Reply: It is the episode statistic of Sec. 3, M(θ)=maxtsˉt(θ): the running mean sˉt of the per-call surprise st=softplus(θ⊤[xt;1]), maximized over the eligible calls. A successful episode is a false alarm when M>τ, counted at the first call where sˉt crosses the deployed threshold. This follows the definition of SAFE (Gu et al., 2025): a successful rollout becomes a false positive whenever its score exceeds the threshold, and the detection time is the first such crossing. We will state it clearly in the camera ready version.
W4 — numbers and conclusions of Sec. 5.2 in the main text
Reply: Sec. 5.2 will state each result with its conclusion. (a) Training under the rule raises early detection over the same head trained by cross-entropy, with every condition within the tolerance: from 39.3% to 57.1% (marginal) and from 32.5% to 52.4% (high-confidence) on LIBERO-Spatial, and from 33.4% to 46.5% and from 24.9% to 34.7% on LIBERO-Long. (b) Differentiating through the calibration rule adds 5.5 to 10.8 points of early detection on top of direct constrained optimization (the stop-gradient control of Table 6). (c) Against SAFE-LSTM, the strongest baseline, CIL leads in both matched modes on both OpenVLA-OFT benchmarks: 85.5 vs 77.6 (marginal) and 69.4 vs 54.1 (high-confidence) on LIBERO-Spatial, and 67.2 vs 63.7 and 51.4 vs 34.2 on LIBERO-Long.
References: Gu et al., 2025 (https://arxiv.org/abs/2506.09937).