- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that a pooled failure score for VLA runtime monitoring hides structurally different failures (choosing the wrong action vs. executing the right action incorrectly) and proposes that monitor validity is jointly determined by four coordinates: fault factor (F), sensor access (S), prediction target (L), and onset (O). It introduces VISTA, which encodes policy and execution prefixes with separate encoders and jointly trains an outcome head with training-only coordinate auxiliary heads (family/subtype/onset). Two squared-error decompositions (Eqs. 1–2) motivate separating accessible Bayes error from information carried only by the latent coordinate Z. Evaluation uses 2,640 trajectories from 360 sources in MuJoCo/robosuite LIBERO scenes with injected faults, plus frozen-model transfer on LIBERO-10, LIBERO-PRO, and VLABench; terminal AURC is 0.061 vs. 0.102 for a matched temporal MLP and 0.081 without coordinate supervision. A finite-sample binomial (Clopper–Pearson) procedure certifies 16 of 30 domain cells under risk ≤10% / coverage ≥60%.
Critical strengths
Fig. 4 and Table 6 show empirically that execution evidence dominates AUROC for execution faults (0.971) while policy evidence dominates for goal faults (0.968), and that terminal- and safety-trained heads produce non-interchangeable accepted sets at equal coverage. This reframes "one pooled failure score" in a way that prior monitoring papers do not.
Critical weaknesses
The 2640 runs are generated by an injector that swaps x/y commands, drops/delays/scales commands, ramps gain, or swaps a noun/relation at a chosen onset τ (4, "Faults and allocation"). Natural failures (contact slippage, drift, occlusion, perception error) are absent, so the claim that coordinates govern validity "in the wild" is supported only by engineered fault mechanisms. The independent actuator and corridor cohorts remain within the same MuJoCo/robosuite stack.
Questions for authors
Can you re-run the strongest external baselines (SAFE, ACE, Hide-and-Seek) on your own 840 held-out trajectories with identical prefixes, calibration, and AURC budget, rather than citing reported numbers?
2 comments
Dear Reviewer XWD6,
Before the discussion window closes, we wanted to briefly share that we have uploaded the revised manuscript PDF (revisions highlighted in blue), which directly incorporates your two points alongside additional baseline certifications:
- Zero-injection natural failures and claim scope (W1; Sec. 1, 5.3, 6 & Table 9): Sections 1 and 6 explicitly scope certified validity to the assessed task-policy distribution
Q, while Table 9 (formerly Table 7) highlights our 720 zero-injection rollouts (Unmodified workspace,Native referent ambiguity, andProtected-object corridor; frozen VISTA terminal Holm-adjustedp = 0.014) alongside native failures acrossLIBERO-10,LIBERO-PRO(6,000 runs), andVLABench. - External baseline interfaces, matched splits, and domain certification (Q1; Sec. 5, p. 5, Tables 3-5, Appendix A & new Table 10): Section 5 explicitly documents the feature-interface boundary (
S = Policyinternal embeddings / action chunks vs.S = Jointblack-box dispersion + controller telemetryX_e), and Appendix A (lines 781-784) clarifies that onLIBERO-10andVLABench(Tables 3 and 5),SAFE,Hide-and-Seek,ACE,STAC, andVISTAuse the exact same rollout banks and identical 210 / 35 / 105 / 150 splits (with VISTA adapting via outcome losslambda = 0only). In addition, new Table 10 (Sec. 5.4) certifies matched Table 2 baselines on the 600 core runs, expanding certified joint cells from 1 / 10 (Temporal MLP) and 3 / 10 (lambda = 0) to 7 / 10 (VISTA).
Thank you again for your time and thoughtful review. If our earlier response and these manuscript revisions have addressed your concerns regarding natural failure coverage and baseline comparisons, we would be very grateful if you would consider reflecting this in an updated rating before the discussion period concludes.
Thank you for your detailed comments on our fault-injection design and external baseline comparisons. We address natural failure coverage (W1) and baseline evaluation protocols (Q1) below.
1. Natural Failures vs. Controlled Fault Injection (W1)
In the 2,640-run study, controlled injection provides exact fault-factor (F) and onset (tau) ground truth across paired rollouts from identical initial states. Alongside controlled runs, Table 7 already evaluates frozen monitors on 720 trajectories with zero fault injection (no command perturbations and no mid-episode prompt edits), in addition to three external benchmarks with native VLA failures:
2. External Baselines (SAFE, ACE, Hide-and-Seek) and Interface Scope (Q1)
SAFE and Hide-and-Seek require internal VLA hidden or action-head embeddings, whereas ACE estimates entropy over multi-sample action chunks. Our 840-run primary suite was collected through a policy-agnostic black-box adapter spanning single-step autoregressive OpenVLA-7B and chunked SmolVLA-450M, recording 4-draw first-action dispersion and five controller-telemetry channels. To compare against each external method in its intended feature setting, Tables 2-4 follow the exact protocols from prior work (three 7-seen/3-unseen splits of 210/35/105/150 runs on LIBERO-10 and VLABench per Park et al., 2026; 50 reference + 50 calibration runs across 12 x 500 shifted runs on LIBERO-PRO per Lee & Har, 2026):