acceptodds
Under review as a conference paper at ICLR 2027

Same Failure, Different Evidence: Validity Domains For VLA Monitaring

Abstract

Vision-language-action (VLA) policies enable robots to follow language instructions across diverse tasks, making reliable failure monitoring essential. Yet strong aggregate detection performance does not reveal which failures a monitor can recognize or which consequences its evidence can support. A shared failure label can hide opposite evidence: a VLA may choose the wrong action or execute the right action incorrectly. We establish that monitor validity is jointly determined by fault factor, sensor access, prediction target, and onset. These coordinates change sensor rankings, accepted executions, and alarm timing. VISTA turns this insight into a learning rule: encode policy and execution prefixes separately, then jointly train outcome prediction and training-only coordinate heads. A study of 2,640 trajectories from 360 sources yields terminal AURC 0.061 on 840 held-out ranking runs, versus 0.102 for a matched temporal model and 0.081 without coordinate supervision. Gains persist on reset-only goals, native referent errors, and independent actuator faults. Simultaneous certification validates 16 of 30 domain cells under common non-alarm risk and coverage requirements. Joint sensing certifies all five completion strata but only two safety strata.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Runtime Failure Monitoring, Validity Domains

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XWD6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a pooled failure score for VLA runtime monitoring hides structurally different failures (choosing the wrong action vs. executing the right action incorrectly) and proposes that monitor validity is jointly determined by four coordinates: fault factor (F), sensor access (S), prediction target (L), and onset (O). It introduces VISTA, which encodes policy and execution prefixes with separate encoders and jointly trains an outcome head with training-only coordinate auxiliary heads (family/subtype/onset). Two squared-error decompositions (Eqs. 1–2) motivate separating accessible Bayes error from information carried only by the latent coordinate Z. Evaluation uses 2,640 trajectories from 360 sources in MuJoCo/robosuite LIBERO scenes with injected faults, plus frozen-model transfer on LIBERO-10, LIBERO-PRO, and VLABench; terminal AURC is 0.061 vs. 0.102 for a matched temporal MLP and 0.081 without coordinate supervision. A finite-sample binomial (Clopper–Pearson) procedure certifies 16 of 30 domain cells under risk ≤10% / coverage ≥60%.

Critical strengths

Fig. 4 and Table 6 show empirically that execution evidence dominates AUROC for execution faults (0.971) while policy evidence dominates for goal faults (0.968), and that terminal- and safety-trained heads produce non-interchangeable accepted sets at equal coverage. This reframes "one pooled failure score" in a way that prior monitoring papers do not.

Critical weaknesses

The 2640 runs are generated by an injector that swaps x/y commands, drops/delays/scales commands, ramps gain, or swaps a noun/relation at a chosen onset τ (4, "Faults and allocation"). Natural failures (contact slippage, drift, occlusion, perception error) are absent, so the claim that coordinates govern validity "in the wild" is supported only by engineered fault mechanisms. The independent actuator and corridor cohorts remain within the same MuJoCo/robosuite stack.

Questions for authors

Can you re-run the strongest external baselines (SAFE, ACE, Hide-and-Seek) on your own 840 held-out trajectories with identical prefixes, calibration, and AURC budget, rather than citing reported numbers?

2 comments

Thank you for your detailed comments on our fault-injection design and external baseline comparisons. We address natural failure coverage (W1) and baseline evaluation protocols (Q1) below.

1. Natural Failures vs. Controlled Fault Injection (W1)

In the 2,640-run study, controlled injection provides exact fault-factor (F) and onset (tau) ground truth across paired rollouts from identical initial states. Alongside controlled runs, Table 7 already evaluates frozen monitors on 720 trajectories with zero fault injection (no command perturbations and no mid-episode prompt edits), in addition to three external benchmarks with native VLA failures:

Zero-Injection Cohort (240 runs each, Table 7)Natural Failures (Term / Safety)Temporal MLP AURC (T / S)No-Aux (lambda = 0) AURC (T / S)Frozen VISTA AURC (T / S)
Unmodified workspace (standard rollouts)61 / 90 (25.4% / 37.5%)0.113 / 0.1590.096 / 0.1320.079 / 0.106
Native referent ambiguity (with distractors)69 / 91 (28.8% / 37.9%)0.132 / 0.1610.107 / 0.1330.085 / 0.108
Protected-object corridor (near obstacles)75 / 104 (31.3% / 43.3%)0.147 / 0.1960.121 / 0.1680.098 / 0.139
  • Natural error mix and statistical significance: On unmodified workspace rollouts, frozen VISTA significantly improves terminal AURC over the no-auxiliary control (Holm-adjusted p = 0.014). On native referent ambiguity, our score-blind audit (Sec. 5.2) finds wrong initial referents in 52 / 69 (75.4%) terminal failures and 17 / 69 (24.6%) failures after selecting the right initial object, matching the gain of Joint VISTA (0.845 AUROC) over Policy-only (0.802) and Execution-only (0.602, Fig. 4b).
  • External benchmarks and claim scope: All failures in LIBERO-10, LIBERO-PRO (6,000 shifted runs across 12 position/object/environment cells), and VLABench (long-horizon manipulation across OpenVLA, pi_0, and pi_0.5, Tables 2-4) occur natively without our injector, and VISTA leads on each benchmark. Sections 1 and 6 also state explicitly that certified validity is defined with respect to the assessed task-policy distribution Q.

2. External Baselines (SAFE, ACE, Hide-and-Seek) and Interface Scope (Q1)

SAFE and Hide-and-Seek require internal VLA hidden or action-head embeddings, whereas ACE estimates entropy over multi-sample action chunks. Our 840-run primary suite was collected through a policy-agnostic black-box adapter spanning single-step autoregressive OpenVLA-7B and chunked SmolVLA-450M, recording 4-draw first-action dispersion and five controller-telemetry channels. To compare against each external method in its intended feature setting, Tables 2-4 follow the exact protocols from prior work (three 7-seen/3-unseen splits of 210/35/105/150 runs on LIBERO-10 and VLABench per Park et al., 2026; 50 reference + 50 calibration runs across 12 x 500 shifted runs on LIBERO-PRO per Lee & Har, 2026):

Runtime MonitorFeature InterfaceVLABench pi_0.5 Seen / Unseen bAccVLABench Unseen TWALIBERO-PRO AUROC / bAcc
ACE (Romer et al., 2025)Multi-sample action-chunk entropy (S = Policy)0.596 / 0.5910.5520.844 / 0.744
SAFE-LSTM (Gu et al., 2025)Internal VLA hidden states (S = Policy)0.629 / 0.4840.482--
SAFE-MLP (Gu et al., 2025)Internal VLA hidden states (S = Policy)0.788 / 0.6410.500--
Hide-and-Seek (Park et al., 2026)Internal VLA action-head states (S = Policy)0.856 / 0.7130.608--
VISTA (Joint)4-draw dispersion + controller X_e (S = Joint)0.864 / 0.8330.6350.996 / 0.929
  • Role of explicit controller telemetry on the 840-run suite: SAFE, ACE, and Hide-and-Seek operate solely on policy-side representations or action samples (S = Policy) and do not take command-motion residual, acknowledgment-age, joint-tracking, or stall telemetry (X_e). On our primary suite, pairing policy signals (including 4-draw action dispersion) with X_e raises execution-fault AUROC from 0.710 / 0.720 (VISTA Policy-only) to 0.985 / 0.976 (Joint VISTA, Fig. 4a), reduces unflagged core execution failures from 27 / 40 (terminal) and 36 / 49 (safety) to 0 / 40 and 2 / 49, and increases certified cells from 4 / 10 to 7 / 10 (Fig. 5). Section 5 now highlights this interface distinction between Tables 1 and 2-4 explicitly.

Dear Reviewer XWD6,

Before the discussion window closes, we wanted to briefly share that we have uploaded the revised manuscript PDF (revisions highlighted in blue), which directly incorporates your two points alongside additional baseline certifications:

  • Zero-injection natural failures and claim scope (W1; Sec. 1, 5.3, 6 & Table 9): Sections 1 and 6 explicitly scope certified validity to the assessed task-policy distribution Q, while Table 9 (formerly Table 7) highlights our 720 zero-injection rollouts (Unmodified workspace, Native referent ambiguity, and Protected-object corridor; frozen VISTA terminal Holm-adjusted p = 0.014) alongside native failures across LIBERO-10, LIBERO-PRO (6,000 runs), and VLABench.
  • External baseline interfaces, matched splits, and domain certification (Q1; Sec. 5, p. 5, Tables 3-5, Appendix A & new Table 10): Section 5 explicitly documents the feature-interface boundary (S = Policy internal embeddings / action chunks vs. S = Joint black-box dispersion + controller telemetry X_e), and Appendix A (lines 781-784) clarifies that on LIBERO-10 and VLABench (Tables 3 and 5), SAFE, Hide-and-Seek, ACE, STAC, and VISTA use the exact same rollout banks and identical 210 / 35 / 105 / 150 splits (with VISTA adapting via outcome loss lambda = 0 only). In addition, new Table 10 (Sec. 5.4) certifies matched Table 2 baselines on the 600 core runs, expanding certified joint cells from 1 / 10 (Temporal MLP) and 3 / 10 (lambda = 0) to 7 / 10 (VISTA).

Thank you again for your time and thoughtful review. If our earlier response and these manuscript revisions have addressed your concerns regarding natural failure coverage and baseline comparisons, we would be very grateful if you would consider reflecting this in an updated rating before the discussion period concludes.

Reviewer_QS5BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how VLA failure monitoring depends on fault mechanisms, sensor access, prediction targets, and fault timing. It proposes VISTA to combine policy and execution evidence, and assesses monitoring reliability through domain-wise risk and coverage certification.

Critical strengths

The controlled experiments show that execution feedback is more informative for execution faults, while policy signals are more informative for goal-related faults.

Critical weaknesses
  1. The paper does not clearly distinguish the conceptual contribution of the four-coordinate framework from the empirical benefits of VISTA’s architecture and auxiliary supervision.
  2. The evidence for reliable alarm decisions on recoverable disturbances is limited. The recovery experiment reports post-restoration score changes, but not whether successful recovered executions trigger alarms at the calibrated threshold.
Questions for authors
  1. What is the distinct conceptual contribution of the four-coordinate framework beyond VISTA’s architecture and auxiliary supervision?
  2. At the calibrated threshold, what fraction of successfully recovered executions trigger an alarm?
5 comments

We appreciate your close reading and direct questions. Below we clarify how the four-coordinate formulation stands apart from VISTA's architecture and loss (W1/Q1), report exact alarm fractions at the calibrated threshold on recovered runs (W2/Q2), and list the manuscript edits made to improve clarity.

1. Conceptual Role of the Four-Coordinate Framework vs. VISTA (W1 & Q1)

The tuple d = (F, S, L, O) defines the monitoring problem and its evaluation strata for any black-box score, whereas VISTA is one concrete learning rule operating within that formulation:

ComponentScope & RoleKey Empirical Finding
Problem tuple d = (F, S, L, O) (Sec. 2, 5.2)Model-agnostic; practitioner sets sensors S and target L, while F and O index unobserved test-time conditionsCompletion and safety labels disagree on 193 / 840 (23.0%) runs; sensor rankings reverse across F (Figs. 4a-b)
VISTA monitor (Sec. 3, 5.1)Targets accessible prediction error (Eq. 2, Term 1) via separate S encoders and training-only (F, O) headsCuts terminal / safety AURC from 0.102 / 0.129 (Temporal MLP) and 0.081 / 0.107 (lambda = 0) to 0.061 / 0.083 (Table 1)
Domain certification (Sec. 5.4, Fig. 5)Audits any fixed score and threshold q across 30 cells for accepted risk <= 0.10 and coverage >= 0.60 (Prop. 1)Despite 0.944 / 0.900 pooled AUROC, Joint VISTA certifies 5 / 5 completion cells vs. 2 / 5 safety cells
  • Information structure vs. learned monitor: Independently of VISTA, Equation (1) specifies when execution telemetry X_e reduces Bayes squared error beyond policy signals X_p, while Equation (2) separates accessible prediction error over (X_p, X_e) (Term 1) from the irreducible Bayes-risk gap caused by unobserved Z = (F, O) (Term 2). Separate policy and execution encoders prevent cross-stream overwriting (mixing routes raises safety AURC from 0.083 to 0.101, Table 5), and at 70% coverage our terminal- and safety-trained VISTA heads select accepted sets whose symmetric difference spans 160 runs (80 unique to each, Table 6).
  • Why ranking still requires certification: Because Term 2 depends on unobserved (F, O), pooled ranking metrics cannot guarantee uniform threshold validity. Section 5.4 therefore certifies conditional non-alarm risk P_d(Y^L = 1 | M <= q) <= 0.10 and coverage P_d(M <= q) >= 0.60 cell by cell using exact Clopper-Pearson bounds.

2. Alarm Fractions on Successfully Recovered Executions at the Calibrated Threshold (W2 & Q2)

Across the 40 paired certification recovery runs (31 core-success to recovery-success, 7 core-failure to recovery-success reversals, and 2 core-failure to recovery-failure, matching the 7 reversed and 33 unchanged pairs in Sec. 5.1), 38 / 40 executions succeed. Evaluating these 38 recovered successes at the fixed nominal threshold q (beta = 0.10 on n_0 = 112 nominal negatives) separates active-disturbance warnings from post-restoration alarms:

Score Evaluated at Fixed Threshold qWhole Episode [1, 300]Active Fault [tau, tau + 20]Post-Restoration (t > tau + 20)Recovery Pair AURC
VISTA (Joint)8 / 38 (21.05%)7 / 38 (18.42%)2 / 38 (5.26%)0.061
Fault-family-only33 / 38 (86.84%)31 / 38 (81.58%)29 / 38 (76.32%)0.193
  • Active fault vs. post-restoration specificity: Almost all VISTA crossings on recovered runs occur during [tau, tau + 20] (7 / 38), where causal prefixes are identical to unrecovered core trajectories. Once the instruction or controller is restored (t > tau + 20), only 2 / 38 (5.26%) cross q, matching VISTA's specificity on non-failing core runs (108 / 112 = 96.4% nominal and 380 / 396 = 96.0% faulted-core non-alarm rates, Fig. 5).
  • Outcome supervision vs. disturbance memory: Prefix statistics retain historical disturbance cues, causing a calibrated fault-family-only score to keep alarming after restoration on 29 / 38 (76.32%) recovered successes. By contrast, VISTA's outcome head revises risk downward after restoration (median delta_r = -0.22 on the 7 reversed runs vs. 0.02 for fault-family probability, Sec. 5.1). We have added this window-level table to Section 5.1.

3. Revisions for Clarity

  • Distinct terminology: We reserve "validity coordinates" for the evaluation domain d = (F, S, L, O) and refer to VISTA's training-only branches in Sec. 3 and Fig. 2b as "auxiliary mechanism-and-onset heads."
  • Clearer Section 2 structure: Section 2 now groups d into practitioner design choices (S, L) and unobserved test-time variables (F, O), paired with a summary table contrasting the problem formulation, VISTA, and certification.
  • Streamlined Sections 4-5: We moved per-seed and per-cell number lists from inline prose into structured tables so the main text stays easy to read.

Thank you for the response. The recovery alarm table answers my second question, and the distinction between the formulation, VISTA, and the certification helped me understand how the paper is organized.

My concern about presentation remains. The response describes several edits, but I do not see an updated PDF, so I can only assess the original manuscript. It does not make the relation between these three parts clear.

With this understanding, I think the formulation needs clearer positioning. Agia et al. [1], which the paper cites, already separate failure types and assign a different detector to each. Jones et al. [2] show that selective classification can improve average accuracy while magnifying disparities across groups. Please state which part of the four-coordinate formulation is new relative to these.

The link between the formulation and VISTA is also untested. The certification covers only VISTA variants, so the results do not show that coordinate supervision enlarges the validity domain.

I also share Reviewer XWD6's concern about the external baselines. I will keep my rating.

[1] Agia et al. Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress. CoRL 2024. [2] Jones et al. Selective Classification Can Magnify Disparities Across Groups. ICLR 2021.

Thank you for pointing to Agia et al. [1] and Jones et al. [2]. We have uploaded the revised PDF (in blue: restructured Sec. 2 with Table 1 and positioning vs. [1, 2], recovery Table 8 in Sec. 5.1, and baseline certification Table 10 in Sec. 5.4).

1. Positioning d = (F, S, L, O) vs. Agia et al. [1] and Jones et al. [2]

Core distinction: Agia et al. [1] split failure symptoms for one shared failure label, while Jones et al. [2] analyze subgroup accuracy gaps under a fixed input X and fixed label Y. Neither captures our central finding: monitoring validity is not a subgroup property of a fixed (X, Y) task, because on the exact same trajectories, changing sensors S, target consequence L, or recovery timing O reverses risk rankings and accepted sets:

FrameworkFault F & Onset OSensors SConsequence LGuarantee
Agia et al. [1]Erratic vs. slow progress; no recovery OPolicy actions + video VLM (no X_e)Single failure labelEmpirical detection rate
Jones et al. [2]Static subgroups g (no prefix timing O)Fixed input X across groupsSingle fixed label YGroup disparity curves
Ours: d=(F,S,L,O)Mechanism F + onset/recovery O=(tau, t-tau)Varies X_p vs. controller X_e (Eq. 1)Distinct L (23.0% disagree)Finite-sample risk/cov. (Prop. 1)
  • New along L: Completion and safety disagree on 193 / 840 (23.0%) runs; at 70% coverage, target-specific heads differ on 160 accepted runs and drop to 0.660 / 0.690 cross-target AUROC (Table 7).
  • New along S: Eq. (1) specifies when closed-loop controller telemetry X_e reduces Bayes squared error over policy signals X_p, explaining opposite sensor rankings across goal vs. execution faults (Fig. 4a).
  • New along O & certification: Eq. (2) separates accessible error from the unobserved-(F, O) Bayes gap (38 / 40 recovered runs, Table 8), and Prop. 1 certifies simultaneous Clopper-Pearson risk <= 0.10 and coverage >= 0.60 with mixture transfer.

2. Evidence Separation and Auxiliary Supervision Enlarge Certified Domains

Applying the Section 5.4 calibration (beta = 0.10, n_0 = 112/110) and Clopper-Pearson rule (L_C >= 0.60, U_R <= 0.10; identical decisions under Eq. 5's delta = 0.05/60 and 50-cell union bound delta = 0.05/100) to the matched Joint Temporal MLP and No-Auxiliary Control (lambda = 0) on all 10 joint cells (600 core runs, Table 10; bold = certified A/K):

Joint Monitor (Seed 0)NominalExec. EarlyExec. LateGoal EarlyGoal LateCertified
Terminal: Temporal MLP104/297/292/395/288/50/5
Terminal: No-aux (lambda=0)111/197/196/296/090/23/5
Terminal: VISTA (lambda=0.3)108/098/094/096/092/05/5
Safety: Temporal MLP106/196/381/481/580/71/5
Safety: No-aux (lambda=0)109/295/286/579/478/50/5
Safety: VISTA (lambda=0.3)107/094/084/282/380/42/5

Certified joint cells grow from 1/10 for Temporal MLP (0/5 terminal, with 2-5 unflagged failures violating U_R <= 0.10 in every stratum) to 3/10 with separate policy/execution encoders (lambda = 0), and to 7/10 with auxiliary supervision (VISTA, lambda = 0.3, unlocking late-onset completion for 5/5 terminal plus 2/5 safety cells).

3. External Baseline Interfaces (Sec. 5, p. 5)

Section 5 now explicitly documents the interface boundary between Table 2 (black-box policy dispersion + five controller-telemetry channels X_e across single-step OpenVLA-7B and chunked SmolVLA-450M) and Tables 3-5, where SAFE, Hide-and-Seek (internal VLA hidden/action-head states, S = Policy), and ACE (action-chunk entropy, S = Policy) follow their native protocols on LIBERO-10, LIBERO-PRO, and VLABench.

Reviewer_AJAVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the problem of creating and evaluating a monitor / verifier / failure detector for VLA systems. Such a monitor can be used to predict the likelihood of failure for a particular trajectory before its completion, which can be used to stop execution / request human assistance / route to a larger VLM. A better monitor can obtain a better risk - coverage tradeoff (like in selective prediction). The authors first introduce a categorization of scenarios a monitor would need to make predictions in, based on four categories: the source of the failure (execution vs policy/planning), sensor access (what sensors the monitor has access to), prediction target (is the monitor trying to predict task failure or safety failure), and 'failure onset (early vs late). This allows for a more fine grained evaluation of failures than aggregate success across all scenarios. The authors also introduce a training rule, that separate reasons for the failure of a trajectory into goal/policy errors vs execution errors, and train a small monitor (<2000 params) based on handcrafted features separating policy and execution information in order to predict failure and also show that adding auxiliary losses based on their scenario categories improves performance.

Critical strengths

Extensive Ablations were done to isolate the benefit of the auxiliary losses based on failure mode axes and some ablations are done for feature selection. Although I did not see much discussion of this, using a small verifier/monitor based on simple features might be more interpretable.

Critical weaknesses

It may be helpful to include experiments on physical robots to show robustness of the system.

Questions for authors
  • How prevalent are goal/policy versus execution errors among naturally occurring failures (e.g., the 61 failures in 240 unmodified trajectories)?
  • Can the author elaborate on the application of the monitor in a downstream task? For example, using the monitor to request for human intervention, vote/rerank, or route to a larger more expensive VLM. A deeper discussion of downstream applicability or an additional experiment, (e.g., using the monitor for re-ranking) can be useful.
  • It’s possible using a small monitor based on aggregate information throws away some useful information, such as temporal information, and cannot take advantage of increased scale / computation. It would make sense to baseline against a larger monitor (e.g., directly fine tuning the VLA) or a hybrid where you allow a larger monitor in addition to key aggregate features.
2 comments

Thank you for your positive feedback and thoughtful questions on natural failures, downstream gating, capacity, and hardware.

1. Goal/Policy vs. Execution Errors in Natural Failures (Q1)

  • Native referent ambiguity (240 unperturbed runs, Table 7): Under correct instructions and unmodified policies/controllers, our score-blind audit of the 69 natural terminal failures (Sec. 5.2) finds wrong initial referents in 52 / 69 (75.4%) cases, while 17 / 69 (24.6%) fail after selecting the right initial object. Matching this mix, Policy-only, Execution-only, and Joint VISTA reach 0.802, 0.602, and 0.845 AUROC (Fig. 4b).
  • Unmodified workspace (240 standard runs, 61 terminal / 90 safety failures): Without distractors or injections, frozen Joint VISTA reaches 0.079 / 0.106 terminal / safety AURC versus 0.096 / 0.132 without auxiliary heads (terminal Holm-adjusted p = 0.014) and 0.113 / 0.159 for the matched Temporal MLP (Table 7).

2. Downstream Use: Selective-Abstention Gate Analysis (Q2)

Figure 5 and Table 6 evaluate VISTA as a retrospective target-specific alarm gate (flagging M > q at beta = 0.10 for abstention, human help, or VLM escalation) on the 600 core runs (base risk: 92/600 = 15.3% terminal; 127/600 = 21.2% safety):

Gate (600 Core Runs, Fig. 5)Alarm RateFalse AlarmsUnflaggedAccepted RiskCertified
Terminal Policy-only25.7% (154/600)17.7% (90/508)28/926.3% (28/446)3/5
Terminal Exec.-only26.8% (161/600)19.7% (100/508)31/927.1% (31/439)3/5
Terminal VISTA (Joint)18.7% (112/600)3.9% (20/508)0/920.0% (0/488)5/5
Safety Policy-only32.3% (194/600)31.3% (148/473)81/12720.0% (81/406)1/5
Safety Exec.-only28.8% (173/600)19.5% (92/473)46/12710.8% (46/427)2/5
Safety VISTA (Joint)25.5% (153/600)7.4% (35/473)9/1272.0% (9/447)2/5
  • Precision and alarm timing: Joint VISTA raises fewer alarms than single-route gates while reaching 82.1% terminal precision (0 unflagged failures) and flagging 118 / 127 (92.9%) safety-violating runs. Across faulted terminal failures (Table 6), 64 / 78 (82.1%) post-onset cases alarm within 40 steps (2.0 s), with median delay 4.0-7.2 steps (0.20-0.36 s). Section 5.4 now includes this gate analysis and feature-level interpretability.

3. Temporal Summarization, Model Scale, and Hybrid Monitors (Q3)

  • Prefix statistics vs. sequence capacity: Moving from single-step joint inputs (0.141 / 0.183 AURC) to prefix summaries (Temporal MLP, 0.102 / 0.129) and VISTA (0.061 / 0.083) cuts terminal AURC by 56.7% over single-step inputs and 40.2% over the matched Temporal MLP at ~1.3k parameters (Table 1). With modest monitor-fitting data (210 runs in Table 4), larger sequence models over VLA embeddings can overfit: SAFE-LSTM drops from 0.629 (seen) to 0.484 (unseen bAcc), behind SAFE-MLP (0.641) and VISTA (0.833). Current values plus prefix mean/max/min, ack age, and stall indicators retain multi-scale history in O(1) memory.
  • Larger VLA probes and hybrid compatibility: On internal pi_0.5 features (Table 4), VISTA reaches 0.833 unseen bAcc versus 0.713 for Hide-and-Seek (+12.0 points). Moreover, internal VLA embeddings can feed directly into or concatenate with VISTA's policy branch h_p alongside the separate execution encoder h_e and auxiliary losses.

4. Physical Robot Experiments (W1)

Physical-robot evaluation is an important next step. Our controlled study uses seven paired variants per source for ranking (840 runs) and five core conditions across 30 sensor-target-stratum cells (600 runs) to certify threshold rules against repeatable > 2 N contact-force predicates. To test low-level actuation mismatches below the policy wrapper, Table 7 adds Independent actuator faults (0.4/0.7 gain scaling and 2- or 6-step servo latency; VISTA AURC 0.097 / 0.113 vs. 0.121 / 0.139; terminal Holm-adjusted p = 0.006). Section 6 discusses hardware transfer and threshold recalibration.

Dear Reviewer AJAV,

Before the discussion window closes, we wanted to briefly thank you again for your supportive review and let you know that we have uploaded the revised manuscript PDF (revisions highlighted in blue) incorporating all of the additions prompted by your questions:

  • Natural failure breakdown & actuator transfer (Q1 & W1): Updated in Section 5.2 (52 / 69 initial referent errors vs. 17 / 69 post-selection failures), Section 5.3 / Table 9 (Independent actuator perturbations, p = 0.006), and Section 6 (hardware transfer and nominal recalibration).
  • Downstream selective-abstention gate & interpretability (Q2): Added in Section 5.4 (p. 9) and Appendix B / Table 13 (full six-configuration gate precision, false-alarm, and accepted-risk breakdown across the 600 core runs, plus (h_p, h_e) branch attribution), alongside new Table 10 showing certified joint cells expanding from 1 / 10 (Temporal MLP) and 3 / 10 (lambda = 0) to 7 / 10 (VISTA).

Your constructive suggestions made the empirical analysis much stronger, and we hope these additions further reinforce your positive assessment of the paper as the discussion concludes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.