acceptodds
Under review as a conference paper at ICLR 2027

Where Is Failure Predictable? Stage-Wise Attribution in Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models map observations and instructions to robot actions, but they still fail in ways that can damage objects or leave the robot in unrecoverable states. A VLA processes each step in three phases: it aligns the scene with the instruction, reasons over the aligned representation, and decodes an action. A failure can originate in any of these phases, yet existing failure detectors return a single score for the whole policy. We derive a theoretical characterization of uncertainty and failure predictability across the VLA processing pipeline. In particular, we establish an exact decomposition of predictive action entropy into alignment, reasoning, and decoding terms with nonnegative population values, and prove that failure predictability likewise decomposes into nonnegative gains from phase-restricted scores. Motivated by our theoretical results, we develop STAGE, a computational framework for phase-wise failure prediction. STAGE extracts alignment and reasoning signals from their stage representations, computes decoding uncertainty directly from the decoder, and integrates the three signals to predict failure in a single VLA forward pass. Extensive experiments demonstrate the effectiveness of STAGE across three VLA families (OpenVLA, , and -FAST), four LIBERO suites, and SimplerEnv. STAGE achieves the highest AUROC in most settings, outperforming the strongest baseline by up to points, and outperforms all evaluated baselines on real-robot Agilex rollouts. Stage-targeted perturbations further validate the phase-wise attribution: each score responds to its corresponding stage while remaining unaffected by perturbations to later stages, revealing where failure signals arise in the VLA pipeline.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision language action models, failure, uncertainty

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_B6YPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether action chunks from a frozen Vision-Language-Action (VLA) model can predict episode failure and trace it to alignment, reasoning, or decoding. The authors introduce STAGE, which extracts one score per stage and combines them in a lightweight logistic predictor trained on episode-level outcomes. Grounded in theoretical analyses of action uncertainty and failure predictability, STAGE is evaluated on OpenVLA, \pi_0, and \pi_0-FAST across LIBERO, SimplerEnv, and 201 real-robot Agilex rollouts. It achieves the highest AUROC in 11 of 14 simulation settings and substantially outperforms baselines on the real robot. The key question is whether its scores truly identify failure stages or form an effective three-feature predictor.

Critical strengths
  • The paper presents a simple, effective design. Using just three stage-specific scores makes ablations straightforward, requires only one pass through the frozen VLA, and avoids stage-level supervision.
  • It evaluates three VLA families, both discrete and continuous decoders, several simulation benchmarks, and real-robot rollouts. The appendices include seen- and unseen-task results and discuss key theoretical limitations, including the collapse of the Shannon decomposition.
  • STAGE outperforms all baselines across the five OpenVLA settings, with a particularly strong gain on LIBERO-Goal (82.58 vs. 74.05 for SAFE-MLP). Table 10’s token-uncertainty results also support using internal representations rather than decoder uncertainty alone.
Critical weaknesses
  • Theorem 3.1 concerns optimal log-linear probes of action distributions, whereas STAGE learns alignment and reasoning scores from failure labels. So its guarantees of nonnegativity and decomposition do not apply to STAGE’s learned scores. Proposition 3.2 follows from nested predictor families in general, not specifically from VLA stages. Its gains depend on stage ordering and differ from the experimental leave-one-stage-out AUROC results. Appendix B also shows that Shannon information assigns no additional information to reasoning, since the final hidden state is determined by earlier representations. Any nonzero reasoning contribution therefore depends on the restricted probe family. Finally, it is unclear whether the theory applies to \pi_0 and its compressed \pi_0-FAST action tokens.
  • Perturbation experiments show that each score responds to changes at its own stage and possibly earlier stages, but not later ones, since changes to decoder temperature cannot affect scores based on earlier representations. This confirms the intended information flow, but not that the scores identify causes of failure. Evidence remains limited: there are no ground-truth stage labels or counterfactual interventions, and Figure 3 offers only one qualitative example. Unclear normalization also leaves open whether the visualization uses information from future rollouts. Overall, the evidence does not support the claim that STAGE reveals where failures originate.
  • The results focus mainly on AUROC, leaving out AUPRC, calibration, false-alarm rates, and early-detection metrics—all important for runtime failure detection, especially when benchmark success rates differ. The largest gain (+10.6 points on \pi_0 LIBERO-Spatial) still reaches only 64.37 AUROC. Combining seen and unseen tasks can hide generalization gaps: \pi_0’s LIBERO-10 AUROC drops from 91.11 on seen tasks to 58.46 ± 11.33 on unseen tasks. SimplerEnv also combines its two embodiments. Reporting these results separately would give a clearer picture of practical reliability.
  • The paper is missing key ablations and implementation details. Its phase ablation covers only OpenVLA LIBERO-Spatial, and the single-stage results vary widely (±13–14 points) compared with ±1.16 for the full model. Without a capacity-matched unrestricted baseline using the same internal representations, it is hard to separate the effects of stage-wise restrictions from those of learned readers. The paper also says the continuous-decoder score is based on predicted-action statistics but gives no formula, limiting reproducibility of the \pi_0 and real-robot results. Finally, the reported Agilex gain (98.25 vs. 68.88 AUROC) is based on just 60 test rollouts across three seen tasks, with uncertainty measured across seeds rather than episodes.
Questions for authors
  • Please provide the exact formula for the decoding score (S_{3,h}) used with \pi_0/open-pi-zero and Agilex, and include a sensitivity analysis.
  • Compare STAGE with a capacity-matched predictor trained on the same alignment and reasoning representations but without the three-score restriction, ideally using both OpenVLA and \pi_0. How much does the restriction contribute?
  • Give greater prominence to unseen-task AUROC and AUPRC. Also report an early-detection metric, such as AUROC over the first 25% or 50% of an episode, or lead time before failure.
  • Test failures caused by known interventions, such as instruction–object mismatches or spatial rearrangements, and assess whether the expected stage score increases. Also examine attribution stability across seeds and stage orderings.
  • Confirm that Agilex splits are episode-disjoint. Report per-task AUROC, the number of test-set failures, and episode-bootstrap confidence intervals for STAGE and SAFE-MLP.
  • Explain why individual-stage predictors in Table 4 have much higher variance than the full model, and extend the ablation to another backbone, preferably \pi_0.
  • Define the normalization used in Figure 3 and clarify whether it can be computed online. Figure 2 also appears to retain the label "TRACE Output”; please correct it.
9 comments

We thank the reviewer for the positive assessment and the valuable feedback. We address the comments as follows:

W1-a. Connection between theory and algorithm. We thank the reviewer for the constructive feedback. We would like to clarify the connection between the theory and the algorithm: the theorem motivates the design of STAGE rather than describing the behavior of its trained readers. The gap between the two can arise in two places. The first is (i) finite samples: the theorem holds in expectation, so individual steps may deviate from it. The second is (ii) learned readers: STAGE's readers are trained MLPs rather than the optimal probes. Not every term is affected, however. Since s2=KL(π∥q1)s_2 = \mathrm{KL}(\pi\|q_1) and s3=H(π)s_3 = H(\pi) are nonnegative at every step for any probe, only the alignment term relies on the population, optimal-probe assumption. We measured both gaps on held-out data and found them small on average.

Factor (i): finite samples. To check whether the population claims of Theorem 3.1 hold on held-out data, we took all OpenVLA rollouts of the four LIBERO suites and fit the nested log-linear probes of Def. 3.1 to the policy's own action distribution, without any failure labels. We then computed s1s_1, s2s_2 and s3s_3 at every step with 5-fold cross-fitting over episodes.

Table 1. Theoretical calculation of each phase estimation.

OpenVLA suiteE[s1s_1]steps with s1s_1 negativeE[s2s_2]E[s3s_3]
LIBERO-Spatial1.4228.4%18.262.42
LIBERO-Object1.5823.4%13.872.46
LIBERO-Goal1.0932.0%17.192.21
LIBERO-101.7921.1%16.202.56

Values in nats, summed over the 7 action tokens. The results agree with the theory: the expected alignment term is positive in every suite, while s2s_2 and s3s_3 are nonnegative by construction. The gap appears only at the level of individual steps, where 21–32% of steps have a negative s1s_1.

Factor (ii): learned readers. We next asked whether Prop. 3.2 still holds when the optimal probes are replaced by STAGE's trained readers. For OpenVLA on the four LIBERO suites, we trained STAGE on every subset of the three phases with 3 seeds and measured the failure log loss on test episodes after a cross-fitted recalibration. Prop. 3.2 predicts that adding a phase kk to any subset SS never increases the loss, so we checked this prediction for every nested pair (S,S∪{k})(S, S\cup\{k\}).

Table 2. Nonnegativity violation rate in real-world models.

OpenVLA suitepairs consistent with Prop. 3.2violation ratio
LIBERO-Spatial92% (33/36)1.4%
LIBERO-Object67% (24/36)7.0%
LIBERO-Goal81% (29/36)15.9%
LIBERO-1078% (28/36)10.9%

Each suite has 36 nested pairs (12 pairs × 3 seeds), all of which the theory predicts to be consistent. The violation ratio divides the total size of the negative gains by the total size of all gains, so 0% means no gap. Overall, 79% of the pairs are consistent with Prop. 3.2, and the violations account for only 7.9% of the total gain. The trained readers therefore follow the theory closely.

What STAGE takes from the theory. Finally, Theorem 3.1 concerns the policy's own action, not failure. STAGE therefore uses it to decide which representation each phase reads (ZZ, FF and π\pi) and adopts S3=∑hH(πh)S_3=\sum_hH(\pi_h) exactly, but learns S1S_1 and S2S_2 from failure labels. This choice matters in practice. When we use the theory terms directly as failure scores, following the protocol of the paper's Tables 1–4 on each seed's own test set, they perform poorly (Table 3): they stay close to chance and sometimes even point the wrong way (below 50), whereas STAGE leads the better of the two by 14–36 points in every suite and split. This is why STAGE keeps the phase structure given by the theory but learns failure-relevant scores within it.

Table 3. AUROC for failure detection using theoretical estimation.

AUROCSpatialObjectGoalLIBERO-10
All test episodes
theory s1s_152.51±1.4558.78±0.6952.53±0.1748.43±0.28
theory s2s_256.20±0.6439.03±1.0061.79±0.5744.63±0.26
STAGE87.35±1.1682.48±1.9882.58±1.6668.73±1.11
Unseen tasks
theory s1s_152.23±3.5957.09±3.0757.98±4.9543.94±0.98
theory s2s_266.40±1.5735.00±9.2268.27±3.4136.55±2.62
STAGE85.85±2.2777.09±4.3382.62±4.3164.28±2.00
Seen tasks
theory s1s_154.32±1.9160.72±0.6949.61±3.1350.51±1.01
theory s2s_251.33±1.6241.89±1.5056.55±1.3349.03±1.09
STAGE90.01±1.5687.22±1.9184.48±0.5271.10±1.18

W1-b. Stage ordering. Thanks to the reviewers for the valuable feedback. We would first like to clarify where an order enters our analysis. It appears only in Theorem 3.1, and only because the VLA computes its stages in sequence: the nested probes read C, then (C, Z), then (C, Z, F) (Def. 3.1, lines 146–151), following the forward pass Z → F → π (Eq. 2, line 140). This order is therefore fixed by the architecture rather than chosen by us. Prop. 3.2, in contrast, makes no assumption about order and holds "for any ordering k₁, k₂, k₃ of the phases" (line 238). STAGE's head is likewise symmetric, σ(b+w1S1+w2S2+w3S3)\sigma(b + w_1S_1 + w_2S_2 + w_3S_3), and is trained jointly on all three scores (Eq. 10, lines 320–323), so its detection AUROC does not depend on any ordering either.

Prop. 3.2 complements Theorem 3.1 rather than restating it. Theorem 3.1 splits the policy's action uncertainty along the pipeline, using nested probe families that read Z, then F, then π\pi. Prop. 3.2 carries the same nesting over to failure prediction: predictors that read more phases contain those that read fewer (lines 236–237), so adding a phase can only lower the population log loss. Unlike Theorem 3.1, it is decoder-agnostic and order-free, and therefore covers all three backbones. We use it because it makes per-phase attribution well-defined: when the predictor sees only the three phase scores, removing one score measures that phase's contribution, and this gain is guaranteed to be non-negative (lines 245–250).

We would also appreciate a clarification of what "stage ordering" refers to. Our understanding is that the gain of a phase in Eq. (9) depends on which phases were added before it, whereas the paper's Table 4's leave-one-stage-out drops measure only one ordering (each phase added last), and in AUROC rather than log loss. Therefore, we report the gains under all orderings in Table 4 below:

Table 4. Order influence evaluation

OpenVLA, gain A / R / DShapley (order-averaged)added firstadded last
LIBERO-105.9 / 19.9 / 2.94.8 / 19.7 / 5.710.6 / 23.8 / 3.7
LIBERO-Spatial35.4 / 39.8 / 14.024.9 / 19.5 / 10.032.5 / 46.7 / 4.5
LIBERO-Object32.6 / 66.5 / 1.042.8 / 65.0 / −0.19.5 / 55.0 / −10.8
LIBERO-Goal25.5 / 21.1 / 1.230.5 / 31.6 / 9.16.9 / −3.2 / −20.4

A / R / D = alignment / reasoning / decoding. Gain in held-out log loss on each seed's test episodes, with the mean over 3 seeds. "Added last" is the log-loss version of the paper's Table 4's leave-one-stage-out drop. The ordering does shift credit between alignment and reasoning on Spatial and Goal, but it does not change the conclusion: the top stage is alignment or reasoning, and decoding has the smallest gain. So our findings do not depend on a particular stage order.

W1-c The reasoning term under Shannon information. We thank the reviewer for the comments. We would like to clarify that a frozen VLA is deterministic: once the image and instruction are given, every intermediate representation, and hence the decoder output π, is fixed, so Shannon information gives no credit to the intermediate stages. This does not mean, however, that the stages carry no signal. The decoder entropy H(π)H(\pi) summarizes the whole pipeline in a single number and therefore cannot tell which stage left the action uncertain and failed. Separating the stages requires reading ZZ and FF directly with a reader of limited capacity, one that cannot simply recompute the network. This is the idea of usable information[1], and it is also the setting that matters in practice, since a runtime failure monitor is exactly such a limited reader.

W1-d, Backbone coverage. We'd like to clarify that our theory applies to the OpenVLA and pi0-fast models. Theorem 3.1 requires a softmax linear head (Assumption 3.1, lines 156–161). It therefore applies to OpenVLA (line 160) and to π0\pi0-FAST, whose FAST tokens come from the same kind of head, although for π0\pi0-FAST the entropy is over the compressed tokens rather than physical actions (lines 108–112). It does not apply to π0\pi0, whose flow-matching decoder has no token softmax and generates the chunk jointly, as the draft states (lines 113–114, 304–307). For $ \ pi_0$, we instead follow SAFE [2] and replace the token entropy with a statistic of the sampled actions as an adaptation to test our method.

W2-a, Q4 Perturbations. We thank the reviewer for the valuable comments. We would first like to clarify the definition of attribution in our paper. We attribute a predicted failure to the phase (alignment, reasoning, or decoding) whose score contributes most to STAGE's failure prediction. Because each score reads only its own stage, this tells which phase in the VLA leads to the failure prediction, not the external reason why the failure happens (e.g., an occluded object, an ambiguous instruction, or a slipped grasp). Under this definition, the perturbation test in Sec. 5.5 (Appendix H) shows that our attribution corresponds to the information flow.

To further test the attribution on actual failures, we therefore added controlled failures with known interventions, in which failures are injected at a known point upstream of the decoder. We use OpenVLA on the LIBERO dataset and include two kinds of intervention: instruction paraphrases (LIBERO-Para[3], available for LIBERO-Goal) and object-layout changes (LIBERO-Plus, all four suites). For each rollout in which the perturbation causes a failure (for paraphrases, the original run succeeds and the paraphrased run from the same initial state fails; for layouts, the run fails), we compare each stage's contribution w_k·S_k to the failure logit before and after the perturbation, over the steps both runs completed. Each comparison is thus a counterfactual pair: the same initial state, with only the instruction or the layout changed.

Table 5. Paraphrase perturbation using OpenVLA

LIBERO-Goal, paraphraseContribution change
Alignment+0.18±0.23
Reasoning+0.95±0.67
Decoding−0.02±0.21

Table 6. Layout perturbation using OpenVLA

Object layout perturbationAlignment changeReasoning changeDecoding change
LIBERO-Spatial−0.16±0.34+0.16±0.11−0.00±0.11
LIBERO-Object−0.01±0.03+0.57±0.38−0.03±0.02
LIBERO-Goal+0.08±0.12+0.50±0.38−0.03±0.28
LIBERO-10+0.10±0.08+0.12±0.07−0.03±0.08

The responses match what each perturbation changes. A paraphrase changes how the target is described, so the model must re-identify the referred object and reason about where it is; accordingly, reasoning rises with alignment. A layout change keeps the same objects and instruction but rearranges them. This, in general, focuses more on reasoning instead of alignment. Reasoning rises in every suite (+0.12 to +0.57) and clearly exceeds alignment on LIBERO-Object (+0.57 vs. −0.01) and LIBERO-Goal (+0.50 vs. +0.08), while on LIBERO-Spatial and LIBERO-10 the reasoning and alignment changes are within seed variation. In both cases, the decoder is untouched, and the mean decoding change stays within ±0.03. On average, failures injected upstream are thus attributed to upstream stages rather than to the decoder. We also report an additional closed-loop study, please refer to the pTMQ W1 response. Due to time and computational resource limitations, we are still running instruction–object experiments and will report them when they are ready.

W2-b, Q7 Figure 3 normalization. In Fig. 3, each phase score is divided by its own maximum score in a rollout. This normalization is for visualization only to easily see the trend of each score, so it will not be used in online tasks. STAGE's detector uses the unnormalized scores and never sees them. We also corrected the typos in Fig. 2 in the updated draft.

W2-c Attribution stability across seeds. To test stability, we score the same failed test rollouts with the three seeds' STAGE models and check, at every step after the failure onset, whether all three name the same top stage. To make the seeds comparable, scores are first converted to percentiles among successful rollouts of the same task. As a reference, we also compare the seeds on different rollouts of the same task and outcome; agreement there can come only from the task and outcome, not from what happens in the rollout.

Table 7. Attribution stability test using OpenVLA on LIBERO dataset

OpenVLA, all 3 seeds agree (%)SpatialObjectGoalLIBERO-10
Same rollout49.854.418.133.0
Different rollouts, same task (reference)27.631.814.617.0

In 3 of the 4 suites, agreement on the same rollout is clearly higher than the reference, so the seeds respond to what happens in that rollout rather than just to the task.

Reviewer_yVCQOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies failure prediction for Vision-Language-Action (VLA) models from a stage-wise perspective. The key observation is that existing failure detectors produce a single scalar score for the whole policy and therefore cannot identify where in the VLA pipeline the failure signal arises. The authors first develop a theoretical analysis showing that predictive action uncertainty admits an exact three-phase decomposition — alignment, reasoning, and decoding — using nested log-linear probe families, carefully avoiding the degeneracy of Shannon information-theoretic decompositions under a frozen deterministic backbone. Building on this, they show that failure predictability (in log-loss) decomposes into nonnegative per-phase gains. Motivated by these results, they propose STAGE, a lightweight framework that reads one score from each VLA stage (the decoding score being the exact policy entropy), aggregates them over the action chunk, and predicts episode failure via a linear head restricted to these scores alone. STAGE is trained with episode-level labels only, requires a single forward pass of the frozen VLA, and achieves the highest AUROC in most settings across three VLA families (OpenVLA, π0, π0-FAST), four LIBERO suites, SimplerEnv, and real-robot Agilex rollouts, with gains of up to 10.6 points over the strongest baseline. Stage-targeted perturbation experiments confirm that each phase score is driven by its own stage, validating the attribution.

Critical strengths
  1. Well-motivated problem formulation. Moving from holistic failure scoring to stage-wise attribution addresses a real gap: a single failure score cannot tell whether the policy failed to align the scene with the instruction, reasoned incorrectly, or decoded an uncertain action. This diagnostic capability is arguably as important as detection accuracy for deploying VLAs safely.
  2. Theory that directly shapes the method. The probe-based three-phase decomposition (Theorem 3.1) is non-trivial and elegantly circumvents the collapse of Shannon decompositions for frozen deterministic pipelines (the reasoning term vanishes identically; Appendix B). Proposition 3.2 then legitimates attribution via nonnegative per-phase gains. Crucially, the STAGE architecture is not loosely inspired by the theory — the restriction of the failure head to chunk-level phase scores alone is a direct structural consequence of Proposition 3.2, which is what makes the attribution meaningful.
Critical weaknesses
  1. Attribution is never used to improve anything. The phase-wise diagnosis — the paper's central claim to novelty — is only validated descriptively (perturbation responses in Appendix H; a post-hoc case study in Figure 3). It is never fed back into any decision process: no intervention, correction, replanning, or data-selection experiment demonstrates that knowing where the failure signal arises leads to better outcomes than a well-calibrated holistic score. Given that the paper motivates failure prediction with safety concerns (collisions, irrecoverable states), the absence of any closed-loop study is a significant gap. The attributed phase may correlate with where an error becomes visible in the representation rather than identify its actionable cause.
  2. The theoretical guarantees hold only at the population level with optimal probes. The nonnegativity of the alignment and reasoning terms (Theorem 3.1) is established in expectation under the optimal nested probes. STAGE's learned readers are finite-sample approximations implemented as small MLPs, so the nonnegativity and exact additivity the theory promises do not hold in the deployed system. The paper is honest about this (it calls it a "population property"), but the practical method is therefore only loosely connected to the theory — the theoretical results motivate the architecture rather than guarantee its behavior.
Questions for authors

See Weakness.

5 comments

We thank the reviewer for the positive assessment and the valuable feedback. We address the comments as follows:

W1. Attribution in closed loop. We thank the reviewer for the valuable comments. We would first like to clarify the definition of attribution in our paper. We attribute a predicted failure to the phase (alignment, reasoning, or decoding) whose score contributes most to STAGE's failure prediction. Because each score reads only its own stage, this tells which phase in the VLA leads to the failure prediction, not the external reason why the failure happens (e.g., an occluded object, an ambiguous instruction, or a slipped grasp).

Thus, in our paper, we show the perturbation experiments in Appendix Figure 5 to show that our attribution can coordinate with each phase-targeted perturbation. Furthermore, we ran a closed-loop experiment in which STAGE's attribution selects a corrective prompt hint. We ran OpenVLA on 50 randomly selected LIBERO-10 test episodes, each started from the exact initial state of its recorded test rollout. At every step, STAGE scores the action chunk before it is executed. When the predicted failure probability exceeds 0.5, we take the phase with the largest contribution, re-query OpenVLA on the same observation with a phase-specific hint appended to the instruction, and execute the hinted action instead. For alignment, the hint asks the policy to "look carefully at <task objects>"; for reasoning, it spells out the plan as "first <sub-step 1>, then <sub-step 2>". The threshold and the hints are fixed throughout, and the alarm fires on about 32% of steps.

Table 1. Closed-loop test performance using OpenVLA on LIBERO-10

armsuccess rate
recorded (baseline)54%
per-step hint58%

The phase-routed hint raises the success rate by 4 points on average. Because of the limited rebuttal time, this study is small, so we see it as preliminary evidence that the attribution can guide a corrective action. Designing a stronger remedy for each phase (e.g., re-grounding for alignment and replanning for reasoning) and running a larger study are left for future work.

W2 Theory vs. the deployed system. We thank the reviewer for the constructive feedback. We would like to clarify that the theorem motivates the design of STAGE rather than describing the behavior of its trained readers. The key factors between the theory and the deployed system can arise in two places. The first is (i) finite samples: the theorem holds in expectation, so individual steps may deviate from it. The second is (ii) learned readers: STAGE's readers are trained MLPs rather than the optimal probes. To see how large the impact of the two factors is in practice, we measured both on held-out data, and both turn out to be small on average.

Factor (i): finite samples. To check whether the population claims of Theorem 3.1 hold on held-out data, we took the OpenVLA rollouts of the four LIBERO suites and fit the nested log-linear probes of Def. 3.1 to the policy's own π, without any failure labels. Here q0q_0 sees the context (task, elapsed time, and previous action tokens), and q1q_1 additionally sees Z. We then computed s1s_1, s2s_2 and s3s_3 at every step with 5-fold cross-fitting over episodes.

Table 2. Theoretical calculation of each phase estimation.

OpenVLA suiteE[s1s_1]steps with s1s_1 negativeE[s2s_2]E[s3s_3]
LIBERO-Spatial1.4228.4%18.262.42
LIBERO-Object1.5823.4%13.872.46
LIBERO-Goal1.0932.0%17.192.21
LIBERO-101.7921.1%16.202.56

Values in nats, summed over the 7 action tokens. The results agree with the theory. The expected alignment term is positive in every suite, while s2s_2 and s3s_3 are nonnegative by construction. The gap appears only at the level of individual steps, where 21–32% of steps have a negative s1s_1, which is exactly the kind of deviation a population statement allows.

Factor (ii): learned readers. We next asked whether Prop. 3.2 still holds once the optimal probes are replaced by STAGE's trained readers. For OpenVLA on the four LIBERO suites, we trained STAGE on every subset of the three phases with 3 seeds and measured the failure log loss on test episodes, after a cross-fitted recalibration. Prop. 3.2 predicts that adding a phase k to any subset S never increases the loss, so we checked this prediction for every nested pair.

Table 3. Nonnegativity violation rate in real-world models.

OpenVLA suitepairs consistent with Prop. 3.2violation ratio
LIBERO-Spatial92% (33/36)1.4%
LIBERO-Object67% (24/36)7.0%
LIBERO-Goal81% (29/36)15.9%
LIBERO-1078% (28/36)10.9%

Each suite has 36 nested pairs (12 pairs × 3 seeds), all of which the theory predicts to be consistent. The violation ratio divides the total size of the negative gains by the total size of all gains, so 0% means no gap. Overall, 79% of the pairs are consistent with Prop. 3.2, and the violations account for only 7.9% of the total gain. The trained readers therefore follow the theory closely.

What STAGE takes from the theory. Finally, we want to clarify the connection of the theory and the algorithm. Theorem 3.1 describes the policy's own action, but it is failure-agnostic. STAGE therefore uses it to decide which representation each phase reads (ZZ, FF and π\pi) and adopts S3=∑hH(πh)S_3=\sum_hH(\pi_h) exactly, but learns S1S_1 and S2S_2 from failure labels, as the policy's own understanding does not necessarily lead to actual failure in the real world. This choice matters in practice when we use the theory terms directly as failure scores, following the protocol of the paper's Tables 1–4 on each seed's own test split, they perform poorly, as shown in Table 4 below in LIBERO using OpenVLA. The theory terms stay close to chance and sometimes even point the wrong way, whereas STAGE leads the better of the two by 14–36 points in every suite and split. This is why STAGE keeps the phase structure given by the theory but learns failure-relevant scores within it.

Table 4. AUROC for failure detection using theoretical estimation.

AUROCSpatialObjectGoalLIBERO-10
All test episodes
theory s1s_152.51±1.4558.78±0.6952.53±0.1748.43±0.28
theory s2s_256.20±0.6439.03±1.0061.79±0.5744.63±0.26
STAGE87.35±1.1682.48±1.9882.58±1.6668.73±1.11
Unseen tasks
theory s1s_152.23±3.5957.09±3.0757.98±4.9543.94±0.98
theory s2s_266.40±1.5735.00±9.2268.27±3.4136.55±2.62
STAGE85.85±2.2777.09±4.3382.62±4.3164.28±2.00
Seen tasks
theory s1s_154.32±1.9160.72±0.6949.61±3.1350.51±1.01
theory s2s_251.33±1.6241.89±1.5056.55±1.3349.03±1.09
STAGE90.01±1.5687.22±1.9184.48±0.5271.10±1.18

Dear Reviewer yVCQ,

Thank you again for your review. As the discussion period is closing, we want to check whether our response addresses your concerns. In summary:

For W1, we clarified that attribution tells which VLA phase a failure becomes predictable from, not the external cause of the failure. We also ran a closed-loop study with OpenVLA on LIBERO-10. In it, STAGE's attributed phase selects a corrective prompt hint, and success rises from 54% to 58%.

For W2, we measured both sources of the gap on held-out data. For finite samples, the expected alignment term is positive in every suite. Only 21–32% of individual steps deviate, which a population-level statement allows. For learned readers, 79% of nested phase pairs are consistent with Prop. 3.2, and violations make up only 7.9% of the total gain. We also show that the theory terms, used directly as failure scores, stay near chance, while STAGE leads by 14–36 AUROC points. This is why STAGE keeps the phase structure from the theory but learns its scores from failure labels.

In addition, we ran a larger closed-loop study with π0\pi0 on LIBERO-Object, covering all test episodes over 3 seeds. The phase-routed hint raises success from 88.1% to 91.1%.

Table 5. Closed-loop test performance using π0\pi0 on LIBERO-Object

Strategyallseen tasksunseen tasks
no hint (baseline)88.1%89.4%86.2%
hint91.1%93.7%87.6%

We would be glad to clarify anything that remains open.

Comments on the paper

6 comments

We thank the reviewer for the positive assessment and the valuable feedback. We address the comments as follows:

W1, Q1 Evidence for stage attribution. We thank the reviewer for this important point. We first state precisely what the attribution claims, and then provide three new pieces of evidence for it. We attribute a predicted failure to the phase (alignment, reasoning, or decoding) whose score contributes most to STAGE's failure prediction. Because each score reads only its own stage, this identifies which phase in the VLA leads to the failure prediction, not the external reason why the failure happens (e.g., an occluded object, an ambiguous instruction, or a slipped grasp).

(i) Why the scores are learned rather than taken from Theorem 3.1. We want to clarify that Theorem 3.1 motivates the design of STAGE. Theorem 3.1 concerns the policy's own action rather than failure, so STAGE takes from it only which representation each phase reads, while S1S_1 and S2S_2 are learned from failure labels. Used directly as failure scores, the theory terms are weak, staying close to chance or even below it, whereas STAGE is 14–36 points higher in every suite and split (Table 1).

Table 1. AUROC for failure detection using theoretical estimation.

AUROCSpatialObjectGoalLIBERO-10
All test episodes
theory s1s_152.51±1.4558.78±0.6952.53±0.1748.43±0.28
theory s2s_256.20±0.6439.03±1.0061.79±0.5744.63±0.26
STAGE87.35±1.1682.48±1.9882.58±1.6668.73±1.11
Unseen tasks
theory s1s_152.23±3.5957.09±3.0757.98±4.9543.94±0.98
theory s2s_266.40±1.5735.00±9.2268.27±3.4136.55±2.62
STAGE85.85±2.2777.09±4.3382.62±4.3164.28±2.00
Seen tasks
theory s1s_154.32±1.9160.72±0.6949.61±3.1350.51±1.01
theory s2s_251.33±1.6241.89±1.5056.55±1.3349.03±1.09
STAGE90.01±1.5687.22±1.9184.48±0.5271.10±1.18

(ii) Controlled failures with known interventions. The perturbations in the draft replay logged actions, so they test information flow mainly. To test the attribution on real failures, we run OpenVLA from all test initial states with 3 seeds and inject failures upstream of the decoder in two ways: instruction paraphrases from LIBERO-Para [1] on LIBERO-Goal, and object-layout changes from LIBERO-Plus on all four suites. For each rollout in which the perturbation causes a failure, we compare each stage's contribution to the failure logit before and after the perturbation, over the steps both runs completed. Each comparison is thus a counterfactual pair from the same initial state, with only the instruction or the layout changed.

Table 2. Paraphrase perturbation using OpenVLA

LIBERO-Goal, paraphraseContribution change
Alignment+0.18±0.23
Reasoning+0.95±0.67
Decoding−0.02±0.21

Table 3. Layout perturbation using OpenVLA

Object layout perturbationAlignment changeReasoning changeDecoding change
LIBERO-Spatial−0.16±0.34+0.16±0.11−0.00±0.11
LIBERO-Object−0.01±0.03+0.57±0.38−0.03±0.02
LIBERO-Goal+0.08±0.12+0.50±0.38−0.03±0.28
LIBERO-10+0.10±0.08+0.12±0.07−0.03±0.08

Change in logit units, and we list mean ± std over 3 seeds. The responses match what each intervention changes. A paraphrase changes how the target is described, so the model must re-identify the referred object and reason about where it is; accordingly, both the alignment and the reasoning contributions rise (+0.18 and +0.95). A layout change keeps the same objects and instruction but rearranges them, which mainly affects reasoning: reasoning rises in every suite (+0.12 to +0.57) and clearly exceeds alignment on LIBERO-Object (+0.57 vs. −0.01) and LIBERO-Goal (+0.50 vs. +0.08), while on LIBERO-Spatial and LIBERO-10 the two changes are within seed variation. In both cases, the decoder is untouched, and the mean decoding change stays within ±0.03. Failures injected upstream are thus attributed, on average, to upstream stages rather than to the decoder.

(iii) Attribution vs. predictive information available at a stage. We want to clarify that what distinguishes attribution from "information available at that stage" is that the forward pass is sequential: everything readable from ZZ is also present in FF, so information available at the reasoning stage always includes the alignment stage. STAGE's attribution therefore measures incremental information, i.e., the gain of a phase given the other phases (Prop. 3.2). These gains are stage-specific (OpenVLA, held-out log loss on test data, each phase added last; full table in Table 4): alignment adds information beyond reasoning and decoding in all four suites (+6.9 to +32.5 millinats), reasoning adds beyond alignment and decoding in three of four (+23.8 to +55.0), and decoding adds at most +4.5. Under counterfactual interventions placed upstream (Tables 2–3), the decoding contribution stays unchanged while the reasoning contribution rises.

Table 4. Order influence evaluation

OpenVLA, gain A / R / DShapley (order-averaged)added firstadded last
LIBERO-105.9 / 19.9 / 2.94.8 / 19.7 / 5.710.6 / 23.8 / 3.7
LIBERO-Spatial35.4 / 39.8 / 14.024.9 / 19.5 / 10.032.5 / 46.7 / 4.5
LIBERO-Object32.6 / 66.5 / 1.042.8 / 65.0 / −0.19.5 / 55.0 / −10.8
LIBERO-Goal25.5 / 21.1 / 1.230.5 / 31.6 / 9.16.9 / −3.2 / −20.4

A / R / D = alignment / reasoning / decoding. Gain in held-out log loss on each seed's test episodes over 3 seeds. "Added last" is the log-loss version of the paper's Table 4's leave-one-stage-out drop.

Taken together, (ii) and (iii) show that the attribution follows where a failure is injected and measures what each stage adds beyond the others.

Furthermore, we ran a closed-loop experiment in which STAGE's attribution selects a corrective prompt hint. We ran OpenVLA on 50 randomly selected LIBERO-10 test episodes, each started from the exact initial state of its recorded test rollout. At every step, STAGE scores the action chunk before it is executed. When the predicted failure probability exceeds 0.5, we take the phase with the largest contribution, re-query OpenVLA on the same observation with a phase-specific hint appended to the instruction, and execute the hinted action instead. For alignment, the hint asks the policy to "look carefully at <task objects>"; for reasoning, it spells out the plan as "first <sub-step 1>, then <sub-step 2>". The threshold and the hints are fixed throughout, and the alarm fires on about 32% of steps.

Table 5. Closed-loop test performance using OpenVLA on LIBERO-10

armsuccess rate
recorded (baseline)54%
per-step hint58%

The phase-routed hint raises the success rate by 4 points on average. Because of the limited rebuttal time, this study is small, so we see it as preliminary evidence that the attribution can guide a corrective action. Designing a stronger remedy for each phase (e.g., re-grounding for alignment and replanning for reasoning) and running a larger study are left for future work.

W2, Q1 Decoding score for continuous decoders. A flow-matching decoder such as π0\pi0's has no token entropy, so we follow the sample-variance uncertainty metric of SAFE [2]. At step tt, we get NN sampled actions from the same observation with different initial noise, aNta_N^t, with mean aˉt\bar a_{t} across the samples. The decoding score is the total sample variance, i.e., the trace of the sample covariance, which is exactly SAFE's "total_var": tr⁡(1N−1∑n=1N(ant−aˉt)(ant−aˉt)⊤)\operatorname{tr}\Big(\frac{1}{N-1}\sum_{n=1}^{N}\big( a_n^t-\bar a_t\big)\big(a_n^t-\bar a_t\big)^{\top}\Big) STAGE keeps the per-DoF per-action-chunk terms S3,htS^t_{3,h}. Also we preserve the per-action variance$\frac{1}{N}\sum_{j=1}^{N}\big(a_{j}^t-\bar a_{t}\big)^2$ and aˉt=1N∑j=1Najt\bar a_{t}=\frac{1}{N}\sum_{j=1}^{N}a_{j}^t.

W3 How AUROC is computed. We would like to clarify that the eventual episode label is assigned to every step only for training. The reported AUROC is computed over episodes, not pooled over steps. Following the SAFE early-alert protocol, each episode's score is the cumulative failure probability at a cutoff equal to the shortest rollout of that task. Every episode, whether it later succeeds or fails, is still running at that point, so no post-failure steps are used. We also report how many failures each method detects at a fixed 10% false-alarm rate in Table 6. At a fixed 10% false-alarm rate, STAGE catches the most failures in most of the settings, with margins of up to 14.1 points over the second-best method (56.9% vs. 42.8% on OpenVLA LIBERO-Goal). And due to the computational and resource limits, we are still running more analyses and will show them when they are ready.

Table 6. Failures detected at fixed false-alarm rate.

Failures detected (%) at 10% false alarmsL-10SpatialObjectGoal
OpenVLA
STAGE33.966.261.356.9
Mahalanobis20.835.223.728.4
Euclidean k-NN22.043.929.630.9
Cosine k-NN22.548.635.835.7
RND13.624.019.015.8
LogpZO15.123.324.330.6
SAFE-LSTM25.863.743.038.3
SAFE-MLP27.567.659.042.8
INSIGHT15.727.615.730.2
π0\pi0
STAGE58.919.740.959.6
Mahalanobis45.95.434.358.2
Euclidean k-NN42.210.858.344.1
Cosine k-NN37.010.861.641.6
RND36.712.449.254.9
LogpZO46.012.556.348.3
SAFE-LSTM47.216.123.647.9
SAFE-MLP51.814.342.758.2
INSIGHT53.710.738.850.6
π0\pi0-FAST
STAGE82.576.892.678.2
Mahalanobis74.962.044.671.8
Euclidean k-NN80.567.779.579.2
Cosine k-NN85.468.076.479.0
RND80.958.763.376.4
LogpZO84.070.268.370.3
SAFE-LSTM85.666.075.181.4
SAFE-MLP53.736.628.838.3
INSIGHT85.454.572.361.6

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.