- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether action chunks from a frozen Vision-Language-Action (VLA) model can predict episode failure and trace it to alignment, reasoning, or decoding. The authors introduce STAGE, which extracts one score per stage and combines them in a lightweight logistic predictor trained on episode-level outcomes. Grounded in theoretical analyses of action uncertainty and failure predictability, STAGE is evaluated on OpenVLA, \pi_0, and \pi_0-FAST across LIBERO, SimplerEnv, and 201 real-robot Agilex rollouts. It achieves the highest AUROC in 11 of 14 simulation settings and substantially outperforms baselines on the real robot. The key question is whether its scores truly identify failure stages or form an effective three-feature predictor.
Critical strengths
- The paper presents a simple, effective design. Using just three stage-specific scores makes ablations straightforward, requires only one pass through the frozen VLA, and avoids stage-level supervision.
- It evaluates three VLA families, both discrete and continuous decoders, several simulation benchmarks, and real-robot rollouts. The appendices include seen- and unseen-task results and discuss key theoretical limitations, including the collapse of the Shannon decomposition.
- STAGE outperforms all baselines across the five OpenVLA settings, with a particularly strong gain on LIBERO-Goal (82.58 vs. 74.05 for SAFE-MLP). Table 10’s token-uncertainty results also support using internal representations rather than decoder uncertainty alone.
Critical weaknesses
- Theorem 3.1 concerns optimal log-linear probes of action distributions, whereas STAGE learns alignment and reasoning scores from failure labels. So its guarantees of nonnegativity and decomposition do not apply to STAGE’s learned scores. Proposition 3.2 follows from nested predictor families in general, not specifically from VLA stages. Its gains depend on stage ordering and differ from the experimental leave-one-stage-out AUROC results. Appendix B also shows that Shannon information assigns no additional information to reasoning, since the final hidden state is determined by earlier representations. Any nonzero reasoning contribution therefore depends on the restricted probe family. Finally, it is unclear whether the theory applies to \pi_0 and its compressed \pi_0-FAST action tokens.
- Perturbation experiments show that each score responds to changes at its own stage and possibly earlier stages, but not later ones, since changes to decoder temperature cannot affect scores based on earlier representations. This confirms the intended information flow, but not that the scores identify causes of failure. Evidence remains limited: there are no ground-truth stage labels or counterfactual interventions, and Figure 3 offers only one qualitative example. Unclear normalization also leaves open whether the visualization uses information from future rollouts. Overall, the evidence does not support the claim that STAGE reveals where failures originate.
- The results focus mainly on AUROC, leaving out AUPRC, calibration, false-alarm rates, and early-detection metrics—all important for runtime failure detection, especially when benchmark success rates differ. The largest gain (+10.6 points on \pi_0 LIBERO-Spatial) still reaches only 64.37 AUROC. Combining seen and unseen tasks can hide generalization gaps: \pi_0’s LIBERO-10 AUROC drops from 91.11 on seen tasks to 58.46 ± 11.33 on unseen tasks. SimplerEnv also combines its two embodiments. Reporting these results separately would give a clearer picture of practical reliability.
- The paper is missing key ablations and implementation details. Its phase ablation covers only OpenVLA LIBERO-Spatial, and the single-stage results vary widely (±13–14 points) compared with ±1.16 for the full model. Without a capacity-matched unrestricted baseline using the same internal representations, it is hard to separate the effects of stage-wise restrictions from those of learned readers. The paper also says the continuous-decoder score is based on predicted-action statistics but gives no formula, limiting reproducibility of the \pi_0 and real-robot results. Finally, the reported Agilex gain (98.25 vs. 68.88 AUROC) is based on just 60 test rollouts across three seen tasks, with uncertainty measured across seeds rather than episodes.
Questions for authors
- Please provide the exact formula for the decoding score (S_{3,h}) used with \pi_0/open-pi-zero and Agilex, and include a sensitivity analysis.
- Compare STAGE with a capacity-matched predictor trained on the same alignment and reasoning representations but without the three-score restriction, ideally using both OpenVLA and \pi_0. How much does the restriction contribute?
- Give greater prominence to unseen-task AUROC and AUPRC. Also report an early-detection metric, such as AUROC over the first 25% or 50% of an episode, or lead time before failure.
- Test failures caused by known interventions, such as instruction–object mismatches or spatial rearrangements, and assess whether the expected stage score increases. Also examine attribution stability across seeds and stage orderings.
- Confirm that Agilex splits are episode-disjoint. Report per-task AUROC, the number of test-set failures, and episode-bootstrap confidence intervals for STAGE and SAFE-MLP.
- Explain why individual-stage predictors in Table 4 have much higher variance than the full model, and extend the ablation to another backbone, preferably \pi_0.
- Define the normalization used in Figure 3 and clarify whether it can be computed online. Figure 2 also appears to retain the label "TRACE Output”; please correct it.
9 comments
W1-b. Stage ordering. Thanks to the reviewers for the valuable feedback. We would first like to clarify where an order enters our analysis. It appears only in Theorem 3.1, and only because the VLA computes its stages in sequence: the nested probes read C, then (C, Z), then (C, Z, F) (Def. 3.1, lines 146–151), following the forward pass Z → F → π (Eq. 2, line 140). This order is therefore fixed by the architecture rather than chosen by us. Prop. 3.2, in contrast, makes no assumption about order and holds "for any ordering k₁, k₂, k₃ of the phases" (line 238). STAGE's head is likewise symmetric, , and is trained jointly on all three scores (Eq. 10, lines 320–323), so its detection AUROC does not depend on any ordering either.
Prop. 3.2 complements Theorem 3.1 rather than restating it. Theorem 3.1 splits the policy's action uncertainty along the pipeline, using nested probe families that read Z, then F, then . Prop. 3.2 carries the same nesting over to failure prediction: predictors that read more phases contain those that read fewer (lines 236–237), so adding a phase can only lower the population log loss. Unlike Theorem 3.1, it is decoder-agnostic and order-free, and therefore covers all three backbones. We use it because it makes per-phase attribution well-defined: when the predictor sees only the three phase scores, removing one score measures that phase's contribution, and this gain is guaranteed to be non-negative (lines 245–250).
We would also appreciate a clarification of what "stage ordering" refers to. Our understanding is that the gain of a phase in Eq. (9) depends on which phases were added before it, whereas the paper's Table 4's leave-one-stage-out drops measure only one ordering (each phase added last), and in AUROC rather than log loss. Therefore, we report the gains under all orderings in Table 4 below:
Table 4. Order influence evaluation
| OpenVLA, gain A / R / D | Shapley (order-averaged) | added first | added last |
|---|---|---|---|
| LIBERO-10 | 5.9 / 19.9 / 2.9 | 4.8 / 19.7 / 5.7 | 10.6 / 23.8 / 3.7 |
| LIBERO-Spatial | 35.4 / 39.8 / 14.0 | 24.9 / 19.5 / 10.0 | 32.5 / 46.7 / 4.5 |
| LIBERO-Object | 32.6 / 66.5 / 1.0 | 42.8 / 65.0 / −0.1 | 9.5 / 55.0 / −10.8 |
| LIBERO-Goal | 25.5 / 21.1 / 1.2 | 30.5 / 31.6 / 9.1 | 6.9 / −3.2 / −20.4 |
A / R / D = alignment / reasoning / decoding. Gain in held-out log loss on each seed's test episodes, with the mean over 3 seeds. "Added last" is the log-loss version of the paper's Table 4's leave-one-stage-out drop. The ordering does shift credit between alignment and reasoning on Spatial and Goal, but it does not change the conclusion: the top stage is alignment or reasoning, and decoding has the smallest gain. So our findings do not depend on a particular stage order.
W1-c The reasoning term under Shannon information. We thank the reviewer for the comments. We would like to clarify that a frozen VLA is deterministic: once the image and instruction are given, every intermediate representation, and hence the decoder output π, is fixed, so Shannon information gives no credit to the intermediate stages. This does not mean, however, that the stages carry no signal. The decoder entropy summarizes the whole pipeline in a single number and therefore cannot tell which stage left the action uncertain and failed. Separating the stages requires reading and directly with a reader of limited capacity, one that cannot simply recompute the network. This is the idea of usable information[1], and it is also the setting that matters in practice, since a runtime failure monitor is exactly such a limited reader.
W1-d, Backbone coverage. We'd like to clarify that our theory applies to the OpenVLA and pi0-fast models. Theorem 3.1 requires a softmax linear head (Assumption 3.1, lines 156–161). It therefore applies to OpenVLA (line 160) and to -FAST, whose FAST tokens come from the same kind of head, although for -FAST the entropy is over the compressed tokens rather than physical actions (lines 108–112). It does not apply to , whose flow-matching decoder has no token softmax and generates the chunk jointly, as the draft states (lines 113–114, 304–307). For $ \ pi_0$, we instead follow SAFE [2] and replace the token entropy with a statistic of the sampled actions as an adaptation to test our method.
W2-a, Q4 Perturbations. We thank the reviewer for the valuable comments. We would first like to clarify the definition of attribution in our paper. We attribute a predicted failure to the phase (alignment, reasoning, or decoding) whose score contributes most to STAGE's failure prediction. Because each score reads only its own stage, this tells which phase in the VLA leads to the failure prediction, not the external reason why the failure happens (e.g., an occluded object, an ambiguous instruction, or a slipped grasp). Under this definition, the perturbation test in Sec. 5.5 (Appendix H) shows that our attribution corresponds to the information flow.
To further test the attribution on actual failures, we therefore added controlled failures with known interventions, in which failures are injected at a known point upstream of the decoder. We use OpenVLA on the LIBERO dataset and include two kinds of intervention: instruction paraphrases (LIBERO-Para[3], available for LIBERO-Goal) and object-layout changes (LIBERO-Plus, all four suites). For each rollout in which the perturbation causes a failure (for paraphrases, the original run succeeds and the paraphrased run from the same initial state fails; for layouts, the run fails), we compare each stage's contribution w_k·S_k to the failure logit before and after the perturbation, over the steps both runs completed. Each comparison is thus a counterfactual pair: the same initial state, with only the instruction or the layout changed.
Table 5. Paraphrase perturbation using OpenVLA
| LIBERO-Goal, paraphrase | Contribution change |
|---|---|
| Alignment | +0.18±0.23 |
| Reasoning | +0.95±0.67 |
| Decoding | −0.02±0.21 |
Table 6. Layout perturbation using OpenVLA
| Object layout perturbation | Alignment change | Reasoning change | Decoding change |
|---|---|---|---|
| LIBERO-Spatial | −0.16±0.34 | +0.16±0.11 | −0.00±0.11 |
| LIBERO-Object | −0.01±0.03 | +0.57±0.38 | −0.03±0.02 |
| LIBERO-Goal | +0.08±0.12 | +0.50±0.38 | −0.03±0.28 |
| LIBERO-10 | +0.10±0.08 | +0.12±0.07 | −0.03±0.08 |
The responses match what each perturbation changes. A paraphrase changes how the target is described, so the model must re-identify the referred object and reason about where it is; accordingly, reasoning rises with alignment. A layout change keeps the same objects and instruction but rearranges them. This, in general, focuses more on reasoning instead of alignment. Reasoning rises in every suite (+0.12 to +0.57) and clearly exceeds alignment on LIBERO-Object (+0.57 vs. −0.01) and LIBERO-Goal (+0.50 vs. +0.08), while on LIBERO-Spatial and LIBERO-10 the reasoning and alignment changes are within seed variation. In both cases, the decoder is untouched, and the mean decoding change stays within ±0.03. On average, failures injected upstream are thus attributed to upstream stages rather than to the decoder. We also report an additional closed-loop study, please refer to the pTMQ W1 response. Due to time and computational resource limitations, we are still running instruction–object experiments and will report them when they are ready.
W2-b, Q7 Figure 3 normalization. In Fig. 3, each phase score is divided by its own maximum score in a rollout. This normalization is for visualization only to easily see the trend of each score, so it will not be used in online tasks. STAGE's detector uses the unnormalized scores and never sees them. We also corrected the typos in Fig. 2 in the updated draft.
W2-c Attribution stability across seeds. To test stability, we score the same failed test rollouts with the three seeds' STAGE models and check, at every step after the failure onset, whether all three name the same top stage. To make the seeds comparable, scores are first converted to percentiles among successful rollouts of the same task. As a reference, we also compare the seeds on different rollouts of the same task and outcome; agreement there can come only from the task and outcome, not from what happens in the rollout.
Table 7. Attribution stability test using OpenVLA on LIBERO dataset
| OpenVLA, all 3 seeds agree (%) | Spatial | Object | Goal | LIBERO-10 |
|---|---|---|---|---|
| Same rollout | 49.8 | 54.4 | 18.1 | 33.0 |
| Different rollouts, same task (reference) | 27.6 | 31.8 | 14.6 | 17.0 |
In 3 of the 4 suites, agreement on the same rollout is clearly higher than the reference, so the seeds respond to what happens in that rollout rather than just to the task.
We thank the reviewer for the positive assessment and the valuable feedback. We address the comments as follows:
W1-a. Connection between theory and algorithm. We thank the reviewer for the constructive feedback. We would like to clarify the connection between the theory and the algorithm: the theorem motivates the design of STAGE rather than describing the behavior of its trained readers. The gap between the two can arise in two places. The first is (i) finite samples: the theorem holds in expectation, so individual steps may deviate from it. The second is (ii) learned readers: STAGE's readers are trained MLPs rather than the optimal probes. Not every term is affected, however. Since s2=KL(π∥q1) and s3=H(π) are nonnegative at every step for any probe, only the alignment term relies on the population, optimal-probe assumption. We measured both gaps on held-out data and found them small on average.
Factor (i): finite samples. To check whether the population claims of Theorem 3.1 hold on held-out data, we took all OpenVLA rollouts of the four LIBERO suites and fit the nested log-linear probes of Def. 3.1 to the policy's own action distribution, without any failure labels. We then computed s1, s2 and s3 at every step with 5-fold cross-fitting over episodes.
Table 1. Theoretical calculation of each phase estimation.
Values in nats, summed over the 7 action tokens. The results agree with the theory: the expected alignment term is positive in every suite, while s2 and s3 are nonnegative by construction. The gap appears only at the level of individual steps, where 21–32% of steps have a negative s1.
Factor (ii): learned readers. We next asked whether Prop. 3.2 still holds when the optimal probes are replaced by STAGE's trained readers. For OpenVLA on the four LIBERO suites, we trained STAGE on every subset of the three phases with 3 seeds and measured the failure log loss on test episodes after a cross-fitted recalibration. Prop. 3.2 predicts that adding a phase k to any subset S never increases the loss, so we checked this prediction for every nested pair (S,S∪{k}).
Table 2. Nonnegativity violation rate in real-world models.
Each suite has 36 nested pairs (12 pairs × 3 seeds), all of which the theory predicts to be consistent. The violation ratio divides the total size of the negative gains by the total size of all gains, so 0% means no gap. Overall, 79% of the pairs are consistent with Prop. 3.2, and the violations account for only 7.9% of the total gain. The trained readers therefore follow the theory closely.
What STAGE takes from the theory. Finally, Theorem 3.1 concerns the policy's own action, not failure. STAGE therefore uses it to decide which representation each phase reads (Z, F and π) and adopts S3=∑hH(πh) exactly, but learns S1 and S2 from failure labels. This choice matters in practice. When we use the theory terms directly as failure scores, following the protocol of the paper's Tables 1–4 on each seed's own test set, they perform poorly (Table 3): they stay close to chance and sometimes even point the wrong way (below 50), whereas STAGE leads the better of the two by 14–36 points in every suite and split. This is why STAGE keeps the phase structure given by the theory but learns failure-relevant scores within it.
Table 3. AUROC for failure detection using theoretical estimation.