acceptodds
Under review as a conference paper at ICLR 2027

Vision-Language-Action Models Lose Information as They Fail

Abstract

Vision-language-action (VLA) models are increasingly deployed on physical robots, where failures can cause irreversible harm. This calls for failure detection that generalizes across policies and explains why a rollout fails. We start from an information-theoretic premise: effective information transfer is a prerequisite for achieving a goal. Formalizing VLA control as a closed-loop information pipeline, we derive eight information-theoretic metrics over states and actions and select three complementary ones, the Triple Information-theoretic (Tri-Info) metrics: action entropy, temporal action coherence, and action–state coupling. They characterize a healthy policy as one that acts with moderate entropy grounded in both its history and its observations, while failure violates at least one of these conditions. A GRU-based detector predicts failure from how these signals evolve. Across various VLA models, simulated benchmarks, and real-world tasks, Tri-Info matches the strongest baselines in domain and transfers across architectures, environments, and the sim-to-real gap without target-domain failure labels, reaching over 70% accuracy on real-world tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Failure Detection, Runtime Monitoring, Information Theory, Mutual Information, Robot Learning, Robot Safety, Cross-Architecture Transfer, Sim-to-Real, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tJpoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Tri-Info, an information-theoretic method for detecting failures in vision-language-action (VLA) models. It focuses on three signals in the perception–action loop: action entropy, temporal action coherence, and action–state coupling. The authors estimate these signals across sliding windows and use them in GRU-based detectors with calibrated thresholds. They evaluate six VLA policies on LIBERO, CALVIN, and both simulated and real-world ALOHA tasks. Tri-Info performs well in-domain, with an average balanced accuracy of 0.83, and generalizes more effectively across policies and environments, reaching an average out-of-distribution balanced accuracy of 0.81. The paper also presents extensive ablations and analyses of different failure modes.

Critical strengths
  • The main contribution is showing that simple information-theoretic signals outperform existing failure detectors under distribution shift. Tri-Info performs consistently across models and environments, including sim-to-real transfer, while several embedding-based baselines degrade substantially. Its focus on thresholded accuracy, not just AUC, is especially useful for deployment.
  • The authors compare alternative action statistics and input representations, and test sensitivity to model capacity and threshold calibration. Appendices report confidence intervals and discuss limitations, helping clarify which parts of the method drive its performance.
  • The detector is lightweight, and its three signals offer intuitive explanations of failures such as freezing, drifting, and phantom grasps. Results on noisy demonstrations also support its robustness.
Critical weaknesses
  • The information-theoretic interpretation is overstated. Metrics estimated from just 29 temporally correlated samples using k-NN estimators on seven-dimensional VAE latents have no meaningful absolute values (Appendix C). The results show these statistics can detect failure, but not that information loss causes it. The main method also relies on internal embeddings rather than executed actions. The promising black-box variant performs worse under distribution shift, with 0.794 versus 0.865 OOD balanced accuracy.
  • The transfer evaluation uses target-domain data. Appendix D.2 states that the VAE is trained on embeddings from all domains, including test trajectories. Although it uses no target-domain failure labels, the evaluation is transductive rather than fully zero-shot. Since cross-domain generalization is the paper's main claim, the authors should test whether the results hold when the VAE is fitted on a separate, unlabeled target-domain dataset.
  • The empirical evidence is incomplete. In-domain performance is competitive but not consistently better: Tri-Info’s average AUC is lower than SAFE’s (0.82 vs. 0.91), and combining all three signals does not always outperform the best individual metric. The full baseline comparison covers only three model–environment combinations, despite Table 1’s broader evaluation. Appendix B’s subset-search results are also nearly identical across several feature combinations, leaving the case for exactly three metrics unclear.
  • Several evaluation details limit the conclusions. For AUC, trajectories are truncated to the shortest successful rollout, but the paper does not report how much failure information this removes. Aligning failures at the detector's trigger may partly explain changes in the interpretability metric. Threshold calibration reuses training trajectories, so it provides no formal conformal coverage guarantee. At the default \alpha=0.05, the true-negative rate is 0.736, suggesting substantial false alarms. The action-selection improvement in Appendix E is not statistically significant (p = 0.148).
Questions for authors
  • Could the authors repeat the three transfer experiments with a VAE trained on a separate, unlabeled target-domain dataset that excludes all test trajectories? This would help isolate the contribution of transductive training to the reported generalization.
  • Which data splits were used for the correlation analysis in Figure 4 and the exhaustive subset search in Appendix B? Were test trajectories included? Since the subsets have similar AUCs, how stable is the selected triplet?
  • Could the authors report full Tri-Info and baseline results for the remaining CALVIN and ALOHA-sim settings? Consistent OOD averages across Tables 2, 3, and 9, along with confidence intervals in the main comparison, would also help.
  • How much of each trajectory is removed by truncation, and do the AUC results hold without it? For Figure 6, could the metric traces also be aligned to independently annotated failure onset times rather than the detector's trigger?
  • At \alpha = 0.05, about 26% of successful trajectories are falsely flagged. What operating point would the authors recommend for real-world deployment, and what alarm precision would it provide? Does the reported inference latency include VAE encoding and k-NN estimation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer3 tJpo (1/4)

We sincerely thank the reviewer tJpo for the detailed and constructive comments. We are glad that the reviewer recognizes the strong generalization of Tri-Info under distribution shift, our deployment-oriented focus on thresholded accuracy, the thoroughness of our ablations and analyses, the lightweight and interpretable design of our detector, and its robustness to noisy demonstrations. Here we address all the questions raised and hope our responses can clear up the concerns.

Q1: “Since cross-domain generalization is the paper's main claim, the authors should test whether the results hold when the VAE is fitted on a separate, unlabeled target-domain dataset.”

Thank you for the constructive suggestion.

We ran the requested experiment. We hold out the test trajectories and fit the target VAE on n∈[2,5,10,20,30,40]n \in [2, 5, 10, 20, 30, 40] of the remaining unlabeled target trajectories. The frozen source detector is then evaluated on the held-out set.

With 20 trajectories, mean balanced accuracy is 0.90 on PI0.5-LIBERO and 0.82 on ACT-ALOHA-real-cup, matching the VAE fitted on all target data (0.89 and 0.82). On PI0.5-ALOHA-real-block, it reaches 0.64, slightly below 0.69. We added these results to the revised paper (Figure 13, Appendix D).

Our main claim is unchanged: Tri-Info transfers to a new domain without labeled target data. This experiment additionally shows that a small set of unlabeled target rollouts is enough, which makes deployment more practical.

Revisions: Appendix D (“How many target trajectories does the VAE need?”, Figure 13)

Q2: “Which data splits were used for the correlation analysis in Figure 4 and the exhaustive subset search in Appendix B? Were test trajectories included? Since the subsets have similar AUCs, how stable is the selected triplet?”

Thank you for the question. The correlation analysis in Figure 4 and the exhaustive subset search in Appendix C.1 exclude all test sets, as well as the two real-world tasks we collected for OOD transfer (PI0.5-ALOHA-Real-Block and ACT-ALOHA-Real-Cup). The subset selection is therefore not fitted to the test set.

Results in the submission support the stability of the selected triplet. In Appendix E.5, OOD balanced accuracy is high, with tight trajectory-level 95% bootstrap confidence intervals, shown in brackets below (Table 17). On cross-model transfer, accuracy holds at 0.91–0.92 throughout the trajectory, with the lower bound of the 95% CI at least 0.86. On ACT-ALOHA-Real-Cup it peaks at 0.95 [0.91, 0.98], and on PI0.5-ALOHA-Real-Block it rises to 0.83 [0.74, 0.91]. In the detector capacity sweep (Table 18), AUC (mean ± std over three seeds) stays within 0.818–0.839 (std ≤ .008) on cross-model transfer and 0.852–0.910 (std ≤ .048) on sim-to-real transfer.

We also added a new multi-metric ablation on five additional policies and benchmarks (Table 7, Appendix A, 95% bootstrap CIs in brackets). There, the triplet reaches the best OOD balanced accuracy, .81 [.79, .83], ahead of the best two-metric subset (.79 [.78, .80]) and the full 8-metric set (.79 [.78, .81]).

Revisions: Appendix A (“Metric ablation”, Table 7)

Q3: "Could the authors report full Tri-Info and baseline results for the remaining CALVIN and ALOHA-sim settings? Consistent OOD averages across Tables 2, 3, and 9, along with confidence intervals in the main comparison, would also help."

Thank you for this important advice. We have added Tri-Info and baseline results for the five remaining CALVIN and ALOHA-sim settings (5 model-environment pairs), each evaluated both in-domain and OOD from PI0.5-LIBERO (Appendix A). The main comparison now includes 95% confidence intervals (Table 4, Appendix A).

Main comparison (Table 5 and Table 4, Appendix A; Figures 5 and 6). The full results are consistent with our main claim. In-domain, Tri-Info is competitive: it has the best average balanced accuracy (0.91), with GRU-STAC close behind (0.89) and slightly ahead on two settings. OOD, Tri-Info is the best on all five settings, with an OOD average of 0.81. STAC, GRU-STAC, Fail-Detect and FIPER fall to chance, while SAFE retains a partial OOD average (0.59) through the VAE latents.

Estimator-input ablation (Table 6, Appendix A). VAE latents of the policy internals give the best average both in-domain (0.91) and OOD (0.81), and the gap to raw inputs is larger OOD. This replicates our conclusion that model internals help most under domain shift.

Multi-metric ablation (Table 7, Appendix A). The three Tri-Info signals keep the best OOD average (0.81), ahead of every 1-metric and 2-metric subset and of the full 8-metric set (0.79). In-domain, the 8-metric set is marginally higher (0.92 versus 0.91, with overlapping confidence intervals), but this gain does not carry over to OOD. This replicates our claim that the three Tri-Info signals are what drive OOD performance.

Revisions: Appendix A (Tables 4–7; Figures 5 and 6)

AuthorsOpenReview ↗

Response to Reviewer3 tJpo (2/4)

Q4.1; “How much of each trajectory is removed by truncation, and do the AUC results hold without it?”

We cut every trajectory to the length of the shortest successful one. Failed rollouts always run to the step limit, so truncation keeps the first 18–40% of each failed trajectory (no truncation on ALOHA-real-drawer):

BenchmarkPolicyTruncation pointStep limit
LIBERO-10PI0 / PI0.5155520
CALVINFLOWER / UniVLA / GR-165 / 76 / 70360
ALOHA-transfer / insertionACT232 / 284800
ALOHA-real-cupACT198500
ALOHA-real-drawerPI0.5850850

The results hold, and AUC is higher without truncation in table below. However, the extra gain can comes from trajectory length rather than detection: failed trajectories are much longer, so their max-pooled score covers many more timesteps and is easily higher than that of a successful trajectory, resulting in a higher AUC. We therefore report the truncated numbers as the fairer comparison.

TruncatedFull length
PI0-LIBERO-100.701.00
PI0.5-LIBERO-100.860.97
PI0-LIBERO → PI0.5-LIBERO0.821.00
PI0.5-LIBERO → ACT-ALOHA-real-cup0.890.93

Q4.2: “For Figure 6, could the metric traces also be aligned to independently annotated failure onset times rather than the detector's trigger?”

Thank you for the constructive suggestion. In the revised paper, we annotated each failed rollout with its failure onset time and added it to Figure 7 (Section 5.1; Figure 6 in the original submission). The metric traces stay aligned at the trigger t∗t^{*}, and the annotated onset is shown as a dashed line (mean) with a shaded band (±std). The table gives the offset between the trigger and the annotated onset, in units of episode progress.

Failure modeTrigger − onset (mean ± std)
Freeze−0.046 ± 0.066
Drift+0.050 ± 0.158
Phantom grasp−0.014 ± 0.059

Overall, the trigger aligns with the independently annotated onset, and the metric changes around it match each failure mode's semantics.

Revisions: Section 5.1 (“Does the trigger agree with the failure onset?”, Figure 7)

AuthorsOpenReview ↗

Response to Reviewer3 tJpo (3/4)

Q5.1: ”At \alpha = 0.05, about 26% of successful trajectories are falsely flagged. What operating point would the authors recommend for real-world deployment, and what alarm precision would it provide?“

We thank the reviewer for the practical question. Our guarantee is pointwise: θ(t)\theta(t) is the ⌈(n+1)(1−α)⌉/n\lceil (n+1)(1-\alpha)\rceil/n quantile over Dcal+\mathcal D^+_{\mathrm{cal}} at each tt, so Pr⁡(Pˉtξ>θ(t))≤α\Pr(\bar P^{\xi}_t>\theta(t))\le\alpha per tt. Since an alarm fires at the first crossing, the trajectory-level false-alarm rate exceeds α\alpha. We added an experiment to verify the pointwise guarantee: across 100 random calibration/test splits, the mean pointwise false-alarm rate stays within α\alpha at every level (Table 21, Appendix E.5).

α\alphaPointwise false-alarm rate (mean ± sd over 100 splits)
0.010.0094 ± 0.0026
0.050.0464 ± 0.0064
0.100.0971 ± 0.0093

Increasing α\alpha lowers θ(t)\theta(t): more successful trajectories are falsely flagged, but fewer failures are missed. The operating point therefore depends on how much the deployment tolerates each error, we have added this into Appendix E.5 (“Threshold calibration”, Table 21):

  • Safety-critical tasks with irreversible failures (e.g., fragile objects, human proximity): a miss costs far more than an unnecessary stop, so we recommend α=0.05\alpha=0.05. In our experiments, this gives TPR 0.995, a first alarm at 9.6% of execution, and alarm precision 0.54.
  • Throughput-oriented tasks with recoverable failures (e.g., pick-and-place): frequent interruptions are the main cost, so we recommend α=0.01\alpha=0.01. In our experiments, this gives a false-alarm rate of 9.4%, TPR 0.958, and alarm precision 0.76.

Revisions: Appendix E.5 (“Threshold calibration”, Table 21)

Q5.2: “Does the reported inference latency include VAE encoding and k-NN estimation?”

Yes. The reported latency covers the full detector pipeline: VAE encoding, k-NN estimation, and GRU prediction. It is at most 5.30 ms per step (Table 20), a small fraction of a VLA inference step, so the detector can run alongside VLA inference with negligible overhead.

Reviewer_e2WKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Tri-Info, a failure detector for VLA models. The goal is failure detection that generalizes across policies and explains why a rollout fails. Starting from the information-theoretic premise that effective information transfer is a prerequisite for achieving a goal, authors formalize VLA control as a closed-loop information pipeline in which states and actions alternate, and hypothesize that success and failure rollouts differ in how information-theoretic quantities evolve over time. To this end, they derive several entropy and mutual information metrics over states and actions, each estimated over a sliding window of the rollout.

Critical strengths
  • Practical problem setting. The goal of detecting failures with architecture-agnostic, low-dimensional scalar signals is clear. The attempt to distinguish failure modes by the pattern of metric shifts could be useful for diagnosis at robot deployment.
  • Broad experimental scope. The authors evaluate Tri-Info on various VLA models, simulated benchmarks, and real-world robot tasks, and the detector works even under the most severe transfer. Specifically, since Tri-Info feeds only scalar signals such as entropy and mutual information into the detector, the same detector can be reused as is even when the policy changes.
  • Clear writing. The paper is written intuitively and is easy to understand.
Critical weaknesses
  • The comparison is narrow relative to the claimed generality. The comparison between the Tri-Info detector and the baselines covers only three in-domain settings and three transfers. The in-domain settings are PI0/PI0.5-LIBERO and PI0.5-ALOHA-real-drawer, and all transfers start from LIBERO.
  • The interpretation that VLA models "lose information" is not supported by the estimates. In the default setting, the k-NN estimators operate on 29 consecutive, hence autocorrelated, steps per window, across up to 21 joint dimensions. As the authors themselves acknowledge, the absolute values of the estimates are not meaningful. Because the samples form a time series from a single rollout, the estimates are closer to a nonlinear geometric statistic of a trajectory segment than to the mutual information of the joint distribution at timestep tt.
  • In Figure 4, the four state–action coupling metrics move together even though they are defined differently. This raises the concern that a common factor such as variance or speed, rather than the dependency specific to each metric, dominates the estimates. The three Tri-Info signals also show non-negligible correlations with each other, with r=0.72r = 0.72 between H(At)H(A_t) and I(At;At+1)I(A_t; A_{t+1}), r=−0.81r = -0.81 between H(At)H(A_t) and I(St,St+1;At)I(S_t, S_{t+1}; A_t), and r=−0.59r = -0.59 between I(At;At+1)I(A_t; A_{t+1}) and I(St,St+1;At)I(S_t, S_{t+1}; A_t). To support the claim that the signals are "complementary," evidence is needed that each signal provides different information in the failing phase.
  • The interpretation of phantom grasp is not sufficiently supported. Since sts_t includes proprioception, the arm's motion should still be reflected in st+1s_{t+1} during a phantom grasp. Why does I(St,St+1;At)I(S_t, S_{t+1}; A_t) drop?
  • It is unclear whether mBA is computed under online deployment conditions. The threshold θ(t)\theta(t) is built by interpolating the score sequences of successful trajectories to a common normalized time grid t∈[0,1]t \in [0,1]. Since normalized time requires knowing the episode length TT, it cannot be computed online. The BA at each trajectory progress may have the same problem. If progress is measured against each trajectory's own length, this becomes a problem in environments such as LIBERO, where failed rollouts usually run until the time limit. Successful and failed trajectories at the same progress then correspond to different absolute timesteps, and length information can leak into the evaluation.
Questions for authors

See above.

7 comments
AuthorsOpenReview ↗

Response to Reviewer2 e2WK (1/3)

We sincerely thank the reviewer e2WK for the very thoughtful comments. We are glad that the reviewer recognizes the practical value of our architecture-agnostic failure detection, the breadth of our experiments, and the clarity of our writing. Below we answer each of the questions and hope they can resolve the concerns.

Q1: "The comparison between the Tri-Info detector and the baselines covers only three in-domain settings and three transfers … all transfers start from LIBERO."

Thank you for this valuable comment. We have added five model–environment pairs, each evaluated both in-domain and as a transfer target from PI0.5-LIBERO (Table 5 and Table 4, Appendix A; last five panels of Figures 5 and 6 in the revised PDF). The new results are consistent with our main claim. In-domain, Tri-Info is competitive: it has the best average (0.91), with GRU-STAC close behind (0.89) and slightly ahead on two settings. Under transfer, Tri-Info is the best on all five settings (0.81 on average). STAC, GRU-STAC, Fail-Detect and FIPER fall to chance, while SAFE retains partial transfer (0.59) through the VAE latents.

SettingTri-Info (ours)STACGRU-STACSAFE-VAEFail-Detect-VAEFIPER-VAE
In-domain
Flower-CALVIN0.930.800.870.820.810.74
UniVLA-CALVIN0.900.710.930.730.620.59
GR-1-CALVIN0.900.730.830.730.600.55
ACT-ALOHA-sim-transfer0.950.830.920.800.820.79
ACT-ALOHA-sim-insertion0.860.610.870.680.670.64
In-domain average0.910.730.890.750.700.66
Transfer from PI0.5-LIBERO
→ Flower-CALVIN0.910.500.500.720.520.50
→ UniVLA-CALVIN0.730.500.500.580.510.50
→ GR-1-CALVIN0.700.500.510.550.500.50
→ ACT-ALOHA-sim-transfer0.850.500.500.570.500.50
→ ACT-ALOHA-sim-insertion0.870.500.500.530.500.50
Transfer average0.810.500.500.590.510.50

We thank the reviewer for raising this important point on why all transfers start from LIBERO. Our original reason was that LIBERO covers many diverse tasks, whereas CALVIN and ALOHA contain only a few. To verify this, we added a new ablation (Figure 9, Section 5.3) that trains the detector on 2 to 10 LIBERO tasks. Transfer accuracy rises steadily with the number of source tasks under all three shifts, which confirms that task diversity drives transferability.

Revisions: Appendix A (Tables 4–5; Figures 5 and 6); Section 5.3 (“Does generalization depend on the number of source tasks?”, Figure 9)

AuthorsOpenReview ↗

Response to Reviewer2 e2WK (2/3)

Q2: "The k-NN estimators operate on 29 consecutive, hence autocorrelated, steps per window, across up to 21 joint dimensions… absolute values of the estimates are not meaningful…Because the samples form a time series from a single rollout, the estimates are closer to a nonlinear geometric statistic of a trajectory segment than to the mutual information of the joint distribution at timestep t."

Thank you for the thoughtful comment.

We acknowledge that the absolute values of the estimates are not meaningful on their own; however, the ranking of the information-theoretic estimates is meaningful and can be used by the GRU to distinguish failed from successful trajectories.

To show that the k-NN estimators still rank information metrics correctly with 29 samples, we have added a new experiment that evaluates the estimators on synthetic data with known ground truth, matching our setting: mutual information with joint dimensions 14 and 21, and entropy with dimension 7. Across four distribution families and lag-1 autocorrelations of 0, 0.5, and 0.9, the lower bound of the 95% confidence interval of the Spearman rank correlation between the mean estimate and the true value is at least 0.83 (Table 10, Appendix D). Repeating the experiment with high-dimensional observations compressed to 7-d latents by VAEs, as in our pipeline, gives a lower bound of at least 0.77 (Table 11 and Figure 11, Appendix D). Under our conditions, the estimates therefore preserve the ranking of the underlying information-theoretic quantities, which justifies our use of information-theoretic metrics for failure detection.

In addition, computing an information quantity over a sliding window of time series data, and using its changes to detect transitions, is established practice, for example with permutation entropy on a single time series [1, 2] and sliding-window mutual information on paired time series [3]. These works use different estimators, scalar variables and longer windows than ours, which is why we validated our estimators directly under our own conditions above. We have added these citations to Appendix D.

[1] Bandt, Christoph, and Bernd Pompe. "Permutation entropy: a natural complexity measure for time series." Physical review letters 88.17 (2002): 174102.

[2] Cao, Yinhe, et al. "Detecting dynamical changes in time series using the permutation entropy." Physical Review E—Statistical, Nonlinear, and Soft Matter Physics 70.4 (2004): 046217.

[3] Gu, Chaoran, Verena Heidrich-Meisner, and Robert F. Wimmer-Schweingruber. "Sliding-window cross-correlation and mutual information methods in the analysis of solar wind measurements." Astronomy & Astrophysics 684 (2024): A125.

Revisions: Appendix D (“Assumptions and what the estimates support”, Tables 10–11, Figure 11; new citations)

AuthorsOpenReview ↗

Response to Reviewer2 e2WK (3/3)

Q3: “The four state–action coupling metrics move together even though they are defined differently. This raises the concern that a common factor such as variance or speed, rather than the dependency specific to each metric, dominates the estimates. The three Tri-Info signals also show non-negligible correlations with each other... ”

This is an interesting question about whether a common factor drives these metrics, given that they are correlated. We address it in three parts.

First, the signals are complementary. The most direct evidence is the metric ablation in Table 2, where adding metrics increases accuracy. Figure 1 and Figure 7 further show that the metrics behave differently in the corresponding failing phases.

Second, the correlations follow from shared terms. By the chain rule of mutual information, I(St,St+1;At)=I(St;At)+I(St+1;At∣St)I(S_t, S_{t+1}; A_t) = I(S_t; A_t) + I(S_{t+1}; A_t \mid S_t) and I(At,St;St+1)=I(St+1;At)+I(St;St+1∣At)I(A_t, S_t; S_{t+1}) = I(S_{t+1}; A_t) + I(S_t; S_{t+1} \mid A_t). Each state–action coupling metric therefore contains its predecessor as a term and differs from it only by an additional non-negative conditional term; moreover, adjacent states are highly similar. Similarly, I(St,St+1;At)=H(At)−H(At∣St,St+1)I(S_t, S_{t+1}; A_t) = H(A_t) - H(A_t \mid S_t, S_{t+1}) and I(At;At+1)=H(At+1)−H(At+1∣At)I(A_t; A_{t+1}) = H(A_{t+1}) - H(A_{t+1} \mid A_t), so the three Tri-Info signals share the action-entropy term.

Third, we tested other factors directly in the submission. In Appendix Figure 17 (and Table 16), we evaluate other factors as detector inputs, including variance and temporal gradient (the rate of change of the action, which is similar to speed). They do have some predictive power, but they do not perform as well as Tri-Info. More factors summarized from trajectories may also predict trajectory success, and this is an interesting direction for future work.

Q4: “The interpretation of phantom grasp is not sufficiently supported. Since sts_t includes proprioception, the arm's motion should still be reflected in st+1s_{t+1} during a phantom grasp. Why does I(St,St+1;At)I(S_t, S_{t+1}; A_t) drop?”

Yes, the arm’s motion is still reflected in the state transition. In the successful phase, the action drives both the gripper and the object, so the state transition couples with AtA_t through both. In a phantom grasp, only the gripper still follows the action, while the object does not. The coupling shrinks from gripper + object to gripper alone, so I(St,St+1;At)I(S_t, S_{t+1}; A_t) drops.

Q5: “It is unclear whether mBA is computed under online deployment conditions.”

Thank you for this Ω question. In our online detection experiments, TT is the same for all trajectories in a domain, successful or failed. It is consistent with the step limit of the labeling protocol: a rollout that has not completed the task within TT steps is labelled a failure, so TT is known before execution. Each step is normalized by this fixed TT, so progress k/Tk/T depends only on the current step and can be computed online. Successful and failed trajectories at the same progress are therefore at the same absolute timestep and are treated identically. We have revised the main text to make this more clearer (Section 3.3 and Section 4, “Evaluation”). You are right that successful rollouts finish before TT, so the later part of the progress axis contains few successful trajectories, which contributes to the rise of balanced accuracy in the second half of Figure 5.

Revisions: Section 3.3; Section 4 (“Evaluation”)

Reviewer_FnDUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies VLA failure detection from the perspective of closed-loop information transmission, formalizing the state-action control process as an information pipeline, and selecting Tri-Info from eight candidate indicators: action entropy, temporal action consistency, and action-state coupling. These three types of indicators respectively depict action collapse or divergence, continuous action loss of coordination, and policy and observation decoupling. Their temporal evolution is modeled using GRU. The experiments cover various VLA, LIBERO, CALVIN, ALOHA simulations and real robot tasks, with a focus on evaluating cross-model, cross-environment and sim-to-real transfer. It achieves strong OOD detection performance without using target domain failure labels.

Critical strengths
  1. The closed-loop information flow perspective has strong conceptual uniformity, categorizing multiple failure modes into the disruption of action information volume, temporal correlation, and perception-action coupling.
  2. Tri-Info retains only three complementary indicators, which is more concise and interpretable compared to directly concatenating all eight indicators, and significantly outperforms in the average performance of OOD.
  3. Cross-strategy, cross-environment, cross-embodiment, and sim-to-real evaluations are comprehensive, particularly focusing on transfer scenarios without target domain failure labels, which aligns with actual deployment requirements.
  4. Experiments show that the detector based on absolute latent geometry significantly degrades after model switching, while the information-theory indicators maintain better performance, supporting the necessity of the proposed invariant design.
Critical weaknesses
  1. Entropy and mutual information are estimated through VAE compression and k-NN window estimation. The results may depend on the quality of representation, window length, and sample density; the estimation bias and confidence interval lack theoretical or empirical characterization.
  2. The complete version still requires accessing the internal action/state embeddings of the policy and re-fitting VAE for new models or embodiments; "cross-architecture transfer" is not completely without target domain adaptation.
  3. The failure mode explanations given by Tri-Info are mainly correlation diagnoses; they cannot prove that the change in information quantity is the cause of failure, nor have they verified whether targeted recovery based on different signals is effective.
  4. The current contribution focuses on alarms rather than closed-loop safety improvements; the detection accuracy has not been converted into success rates and risk benefits after stopping, resampling, recovery, or manual takeover.
  5. Sliding windows and temporal GRU may cause detection delays, especially for rapid contact errors or irreversible failures, but the paper lacks reports on time-to-detection, the lead time of the first alarm, and the cost of delay.
Questions for authors

Please refer to weakness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer1 FnDU (1/3)

We sincerely thank the reviewer FnDU for the thorough and constructive comments. We are glad that the reviewer finds our closed-loop information-flow perspective conceptually unifying, Tri-Info concise and interpretable, our transfer evaluations comprehensive, and the proposed invariant design necessary. Here we respond to all the questions and hope our answers can address the concerns.

Q1: “Entropy and mutual information are estimated through VAE compression and k-NN window estimation. The results may depend on the quality of representation, window length, and sample density; the estimation bias and confidence interval lack theoretical or empirical characterization.”

Thank you for this valuable comment. The submission already reported sensitivity to window length and k, and we have now added a synthetic study with known ground truth to characterize the estimation error.

  • Window length and sample density (Table 12): AUC stays within 0.875–0.915 for W from 10 to 40, and within 0.893–0.918 for k ∈ {5, 6, 8, 9, 10}.
  • Bias and confidence intervals (new, Table 10, Appendix D): on synthetic data matching our setting (29 samples, dimensions 7, 14 and 21, four distribution families, lag-1 autocorrelation up to 0.9), the Spearman rank correlation between estimated and true values has a 95% confidence interval that stays above 0.83.
  • Quality of representation (new, Table 11 and Figure 11, Appendix D): with observations compressed to 7-d latents by VAEs, as in our pipeline, the rank correlation is at least 0.94 once the VAEs are trained for enough epochs. Moreover, replacing the VAE latents with raw actions and proprioception keeps in-domain balanced accuracy within 0.876–0.897 (Table 3).

We agree that absolute values are biased at this sample size, as stated in the submission. Our detector relies only on the ranking of the estimates, which these results show is preserved.

Revisions: Appendix D (“Assumptions and what the estimates support”, Tables 10–11, Figure 11)

Q2: “The complete version still requires accessing the internal action/state embeddings of the policy and re-fitting VAE for new models or embodiments; "cross-architecture transfer" is not completely without target domain adaptation.”

Yes, we agree that the full Tri-Info pipeline re-fits the VAE for each new model, which is why our claim in the submission is that Tri-Info transfers “without labeled failure data from the target domain”: the VAE is label-free, and the GRUs, normalization, and threshold are fitted on the source domain only (Table 14). Moreover, Table 3 shows that Tri-Info also works with only the executed action and raw proprioception, which requires neither internal embeddings nor VAE re-fitting, and still reaches 0.794 mean balanced accuracy out of distribution (0.865 with embeddings).

AuthorsOpenReview ↗

Response to Reviewer1 FnDU (2/3)

Q3 Q4: “The failure mode explanations given by Tri-Info are mainly correlation diagnoses; they cannot prove that the change in information quantity is the cause of failure, nor have they verified whether targeted recovery based on different signals is effective… the detection accuracy has not been converted into success rates and risk benefits after stopping, resampling, recovery, or manual takeover“

We agree with the reviewer that our analysis is correlational: we do not intend to claim that the change in information causes failure. Tri-Info shows that failing trajectories deviate from successful ones in their information metrics and uses this deviation as a detection signal. We acknowledge in the Limitations (Appendix B) that the scope of this paper is failure detection, rather than recovery or establishing the cause of failure.

We also have an experiment explore this in the submission. We use Tri-Info as an inference-time action selector (Appendix F, Table 25): at each replan point, we sample 32 candidate action chunks from π0.5 and execute the one with the lowest predicted failure probability. This raises the success rate from 0.670 to 0.710 on robosuite PickPlaceCan, which we present as preliminary evidence that the signal can guide intervention.

In the future, there are several possible ways to turn the signal into higher success rates. Recovery-based approaches act after an alarm, such as stopping, resampling the action, rolling back to an earlier state, or handing over to a human. Direct approaches improve the policy itself: selecting or rejecting candidate actions at inference time, as in our preliminary experiment, or using the metrics as a training objective or reward. A systematic study of these is a valuable direction that we leave to future work.

Revisions: Appendix F (“Does selecting actions with better information raise the success rate?”, Table 25)

AuthorsOpenReview ↗

Response to Reviewer1 FnDU (3/3)

Q5: “Sliding windows and temporal GRU may cause detection delays, especially for rapid contact errors or irreversible failures, but the paper lacks reports on time-to-detection, the lead time of the first alarm, and the cost of delay.”

Thank you for the constructive advice. We address the three points below.

1. Time-to-detection (latency). Appendix E.5 (Table 20) reports the cost of the full detector pipeline: VAE encoding, k-NN estimation and GRU prediction. It is at most 5.30 ms per step, a small fraction of one VLA inference step. The detector can therefore run alongside the VLA with negligible overhead.

2. Lead time of the first alarm. We added all baselines to a first-alarm comparison (Appendix E.5, Table 22). It reports when each detector first alarms, as a percentage of trajectory length, averaged over the failing trajectories it detects (lower is earlier).

  • In-domain: Tri-Info alarms at 15.9% on average. This is earlier than STAC (55.9%) and SAFE (34.9%), and on par with GRU-STAC, Fail-Detect and FIPER (11.1–17.5%).
  • Out-of-domain: Tri-Info is the only method with a valid alarm time on all eight transfers (27.0% on average). Every baseline is invalid on at least half of them, because it either never alarms or also alarms on ≥90% of successful trajectories.

3. Agreement between the trigger and the real failure onset. We annotated the failure onset time of each failed rollout and added it to Figure 7 (Section 5.1), shown as a dashed line (mean) with a shaded band (±std). The table below gives the offset between the trigger t∗t^{*} and the annotated onset, in units of episode progress. A negative value means the trigger fires before the onset.

Failure modeTrigger − onset (mean ± std)
Freeze−0.046 ± 0.066
Drift+0.050 ± 0.158
Phantom grasp−0.014 ± 0.059

On average, the trigger is within 5% of the episode of the annotated onset in all three failure modes.

In summary, Tri-Info adds negligible overhead to VLA inference, raises competitive early warnings, and triggers close to the annotated failure onset

Revisions: Section 5.1 (“Does the trigger agree with the failure onset?”, Figure 7); Appendix E.5 (“First-alarm time”, Table 22)

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the rebuttal and the major changes in the revised manuscript

Dear Reviewers,

We sincerely thank you for the time and care you devoted to reviewing our manuscript. Your constructive comments have helped us improve both the rigor and the clarity of this work.

We appreciate that the reviewers recognize the strengths of our work: unifying perspective, interpretability, comprehensive evaluation, and invariant design (Reviewer FnDU); practicality, broad experiments, and clear writing (Reviewer e2WK); and generalization, deployment focus, thorough ablations, lightweight design, and robustness (Reviewer tJpo).

The reviewers also raised valuable concerns and questions. We have addressed each of them in the individual responses, with additional experiments and analyses in support. The revised manuscript has been uploaded, with all additions and changes marked in blue color. Owing to the page limit, the key results are reported in the main text and the complete results in the appendix. All table, figure and section numbers in our responses refer to the revised PDF. The major changes are summarized below.

  1. More model–environment pairs: We add five pairs on CALVIN and ALOHA-sim, each evaluated in-domain and as a transfer target from PI0.5-LIBERO, with baselines, ablations and 95% confidence intervals. Tri-Info is the best on all five transfers (0.81 vs. at most 0.59) (Figures 5 and 6, Appendix A, Tables 4–7).
  2. Validity of the information estimates: On synthetic data with known ground truth matching our setting, the estimators preserve the ranking of the true quantities, also after VAE compression (Appendix D, Tables 10–11, Figure 11).
  3. Failure onset and first alarm: The trigger falls within 5% of the episode of the annotated failure onset (Section 5.1, Figure 7), and Tri-Info is the only method with a valid first-alarm time on all eight transfers (Appendix E.5, Table 22).
  4. Source-task diversity: Transfer accuracy rises with the number of LIBERO source tasks (Section 5.3, Figure 9).
  5. Target VAE on held-out data: Twenty unlabeled target rollouts suffice to fit the target VAE for two of the three shifts (Appendix D, Figure 13).
  6. Threshold calibration: We verify the pointwise false-alarm guarantee and recommend operating points for deployment (Appendix E.5, Table 21).
  7. Clarifications: We clarify the online evaluation protocol (Section 3.3, Section 4), and frame action selection as a preliminary study with future directions (Appendix F, Appendix B).

Thank you again for your valuable feedback. We look forward to further discussion.

Best regards,

Authors of Submission 9438

AuthorsOpenReview ↗

Thank You and Response to the New Question

Thank you for your follow-up and for confirming that our rebuttal has addressed your concerns. We address the new question below.

Q: "My remaining reservation is whether the success/failure separation holds across layers/estimators."

The separation holds across layers. On a newly generated small PI0.5-LIBERO set (64 training / 42 test trajectories; 3 seeds), we take the action and/or visual embedding from layer ℓ\ell of PI0.5 (18 layers). The choice of layer does not materially affect performance (mBA) and shows no trend with layer choice (Figure 19, Appendix F).

Layer ℓ\ell1369121518Default
Visual only0.821 ± 0.0380.789 ± 0.0190.829 ± 0.0210.821 ± 0.0250.799 ± 0.0050.830 ± 0.0180.827 ± 0.0250.804 ± 0.019
Action only0.814 ± 0.0030.825 ± 0.0070.784 ± 0.0020.827 ± 0.0000.754 ± 0.0230.824 ± 0.0290.814 ± 0.0280.804 ± 0.019

The separation holds across estimators. On PI0.5-LIBERO, we replace only the default KSG/KL estimator and keep everything else fixed. We test four alternative estimators: kNN 3H, Gaussian log-det, Gaussian copula and KDE. All of them reach similar performance(Figure 14, Appendix D).

EstimatorKSG/KL (ours)kNN 3HGaussianGaussian copulaKDE
mBA0.888 ± 0.0010.884 ± 0.0030.873 ± 0.0040.856 ± 0.0080.880 ± 0.004

Revisions: Appendix D ("Does the detector performance hold under other estimators?", Figure 14); Appendix F ("Does changing the embedding layer change the performance?", Figure 19)

We thank the reviewer for raising these two points. These additional results further show that our method is robust to the choice of embedding layer and estimator. Given your confirmation that our rebuttal has addressed your concerns, and if these additional results also resolve your remaining reservation, we would be grateful if you could consider whether the strengthened evidence warrants a higher score. Your consideration would mean a lot to us. Thank you again for your time and thoughtful feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.