- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Tri-Info, an information-theoretic method for detecting failures in vision-language-action (VLA) models. It focuses on three signals in the perception–action loop: action entropy, temporal action coherence, and action–state coupling. The authors estimate these signals across sliding windows and use them in GRU-based detectors with calibrated thresholds. They evaluate six VLA policies on LIBERO, CALVIN, and both simulated and real-world ALOHA tasks. Tri-Info performs well in-domain, with an average balanced accuracy of 0.83, and generalizes more effectively across policies and environments, reaching an average out-of-distribution balanced accuracy of 0.81. The paper also presents extensive ablations and analyses of different failure modes.
Critical strengths
- The main contribution is showing that simple information-theoretic signals outperform existing failure detectors under distribution shift. Tri-Info performs consistently across models and environments, including sim-to-real transfer, while several embedding-based baselines degrade substantially. Its focus on thresholded accuracy, not just AUC, is especially useful for deployment.
- The authors compare alternative action statistics and input representations, and test sensitivity to model capacity and threshold calibration. Appendices report confidence intervals and discuss limitations, helping clarify which parts of the method drive its performance.
- The detector is lightweight, and its three signals offer intuitive explanations of failures such as freezing, drifting, and phantom grasps. Results on noisy demonstrations also support its robustness.
Critical weaknesses
- The information-theoretic interpretation is overstated. Metrics estimated from just 29 temporally correlated samples using k-NN estimators on seven-dimensional VAE latents have no meaningful absolute values (Appendix C). The results show these statistics can detect failure, but not that information loss causes it. The main method also relies on internal embeddings rather than executed actions. The promising black-box variant performs worse under distribution shift, with 0.794 versus 0.865 OOD balanced accuracy.
- The transfer evaluation uses target-domain data. Appendix D.2 states that the VAE is trained on embeddings from all domains, including test trajectories. Although it uses no target-domain failure labels, the evaluation is transductive rather than fully zero-shot. Since cross-domain generalization is the paper's main claim, the authors should test whether the results hold when the VAE is fitted on a separate, unlabeled target-domain dataset.
- The empirical evidence is incomplete. In-domain performance is competitive but not consistently better: Tri-Info’s average AUC is lower than SAFE’s (0.82 vs. 0.91), and combining all three signals does not always outperform the best individual metric. The full baseline comparison covers only three model–environment combinations, despite Table 1’s broader evaluation. Appendix B’s subset-search results are also nearly identical across several feature combinations, leaving the case for exactly three metrics unclear.
- Several evaluation details limit the conclusions. For AUC, trajectories are truncated to the shortest successful rollout, but the paper does not report how much failure information this removes. Aligning failures at the detector's trigger may partly explain changes in the interpretability metric. Threshold calibration reuses training trajectories, so it provides no formal conformal coverage guarantee. At the default \alpha=0.05, the true-negative rate is 0.736, suggesting substantial false alarms. The action-selection improvement in Appendix E is not statistically significant (p = 0.148).
Questions for authors
- Could the authors repeat the three transfer experiments with a VAE trained on a separate, unlabeled target-domain dataset that excludes all test trajectories? This would help isolate the contribution of transductive training to the reported generalization.
- Which data splits were used for the correlation analysis in Figure 4 and the exhaustive subset search in Appendix B? Were test trajectories included? Since the subsets have similar AUCs, how stable is the selected triplet?
- Could the authors report full Tri-Info and baseline results for the remaining CALVIN and ALOHA-sim settings? Consistent OOD averages across Tables 2, 3, and 9, along with confidence intervals in the main comparison, would also help.
- How much of each trajectory is removed by truncation, and do the AUC results hold without it? For Figure 6, could the metric traces also be aligned to independently annotated failure onset times rather than the detector's trigger?
- At \alpha = 0.05, about 26% of successful trajectories are falsely flagged. What operating point would the authors recommend for real-world deployment, and what alarm precision would it provide? Does the reported inference latency include VAE encoding and k-NN estimation?
5 comments
Response to Reviewer3 tJpo (2/4)
Q4.1; “How much of each trajectory is removed by truncation, and do the AUC results hold without it?”
We cut every trajectory to the length of the shortest successful one. Failed rollouts always run to the step limit, so truncation keeps the first 18–40% of each failed trajectory (no truncation on ALOHA-real-drawer):
| Benchmark | Policy | Truncation point | Step limit |
|---|---|---|---|
| LIBERO-10 | PI0 / PI0.5 | 155 | 520 |
| CALVIN | FLOWER / UniVLA / GR-1 | 65 / 76 / 70 | 360 |
| ALOHA-transfer / insertion | ACT | 232 / 284 | 800 |
| ALOHA-real-cup | ACT | 198 | 500 |
| ALOHA-real-drawer | PI0.5 | 850 | 850 |
The results hold, and AUC is higher without truncation in table below. However, the extra gain can comes from trajectory length rather than detection: failed trajectories are much longer, so their max-pooled score covers many more timesteps and is easily higher than that of a successful trajectory, resulting in a higher AUC. We therefore report the truncated numbers as the fairer comparison.
| Truncated | Full length | |
|---|---|---|
| PI0-LIBERO-10 | 0.70 | 1.00 |
| PI0.5-LIBERO-10 | 0.86 | 0.97 |
| PI0-LIBERO → PI0.5-LIBERO | 0.82 | 1.00 |
| PI0.5-LIBERO → ACT-ALOHA-real-cup | 0.89 | 0.93 |
Q4.2: “For Figure 6, could the metric traces also be aligned to independently annotated failure onset times rather than the detector's trigger?”
Thank you for the constructive suggestion. In the revised paper, we annotated each failed rollout with its failure onset time and added it to Figure 7 (Section 5.1; Figure 6 in the original submission). The metric traces stay aligned at the trigger , and the annotated onset is shown as a dashed line (mean) with a shaded band (±std). The table gives the offset between the trigger and the annotated onset, in units of episode progress.
| Failure mode | Trigger − onset (mean ± std) |
|---|---|
| Freeze | −0.046 ± 0.066 |
| Drift | +0.050 ± 0.158 |
| Phantom grasp | −0.014 ± 0.059 |
Overall, the trigger aligns with the independently annotated onset, and the metric changes around it match each failure mode's semantics.
Revisions: Section 5.1 (“Does the trigger agree with the failure onset?”, Figure 7)
Response to Reviewer3 tJpo (3/4)
Q5.1: ”At \alpha = 0.05, about 26% of successful trajectories are falsely flagged. What operating point would the authors recommend for real-world deployment, and what alarm precision would it provide?“
We thank the reviewer for the practical question. Our guarantee is pointwise: is the quantile over at each , so per . Since an alarm fires at the first crossing, the trajectory-level false-alarm rate exceeds . We added an experiment to verify the pointwise guarantee: across 100 random calibration/test splits, the mean pointwise false-alarm rate stays within at every level (Table 21, Appendix E.5).
| Pointwise false-alarm rate (mean ± sd over 100 splits) | |
|---|---|
| 0.01 | 0.0094 ± 0.0026 |
| 0.05 | 0.0464 ± 0.0064 |
| 0.10 | 0.0971 ± 0.0093 |
Increasing lowers : more successful trajectories are falsely flagged, but fewer failures are missed. The operating point therefore depends on how much the deployment tolerates each error, we have added this into Appendix E.5 (“Threshold calibration”, Table 21):
- Safety-critical tasks with irreversible failures (e.g., fragile objects, human proximity): a miss costs far more than an unnecessary stop, so we recommend . In our experiments, this gives TPR 0.995, a first alarm at 9.6% of execution, and alarm precision 0.54.
- Throughput-oriented tasks with recoverable failures (e.g., pick-and-place): frequent interruptions are the main cost, so we recommend . In our experiments, this gives a false-alarm rate of 9.4%, TPR 0.958, and alarm precision 0.76.
Revisions: Appendix E.5 (“Threshold calibration”, Table 21)
Q5.2: “Does the reported inference latency include VAE encoding and k-NN estimation?”
Yes. The reported latency covers the full detector pipeline: VAE encoding, k-NN estimation, and GRU prediction. It is at most 5.30 ms per step (Table 20), a small fraction of a VLA inference step, so the detector can run alongside VLA inference with negligible overhead.
Response to Reviewer3 tJpo (1/4)
We sincerely thank the reviewer tJpo for the detailed and constructive comments. We are glad that the reviewer recognizes the strong generalization of Tri-Info under distribution shift, our deployment-oriented focus on thresholded accuracy, the thoroughness of our ablations and analyses, the lightweight and interpretable design of our detector, and its robustness to noisy demonstrations. Here we address all the questions raised and hope our responses can clear up the concerns.
Q1: “Since cross-domain generalization is the paper's main claim, the authors should test whether the results hold when the VAE is fitted on a separate, unlabeled target-domain dataset.”
Thank you for the constructive suggestion.
We ran the requested experiment. We hold out the test trajectories and fit the target VAE on n∈[2,5,10,20,30,40] of the remaining unlabeled target trajectories. The frozen source detector is then evaluated on the held-out set.
With 20 trajectories, mean balanced accuracy is 0.90 on PI0.5-LIBERO and 0.82 on ACT-ALOHA-real-cup, matching the VAE fitted on all target data (0.89 and 0.82). On PI0.5-ALOHA-real-block, it reaches 0.64, slightly below 0.69. We added these results to the revised paper (Figure 13, Appendix D).
Our main claim is unchanged: Tri-Info transfers to a new domain without labeled target data. This experiment additionally shows that a small set of unlabeled target rollouts is enough, which makes deployment more practical.
Revisions: Appendix D (“How many target trajectories does the VAE need?”, Figure 13)
Q2: “Which data splits were used for the correlation analysis in Figure 4 and the exhaustive subset search in Appendix B? Were test trajectories included? Since the subsets have similar AUCs, how stable is the selected triplet?”
Thank you for the question. The correlation analysis in Figure 4 and the exhaustive subset search in Appendix C.1 exclude all test sets, as well as the two real-world tasks we collected for OOD transfer (PI0.5-ALOHA-Real-Block and ACT-ALOHA-Real-Cup). The subset selection is therefore not fitted to the test set.
Results in the submission support the stability of the selected triplet. In Appendix E.5, OOD balanced accuracy is high, with tight trajectory-level 95% bootstrap confidence intervals, shown in brackets below (Table 17). On cross-model transfer, accuracy holds at 0.91–0.92 throughout the trajectory, with the lower bound of the 95% CI at least 0.86. On ACT-ALOHA-Real-Cup it peaks at 0.95 [0.91, 0.98], and on PI0.5-ALOHA-Real-Block it rises to 0.83 [0.74, 0.91]. In the detector capacity sweep (Table 18), AUC (mean ± std over three seeds) stays within 0.818–0.839 (std ≤ .008) on cross-model transfer and 0.852–0.910 (std ≤ .048) on sim-to-real transfer.
We also added a new multi-metric ablation on five additional policies and benchmarks (Table 7, Appendix A, 95% bootstrap CIs in brackets). There, the triplet reaches the best OOD balanced accuracy, .81 [.79, .83], ahead of the best two-metric subset (.79 [.78, .80]) and the full 8-metric set (.79 [.78, .81]).
Revisions: Appendix A (“Metric ablation”, Table 7)
Q3: "Could the authors report full Tri-Info and baseline results for the remaining CALVIN and ALOHA-sim settings? Consistent OOD averages across Tables 2, 3, and 9, along with confidence intervals in the main comparison, would also help."
Thank you for this important advice. We have added Tri-Info and baseline results for the five remaining CALVIN and ALOHA-sim settings (5 model-environment pairs), each evaluated both in-domain and OOD from PI0.5-LIBERO (Appendix A). The main comparison now includes 95% confidence intervals (Table 4, Appendix A).
Main comparison (Table 5 and Table 4, Appendix A; Figures 5 and 6). The full results are consistent with our main claim. In-domain, Tri-Info is competitive: it has the best average balanced accuracy (0.91), with GRU-STAC close behind (0.89) and slightly ahead on two settings. OOD, Tri-Info is the best on all five settings, with an OOD average of 0.81. STAC, GRU-STAC, Fail-Detect and FIPER fall to chance, while SAFE retains a partial OOD average (0.59) through the VAE latents.
Estimator-input ablation (Table 6, Appendix A). VAE latents of the policy internals give the best average both in-domain (0.91) and OOD (0.81), and the gap to raw inputs is larger OOD. This replicates our conclusion that model internals help most under domain shift.
Multi-metric ablation (Table 7, Appendix A). The three Tri-Info signals keep the best OOD average (0.81), ahead of every 1-metric and 2-metric subset and of the full 8-metric set (0.79). In-domain, the 8-metric set is marginally higher (0.92 versus 0.91, with overlapping confidence intervals), but this gain does not carry over to OOD. This replicates our claim that the three Tri-Info signals are what drive OOD performance.
Revisions: Appendix A (Tables 4–7; Figures 5 and 6)