- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies runtime failure detection for VLAs. The key idea is to replace the full hidden-state representation used by prior work with a selective readout from FFN neurons. To select the readabou, the authors compute a rollout-level activation statistic for each FFN neuron, rank neurons according to the absolute deviation, and reconstruct their activation-weighted output contribution through projection. This selective representation is then provided to an MLP or LSTM failure detector. The VLA itself remains frozen. The appraoch is empirically validated on LIBERO10 using four VLA policies.
Critical strengths
-
The proposed method is simple and computationally efficient. Neuron selection requires only rollout-level outcome labels from an offline dataset. It is substantially simpler than approaches requiring policy fine-tuning.
-
The empirical study is broad. The evaluation is across both autoregressive and flow-matching VLA architectures. The ablation study is also comprehensive and interesting. Reversing the ranking substantially degrades performance. The neuron-budget experiment is also informative, where using selected neurons outperforms reading all FFN neurons on unseen tasks
Critical weaknesses
-
Now the representation part needs two steps: first neuron selection, and then representation reconstruction. The experiments do not yet establish whether the gain comes more from representation construction or simply from supervised neuron selection. This is directly relevant to the main claim of this paper, so it is an important missing piece.
-
The neuron selector has access to the future consequence of failure, although the claimed application is online failure detection. SAFER selects neurons using the activation averaged over the entire rollout. But at deployment time, the detector is supposed to operate from a prefix to give timely detection/early warning, which is also the paper's emphasis.
Questions for authors
-
What happens if the detector receives the selected FFN activations directly rather than the reconstructed one with FFN? Is the output-direction reconstruction actually important?
-
How different are the results if neuron selectivity is computed within each task and then aggregated across training tasks? This seems important for ruling out task-identity confounding.
-
Are SAFE and SAFER hyperparameters tuned over identical search spaces and identical budgets? If so, please specify the hyperparameter search spaces.
5 comments
Response to Reviewer LEy2 (2/3)
[W2] The neuron selector has access to the future consequence of failure, although the claimed application is online failure detection. SAFER selects neurons using the activation averaged over the entire rollout. But at deployment time, the detector is supposed to operate from a prefix to give timely detection/early warning, which is also the paper's emphasis.
A: Thank you for pointing out this issue. To examine the effect of selecting neurons using activations averaged over full rollouts, we conducted additional experiments with different trajectory prefix lengths. In Table R2, we evaluate SAFER on OpenVLA and π₀ using prefix lengths ranging from 200 steps to the full rollout. The average successful rollout lengths are 284.5 and 260.5 steps for OpenVLA and π₀, respectively.
At Prefix-300, SAFER already achieves meaningful cross-task failure detection performance. However, full-rollout neuron selection still achieves the highest average Unseen AUC. These results suggest that shorter trajectory prefixes already contain useful signals for failure detection. In contrast, full rollouts provide more information for offline neuron selection and further improve cross-task detection performance. These findings support the feasibility of our neuron selection strategy. The updated results are reported in Table 9 of the revised paper.
Table R2. Effect of Trajectory Prefix Length on Downstream Failure Detection Performance.
\hline \scriptstyle\text{Method} &\scriptstyle\text{OpenVLA-Seen} &\scriptstyle\text{OpenVLA-Unseen} &\scriptstyle\pi_0\text{-Seen} &\scriptstyle\pi_0\text{-Unseen} &\scriptstyle\text{Avg.-Seen} &\scriptstyle\text{Avg.-Unseen}\\\\ \hline \scriptstyle\text{SAFE-MLP} &\scriptstyle68.78&\scriptstyle62.93&\scriptstyle70.00&\scriptstyle75.12&\scriptstyle69.39&\scriptstyle69.03\\\\ \scriptstyle\text{SAFER-MLP (Prefix-200)} &\scriptstyle68.16&\scriptstyle69.66&\scriptstyle72.29&\scriptstyle76.83&\scriptstyle70.23&\scriptstyle73.25\\\\ \scriptstyle\text{SAFER-MLP (Prefix-250)} &\scriptstyle72.14&\scriptstyle71.00&\scriptstyle68.51&\scriptstyle72.41&\scriptstyle70.33&\scriptstyle71.71\\\\ \scriptstyle\text{SAFER-MLP (Prefix-300)} &\scriptstyle\mathbf{74.92}&\scriptstyle68.13&\scriptstyle75.08&\scriptstyle79.91&\scriptstyle\mathbf{75.00}&\scriptstyle74.02\\\\ \scriptstyle\text{SAFER-MLP} &\scriptstyle68.32&\scriptstyle\mathbf{71.60}&\scriptstyle\mathbf{77.83}&\scriptstyle\mathbf{81.43}&\scriptstyle73.08&\scriptstyle\mathbf{76.52}\\\\ \hline \scriptstyle\text{SAFE-LSTM} &\scriptstyle\mathbf{63.34}&\scriptstyle65.33&\scriptstyle59.46&\scriptstyle63.91&\scriptstyle61.40&\scriptstyle64.62\\\\ \scriptstyle\text{SAFER-LSTM (Prefix-200)} &\scriptstyle60.63&\scriptstyle58.29&\scriptstyle72.52&\scriptstyle70.77&\scriptstyle66.58&\scriptstyle64.53\\\\ \scriptstyle\text{SAFER-LSTM (Prefix-250)} &\scriptstyle62.60&\scriptstyle49.48&\scriptstyle75.29&\scriptstyle68.04&\scriptstyle68.95&\scriptstyle58.76\\\\ \scriptstyle\text{SAFER-LSTM (Prefix-300)} &\scriptstyle59.72&\scriptstyle57.63&\scriptstyle75.36&\scriptstyle64.73&\scriptstyle67.54&\scriptstyle61.18\\\\ \scriptstyle\text{SAFER-LSTM} &\scriptstyle61.89&\scriptstyle\mathbf{60.94}&\scriptstyle\mathbf{76.64}&\scriptstyle69.84&\scriptstyle\mathbf{69.27}&\scriptstyle\mathbf{65.39}\\\\ \hline \end{array}$$Response to Reviewer LEy2 (3/3)
[Q2] How different are the results if neuron selectivity is computed within each task and then aggregated across training tasks? This seems important for ruling out task-identity confounding.
A: We thank the reviewer for this valuable suggestion. We recomputed neuron rankings using Within-Task AUC on OpenVLA-LIBERO10 and π₀-LIBERO10, and compared the selected neurons with those from the original Pooled-AUC selection. Table R3 shows that the average Top-100 overlap is 50.67% for OpenVLA and 42.33% for π₀. These results indicate that the two selection methods identify partially overlapping sets of neurons. We have added these results to Table 13 of the revised appendix.
Table R3. Top-100 Neuron Overlap Between Pooled-AUC and Within-Task AUC Selection.
\hline \scriptstyle\text{Benchmark}&\scriptstyle\text{Top-100 Overlap (\\%)}\\\\ \hline \scriptstyle\text{OpenVLA-LIBERO10}&\scriptstyle50.67\\\\ \scriptstyle\pi_0\text{-LIBERO10}&\scriptstyle42.33\\\\ \hline \end{array}$$ We further compared SAFER's detection performance under the original Pooled-AUC and Within-Task AUC selection methods. **Table R4 shows that Within-Task AUC selection does not substantially reduce SAFER's detection performance.** The original Pooled-AUC selection performs better with LSTM detectors, while MLP detectors achieve comparable results under both selection methods. These results suggest that SAFER's detection capability does not rely solely on differences in failure rates across tasks. We have added these results to Table 12 of the revised appendix. **Table R4.** Within-Task AUC Neuron Selection Ablation. $$\begin{array}{lcccccc} \hline \scriptstyle\text{Method} &\scriptstyle\text{OpenVLA-Seen} &\scriptstyle\text{OpenVLA-Unseen} &\scriptstyle\pi_0\text{-Seen} &\scriptstyle\pi_0\text{-Unseen} &\scriptstyle\text{Avg.-Seen} &\scriptstyle\text{Avg.-Unseen}\\\\ \hline \scriptstyle\text{SAFER-MLP (Within-Task)} &\scriptstyle70.91&\scriptstyle72.79&\scriptstyle73.46&\scriptstyle81.48&\scriptstyle72.19&\scriptstyle\mathbf{77.14}\\\\ \scriptstyle\text{SAFER-MLP (Original)} &\scriptstyle68.32&\scriptstyle71.60&\scriptstyle77.83&\scriptstyle81.43&\scriptstyle\mathbf{73.08}&\scriptstyle76.52\\\\ \hline \scriptstyle\text{SAFER-LSTM (Within-Task)} &\scriptstyle58.82&\scriptstyle58.60&\scriptstyle74.83&\scriptstyle65.98&\scriptstyle66.83&\scriptstyle62.29\\\\ \scriptstyle\text{SAFER-LSTM (Original)} &\scriptstyle61.89&\scriptstyle60.94&\scriptstyle76.64&\scriptstyle69.84&\scriptstyle\mathbf{69.27}&\scriptstyle\mathbf{65.39}\\\\ \hline \end{array}$$ >**[Q3]** Are SAFE and SAFER hyperparameters tuned over identical search spaces and identical budgets? If so, please specify the hyperparameter search spaces. **A:** Sorry for the confusion. We use the same hyperparameter tuning protocol for SAFE and SAFER, with identical search spaces for learning rates and regularization coefficients in matched comparisons. We provide the complete hyperparameter search spaces in the appendix to ensure reproducibility. We have added these details to Table 36 and 37 of the revised appendix.
Response to Reviewer LEy2 (1/3)
A: We thank the reviewer for raising this important question. Table R1 shows that both supervised neuron selection and FFN output-direction reconstruction contribute to SAFER's performance gains. Neuron selection provides discriminative information, while output-direction reconstruction further improves detection performance.
We compare SAFER-Raw, which directly uses AUC-selected neuron activations, with the full SAFER using reconstructed representations. With the same selected Top-100 neurons, reconstruction further increases the average Unseen AUC to 80.74% for MLP and 67.79% for LSTM.
These results indicate that neuron selection provides performance gains, while representation reconstruction using the pretrained FFN output projection weights offers additional improvements. The updated results are reported in Table 6 of the revised paper.
Table R1. Effect of Neuron Selection and FFN Reconstruction.
\hline \scriptscriptstyle\text{Method} &\scriptscriptstyle\text{OpenVLA-Seen} &\scriptscriptstyle\text{OpenVLA-Unseen} &\scriptscriptstyle\pi_0\text{-Seen} &\scriptscriptstyle\pi_0\text{-Unseen} &\scriptscriptstyle\text{WidowX-Seen} &\scriptscriptstyle\text{WidowX-Unseen} &\scriptscriptstyle\text{Avg.-Seen} &\scriptscriptstyle\text{Avg.-Unseen}\\\\ \hline \scriptscriptstyle\text{SAFE-LSTM} &\scriptscriptstyle\mathbf{63.34}&\scriptscriptstyle\mathbf{65.33}&\scriptscriptstyle59.46&\scriptscriptstyle63.91&\scriptscriptstyle81.14&\scriptscriptstyle69.31&\scriptscriptstyle67.98&\scriptscriptstyle66.18\\\\ \scriptscriptstyle\text{Raw Random-100-LSTM} &\scriptscriptstyle54.20&\scriptscriptstyle54.91&\scriptscriptstyle70.32&\scriptscriptstyle66.30&\scriptscriptstyle48.39&\scriptscriptstyle46.96&\scriptscriptstyle57.64&\scriptscriptstyle56.08\\\\ \scriptscriptstyle\text{SAFER-Raw-LSTM} &\scriptscriptstyle58.91&\scriptscriptstyle61.04&\scriptscriptstyle73.23&\scriptscriptstyle65.80&\scriptscriptstyle75.26&\scriptscriptstyle65.25&\scriptscriptstyle69.13&\scriptscriptstyle64.03\\\\ \scriptscriptstyle\text{SAFER-LSTM} &\scriptscriptstyle61.89&\scriptscriptstyle60.94&\scriptscriptstyle\mathbf{76.64}&\scriptscriptstyle\mathbf{69.84}&\scriptscriptstyle\mathbf{85.31}&\scriptscriptstyle\mathbf{72.60}&\scriptscriptstyle\mathbf{74.61}&\scriptscriptstyle\mathbf{67.79}\\\\ \hline \scriptscriptstyle\text{SAFE-MLP} &\scriptscriptstyle\mathbf{68.78}&\scriptscriptstyle62.93&\scriptscriptstyle70.00&\scriptscriptstyle75.12&\scriptscriptstyle88.69&\scriptscriptstyle87.78&\scriptscriptstyle75.82&\scriptscriptstyle75.28\\\\ \scriptscriptstyle\text{Raw Random-100-MLP} &\scriptscriptstyle64.26&\scriptscriptstyle57.44&\scriptscriptstyle45.48&\scriptscriptstyle51.44&\scriptscriptstyle79.96&\scriptscriptstyle77.82&\scriptscriptstyle63.23&\scriptscriptstyle62.23\\\\ \scriptscriptstyle\text{SAFER-Raw-MLP} &\scriptscriptstyle68.28&\scriptscriptstyle71.05&\scriptscriptstyle77.08&\scriptscriptstyle77.00&\scriptscriptstyle89.16&\scriptscriptstyle\mathbf{89.19}&\scriptscriptstyle78.17&\scriptscriptstyle79.08\\\\ \scriptscriptstyle\text{SAFER-MLP} &\scriptscriptstyle68.32&\scriptscriptstyle\mathbf{71.60}&\scriptscriptstyle\mathbf{77.83}&\scriptscriptstyle\mathbf{81.43}&\scriptscriptstyle\mathbf{89.44}&\scriptscriptstyle89.18&\scriptscriptstyle\mathbf{78.53}&\scriptscriptstyle\mathbf{80.74}\\\\ \hline \end{array}$$