- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies failure detection and diagnosis for frozen vision-language-action policies. It trains SAEs on token-level internal activations, assigns features names from predefined state and decision concepts, and tests these names on held-out episodes. Temporal statistics of the named features feed a linear failure detector, whose concept-level contributions and activation histories are then verbalized by a language model.
The main contribution is using the same interpretable features for detection and diagnosis. Experiments cover several VLA models in simulation and real-world.
Critical strengths
-
I appreciate that the explanation is tied to the features used by the detector, rather than generated independently of the alarm. The linear score decomposition makes the contribution of each named concept inspectable. This provides a good basis for understanding why the monitor flagged an execution, without modifying the deployed policy.
-
The naming procedure uses quantitative criteria and evaluates whether the selected names remain valid on held-out episodes. This is better evidence than only inspecting a feature’s highest-activation examples. Closed-loop interventions further examine the behavioral relevance of selected decision features, complementing the observational concept-alignment results
Critical weaknesses
-
Limited Novelty. Paper [1] already use SAEs to interpret VLA features and validate them through closed-loop steering. Paper [2] also employ per-token SAEs for concept identification, while paper [3] detect failures from VLA activations. MIND introduces a more systematic naming procedure and integrates these components for failure diagnosis. However, I find the methodological advance limited, and the added value of this integration is not sufficiently established.
-
Insufficient Justification for SAE. Since MIND relies on predefined concepts and their labels, a natural alternative is to train linear concept probes on the same frozen activations and use their predictions in the same temporal monitoring and diagnosis pipeline. I notice the paper reports that linear probes recover all tested concepts on held-out robot attempts, but does not provide this end-to-end comparison. Without it, the necessity of SAE-based decomposition remains unclear.
-
Limited Predefined Concept Vocabulary. Only features aligned with predefined concepts enter the monitor, potentially excluding informative but unnamed features. Held-out naming accuracy does not guarantee that the vocabulary captures the distinctions needed for failure diagnosis. This concern is particularly relevant given the limited failure-category accuracy: 15/48 for OpenVLA and approximately one-third for π0.5. I would appreciate further analysis of how vocabulary coverage affects diagnostic performance.
[1] Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Models, 2026
[2] Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models, 2026
[3] SAFE: Multitask Failure Detection for Vision-Language-Action Models, 2025
I would consider raising my score if the response establishes a substantive benefit over direct concept probing and provides stronger evidence for the reliability and scope of the diagnoses.
Questions for authors
-
How does MIND compare with linear concept probes followed by the same temporal monitor and diagnosis module, using the same activations, concept labels, and data splits? A comparison of failure detection and diagnosis would directly clarify the added value of the SAE representation.
-
For the reported failure-category errors, can you distinguish cases where the necessary concept is absent from the vocabulary from cases where an available concept is incorrectly detected or interpreted? This would clarify whether the main limitation lies in concept coverage, feature alignment, or the language diagnosis.