- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a theoretical analysis of the components of AUROC score in VLA failure detection, and leverages this view to gain empirical insight about reporting practices. The analysis centers around a decomposition of AUROC into components corresponding to representation loss, model misspecification, and estimation loss, with each component corresponding to a step in a typical machine learning pipeline. Experiments are performed in the LIBERO simulation environment, using two OpenVLA variants and .
Critical strengths
The goal of rigorous end-to-end understanding of failure detection pipelines in robot policies is important and understudied.
Critical weaknesses
I find that the paper's main analysis is largely a restatement of well-established principles in modern machine learning, while the experiments do not convincingly demonstrate the work’s practical significance.
On the theory side, the decomposition is well-known, that to learn a predictor atop a frozen encoder, you need the right representation, the right model, and enough data. In my opinion this is common knowledge to machine learning practitioners, both in industry and academia. Offering some novel way to leverage this framing to improve performance could be a meaningful contribution (see below), but the observation itself is somewhat trivial. I am not familiar enough with the relevant statistical learning literature to judge the novelty of the theoretical results themselves, but I find that the authors do not convincingly establish their significance for the problem of VLA failure detection, which I believe is the goal here.
I find that the usefulness of the theory is not demonstrated via experiments. Given the setup, one might expect experiments that measure the different terms in Equation 6, and use that information to inform how we might better build or choose our systems. However, the experiments do not separately quantify any of these terms (e.g., the representation loss, which is arguably the most challenging to evaluate), and thus do not demonstrate how the proposed decomposition can be used to address challenges in building failure detection pipelines. Overall, it is hard to identify any new empirical insights that this work gives us.
Also, the focus on SAEs is not motivated, especially for a reader who is not deeply familiar with them. Table 3 seems to show that detectors trained atop the typical hidden state perform better, and I think that hidden states are the most standard choice for such systems.
Questions for authors
Equation 6 seems straightforward, which subsequent results do the authors consider most significant, and what do they show about building VLA failure detectors that was not previously understood?