- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a diagnostic framework for current VLA failure cases, by attributing failures along Planning, Grounding, and Action dimensions. It extracts stage-level evidence from robot rollouts and uses a Bayesian model with event gating to infer the posterior attribution of different failure modes. The paper evaluates the framework across several VLA models on manipulation benchmarks, and further analyses training dynamics, distribution shifts, and action-level interventions.
Critical strengths
- The problem of going beyond binary success/failure evaluation is meaningful for understanding VLA behavior.
- The stage-based evidence representation and event-gating mechanism are intuitive and well motivated, and the framework is evaluated across multiple models, tasks, and benchmarks.
Critical weaknesses
-
The paper motivates Bayesian inference as a way to represent competing and overlapping failure explanations, but does not establish its empirical advantage over a rule-based baseline using the same evidence and event gates. In the simulated settings, a much simpler event-based aggregation or rule-based attribution could potentially provide a similar diagnostic profile, yet the paper does not establish what additional value the Bayesian formulation provides. Moreover, the prior is fixed and the likelihoods are manually specified rather than learned from data. The posterior magnitudes depend on manually specified priors and likelihoods, but the paper does not establish their sensitivity to these choices or their empirical calibration.
-
The main evaluation is primarily simulation-based, where privileged physical information is readily available. The only real world evidence is a single qualitative DROID episode, which is insufficient to establish that the whole framework works reliably on real robots.
Questions for authors
-
Why is the Bayesian formulation necessary compared with directly counting or aggregating the observable failure events in simulation?
-
Since the likelihoods are manually specified, how sensitive are the results to the choice of prior and likelihood parameters?