acceptodds
Under review as a conference paper at ICLR 2027

From Rollouts to Failure Modes: Behavioral Diagnosis of Vision-Language-Action Models

Abstract

Task success is the standard measure for evaluating Vision-Language-Action (VLA) models, but it provides limited insight into the failures underlying aggregate performance. We introduce embodied failure attribution, a framework that extends success-rate evaluation with behavioral diagnosis. Using Bayesian inference over stage-specific physical interaction evidence from closed-loop rollouts, the framework assigns posterior probabilities to Planning, Grounding, and Action (PGA) failure modes without requiring access to model internals. This provides a common diagnostic representation for characterizing failure structure, tracking its evolution, and informing targeted interventions. Across diverse VLA architectures and task settings, we find that Action is a recurring source of failure, while the relative contributions of PGA failure modes vary across models and tasks. During training, the framework distinguishes improvements in overall performance from persistent failure modes: Qwen3VL-PI achieves higher task completion on CALVIN, yet Action remains the dominant failure dimension at every evaluated checkpoint, while Grounding attribution declines overall and Planning attribution fluctuates. Under unseen-object conditions, attribution identifies Action as the dominant failure mode, providing additional insight into the observed generalization failures. Finally, an intervention study illustrates how diagnosis can inform model improvement. Action-level correction raises success on an Action-limited task from 51% to 61%, but lowers success on a task with a different failure profile from 65% to 55%, suggesting that the benefit of a correction depends on the failure it addresses. Together, these results establish behavioral failure attribution as a diagnostic complement to aggregate performance evaluation, supporting the analysis of VLA systems across model comparison, training, and generalization, while providing a basis for targeted improvement.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language-Action models, capability diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mSTvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a diagnostic framework for current VLA failure cases, by attributing failures along Planning, Grounding, and Action dimensions. It extracts stage-level evidence from robot rollouts and uses a Bayesian model with event gating to infer the posterior attribution of different failure modes. The paper evaluates the framework across several VLA models on manipulation benchmarks, and further analyses training dynamics, distribution shifts, and action-level interventions.

Critical strengths
  1. The problem of going beyond binary success/failure evaluation is meaningful for understanding VLA behavior.
  2. The stage-based evidence representation and event-gating mechanism are intuitive and well motivated, and the framework is evaluated across multiple models, tasks, and benchmarks.
Critical weaknesses
  1. The paper motivates Bayesian inference as a way to represent competing and overlapping failure explanations, but does not establish its empirical advantage over a rule-based baseline using the same evidence and event gates. In the simulated settings, a much simpler event-based aggregation or rule-based attribution could potentially provide a similar diagnostic profile, yet the paper does not establish what additional value the Bayesian formulation provides. Moreover, the prior is fixed and the likelihoods are manually specified rather than learned from data. The posterior magnitudes depend on manually specified priors and likelihoods, but the paper does not establish their sensitivity to these choices or their empirical calibration.

  2. The main evaluation is primarily simulation-based, where privileged physical information is readily available. The only real world evidence is a single qualitative DROID episode, which is insufficient to establish that the whole framework works reliably on real robots.

Questions for authors
  1. Why is the Bayesian formulation necessary compared with directly counting or aggregating the observable failure events in simulation?

  2. Since the likelihoods are manually specified, how sensitive are the results to the choice of prior and likelihood parameters?

Reviewer_VY94OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a behavioral failure attribution framework for Vision-Language-Action (VLA) models. The method decomposes failures into Planning, Grounding, and Action (PGA) dimensions and uses Bayesian inference over stage-specific physical interaction evidence to estimate failure probabilities. The framework is evaluated across multiple VLA architectures and benchmarks, including SimplerEnv, CALVIN, RoboTwin, and VLA-Arena. The authors analyze failure profiles across models, tasks, training checkpoints, and distribution shifts, and present an intervention study to illustrate the potential utility of the proposed diagnosis.

Critical strengths

S1. Relevant problem formulation.

Understanding why VLA models fail, rather than relying solely on task success rates, is an important research direction. The proposed decomposition into Planning, Grounding, and Action provides an interpretable way to organize behavioral failures.

S2. Broad experimental coverage.

The paper considers multiple VLA architectures, benchmarks, and evaluation conditions, demonstrating the feasibility of applying the proposed framework across different settings.

Critical weaknesses

W1. The central Bayesian diagnosis relies on manually specified likelihoods without validation.

The proposed framework uses manually specified likelihood parameters (Appendix B.2, Table 8) and a uniform prior over the eight PGA states. These parameters are neither learned nor calibrated from data. Consequently, the resulting posterior probabilities depend directly on subjective modeling choices.

This raises a fundamental concern about the reliability of the proposed diagnosis. In particular:

  • No quantitative evaluation using ground-truth failure annotations or controlled fault injection is provided to establish attribution accuracy.

  • No posterior calibration or sensitivity analysis is conducted for the manually specified likelihood parameters, priors, or evidence thresholds.

  • Different likelihood configurations are used for certain benchmarks (Appendix B.2), making it difficult to determine whether differences in failure profiles reflect actual behavioral differences or changes in the inference model.

Since the main contribution is the identification of failure modes, demonstrating that the inferred probabilities reliably reflect the underlying functional failures is essential. The current experiments primarily report outputs of the manually specified probabilistic model without independently validating their correctness.

W2. The methodological contribution and practical utility of Bayesian attribution are not established.

The proposed inference procedure primarily combines manually designed evidence-to-failure mappings with standard Bayesian inference. However, the paper does not demonstrate why this probabilistic formulation is preferable to simpler rule-based diagnostics using the same physical evidence.

For example, target mismatch, localization error, contact failure, and lift failure already provide direct information about different stages of task execution. Without comparisons against simple heuristic or rule-based baselines, it is unclear what additional diagnostic value the Bayesian posterior provides.

Furthermore, the intervention experiment in Section 4.5 does not directly validate attribution-guided decision-making. The same action-level correction is applied across tasks, and the resulting performance differences are interpreted retrospectively using the inferred failure profiles. There is no prospective evaluation showing that PGA-based intervention selection outperforms fixed or evidence-based intervention strategies.

Consequently, neither the reliability nor the practical advantage of the proposed diagnostic framework is convincingly demonstrated.

Questions for authors

Q1. Reliability of manually specified likelihoods.

How were the likelihood parameters in Table 8 determined? Can the authors provide quantitative sensitivity or calibration results demonstrating that the inferred failure probabilities and dominant failure classifications remain reliable under alternative reasonable parameter settings?

Q2. Attribution accuracy and added value.

Can the authors evaluate the proposed PGA attribution against independently annotated failure modes or controlled fault injection, and compare it with simple rule-based diagnostics using the same physical evidence?

Q3. Attribution-guided intervention.

Can the authors demonstrate that selecting interventions based on the proposed PGA posterior provides measurable improvements over fixed or heuristic intervention selection?

Reviewer_ZBkFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a behavioral failure attribution framework for Vision-Language-Action (VLA) models, which diagnoses failures along three functional dimensions: Planning, Grounding, and Action (PGA). By combining stage-specific physical interaction evidence with gated Bayesian inference, the framework provides interpretable failure profiles without requiring access to model internals. Experiments across multiple VLA architectures, benchmarks, training checkpoints, and distribution shifts demonstrate its potential for characterizing failure behaviors beyond aggregate task success.

Critical strengths
  1. The paper addresses a meaningful limitation of conventional VLA evaluation: task success rates provide limited insight into the underlying causes of failure. A unified behavioral diagnosis framework could offer valuable information for model comparison and improvement.

  2. The stage-specific evidence extraction and event-gated Bayesian inference are conceptually well motivated. In particular, distinguishing unobserved downstream outcomes from actual failures and allowing multiple contributing failure dimensions are sensible design choices.

  3. The evaluation spans diverse VLA architectures, manipulation benchmarks, training checkpoints, and distribution shifts. The shared-backbone and primitive-level comparisons are particularly useful for revealing differences in failure profiles across models and tasks.

  4. The intervention experiments provide preliminary evidence that the effectiveness of action-level corrections varies across tasks. Although exploratory, these results motivate the potential use of behavioral diagnosis to inform targeted model improvement.

Critical weaknesses
  1. The most significant limitation is the lack of independent evidence that the inferred PGA probabilities correspond to actual failure causes. Without expert annotations, controlled fault injection, or other ground-truth validation, it remains unclear whether the framework identifies genuine functional bottlenecks or primarily reflects its predefined diagnostic assumptions.

  2. The Bayesian likelihoods are manually designed, and the inference assumes conditional independence among observations given the latent PGA states. Different likelihood configurations are also used across some benchmarks. Without sensitivity analysis or posterior calibration, the robustness and comparability of the resulting failure profiles remain uncertain. In particular, the recurring dominance of Action might partly reflect the evidence model rather than an independently established behavioral property.

3.While the integration of physical interaction evidence, event gating, and Bayesian inference is useful, the incremental value of this formulation is not clearly established. Comparisons with simpler rule-based diagnostics, alternative probabilistic formulations, or existing process-aware evaluation methods would help clarify whether the proposed approach provides meaningful advantages beyond heuristic failure attribution.

Questions for authors
  1. How can the correctness of PGA attribution be independently validated?

  2. How robust are the inferred failure profiles to the probabilistic assumptions? Do the dominant failure dimensions remain consistent under alternative likelihood parameters, priors, or gating configurations? Can the authors assess posterior calibration or demonstrate that Action dominance is not primarily induced by the attribution model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.