acceptodds
Under review as a conference paper at ICLR 2027

VLA-CoRe: Empowering Vision Language Action Model with Cognitive Reasoning

Abstract

Vision Language Action (VLA) models have demonstrated strong behavioral cloning performance on short-horizon manipulation tasks, yet they fundamentally lack cognitive awareness: the ability to estimate their own uncertainty, identify the source of potential failure, and selectively invoke specialized reasoning modules before committing to an action. We propose VLA-CoRe, a cognitive VLA framework that augments a pretrained multimodal LLM with an Uncertainty Monitor that gates three differentiable reasoning modules: (1) Perception Query, which resolves ambiguity by re-examining occluded or visually complex scene regions; (2) World Imagination, which performs counterfactual "what-if" simulations to predict the outcome of an action before physical execution; and (3) Causal Tracing, which attributes unexpected failures to specific past reasoning steps to revise the current plan, based on uncertainty estimates. The reasoning outcomes (from these three reasoning modules) are combined into a visual plan latent that conditions a Diffusion Transformer action decoder for robust low-level control. To train VLA-CoRe without expensive manual annotation, we introduce CIDAS (Cognitive Instruction Data for Autonomous robot reaSoning), a dataset of 170k conversation-style cognitive reasoning samples extracted from existing behavior cloning trajectories across seven structured task types. Training combines (i) a Supervised Fine-Tuning stage on CIDAS and (ii) Group Relative Policy Optimization (GRPO) guided by a cognitive objective. Crucially, our framework improves failure recovery rates by 21% and reduces expected calibration error by 38%, marking a significant step toward resilient & self-aware robots. Code and data will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language Action Model, Robotics, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5JxWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VLA-CoRe, which uses a confidence monitor to selectively invoke local perception queries, world-model predictions, and history tracing before generating robot actions through a diffusion decoder. The authors construct CIDAS, a dataset of 171K cognitive dialogue samples, and combine supervised fine-tuning with GRPO, evaluating task success, calibration, and failure recovery in simulation and on a physical robot.

Critical strengths
  1. Using confidence to decide whether to act directly, re-examine observations, or predict outcomes is a reasonable approach to manipulation under uncertainty. In Table 4, the full method outperforms both always-on and disabled reasoning variants, providing preliminary support for dynamic routing.

2.The paper evaluates long-horizon manipulation, recovery from external perturbations, and physical robot execution, and reports a substantial recovery-rate decrease when history tracing is removed. If the experimental data and protocols are clarified, these results could provide meaningful support for practical utility.

Critical weaknesses
  1. The CALVIN average task count mathematically contradicts the sequential success rates. Under Table 20’s definition, the average number of completed tasks equals SR1SR_1 +⋯\cdots+SR5SR_5, with SR5≤SR4SR_5\le SR_4. The reported first four rates imply an upper bound of 3.31, incompatible with the reported 4.14, directly undermining confidence in a main performance result.

  2. The proposed “calibration reward” does not incentivize reporting the true success probability. Equation (13) rewards high confidence after success and low confidence after failure, making its expectation linear in predicted confidence and generally favoring extreme values rather than the true probability. The additional overconfidence penalty does not guarantee calibration either, requiring justification and comparisons with Brier or logarithmic scoring.

  3. Algorithm 2 selects the “best counterfactual branch” by proximity to the next expert action, while Algorithm 3 identifies failure onset through motion or pixel changes and assigns causes heuristically. These labels may support auxiliary training, but they do not directly substantiate the claimed counterfactual reasoning and causal attribution capabilities.

  4. The main text moves the target at a randomly sampled timestep, whereas Appendix E.4 applies perturbations at the model’s highest-confidence timestep, without clearly identifying these as separate experiments. These protocols may produce different recovery difficulty, leaving the relationship between the main and supplementary results—and their comparison fairness—unclear.

Questions for authors
  1. Please explain the CALVIN numerical inconsistency. Can you provide SR1SR_1–SR5SR_5, the average completed-task count, and the aggregation code for the same evaluation episodes?

  2. Why is Equation (13) appropriate for calibration? How do task performance and ECE compare when using Brier or logarithmic scoring instead?

  3. Have the counterfactual and attribution labels been independently validated? Can actual branch execution or interventions with known causes establish label accuracy, rather than merely correlation with failure outcomes?

  4. Which perturbation-timing rule is used for failure recovery? Please specify the protocols underlying the main and supplementary tables and whether different models receive perturbations in comparable states.

Reviewer_rgZKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VLA-CoRe, a VLA framework that adds an Uncertainty Monitor on top of a LLaVA-style backbone (CLIP ViT-L/14 + Vicuna-7B). The Monitor has two parts. A ConfidenceHead predicts a per-step confidence c_t from the LLM hidden state. A FailureDetector raises a failure flag f_t based on attention entropy. Using fixed thresholds on (c_t, f_t), the Monitor routes each step to one of four outcomes: proceed directly, invoke Perception Query (RoI re-encoding), invoke World Imagination (a pretrained latent dynamics model), or invoke Causal Tracing (cross-attention over a history buffer to identify a "blamed" past step). The module outputs are fused into a plan latent that conditions a DiT action decoder. To train the system, the authors introduce CIDAS, about 171K conversation-style samples generated automatically from LIBERO and Open-X Bridge demonstrations. The samples span seven task types, and all labels derive from a heuristic uncertainty score u_t computed from pixel statistics and action differences. Training proceeds in two stages: SFT on CIDAS, then GRPO with a composite reward covering task success, calibration, routing agreement with the u_t oracle, and output format. The paper reports gains over ThinkAct on LIBERO-Long (+8.5 TSR), MetaWorld-Hard, and Calvin ABC-D, along with a 38% ECE reduction, a +21.4 pp improvement in failure recovery rate, and real-robot results on a Franka arm.

Critical strengths
  1. Well-motivated problem. Equipping VLAs with calibrated self-assessment, and spending deliberation only when it is warranted, is an important and timely direction. Framing the question as when to reason, rather than whether to reason, is a useful perspective, and the failure-recovery metric targets a capability that standard success-rate benchmarks miss.
  2. Interpretable design. Mapping distinct sources of difficulty (local visual ambiguity, outcome uncertainty, failure) to distinct reasoning modules, connected through a structured token interface, is intuitive and makes model behavior inspectable (e.g., Figures 3–4).
  3. Cheap data pipeline. CIDAS requires no manual annotation and runs on CPU in a few hours. If its labels were reliable, this would be a practical recipe others could reuse.
Critical weaknesses

W1. The supervision signal u_t does not measure epistemic uncertainty The entire method (CIDAS labels, ConfidenceHead BCE targets, and the GRPO routing oracle) rests on the heuristic score u_t (Table 9): σ_occ = 1 − min(std(center crop) / std(full image), 1) is a contrast ratio between image regions. It does not measure occlusion. σ_depth = min(Var(grayscale image) / 10^4, 1) is RGB intensity variance. No depth information is used. σ_spike = min(‖a_t − a_{t−1}‖ / 2, 1) is computed on expert actions in the demonstrations, not on the model's own predictions. None of these quantities depends on the model, so u_t cannot be "epistemic" uncertainty in the usual sense (uncertainty arising from the model's lack of knowledge). It is a fixed property of the image and the demonstration. As a result, the ConfidenceHead is trained to regress image statistics, and calling its output "calibrated epistemic confidence" is not justified.

W2. Several CIDAS labels are not grounded in actual outcomes T2 (Counterfactual Branch). The "ground-truth best branch" is the random perturbation δ_i for which a_t + δ_i lies closest to a_{t+1}. This selects by noise proximity to the next expert action; no outcome is ever simulated. Yet the target text asserts outcomes such as "Branch A: clean contact. Branch B: edge risk." (Table 7). The supervision therefore teaches the model to produce outcome claims that nothing grounds. T7 (Imagination Verification). "World-model accuracy" is proxied by pixel similarity between o_t and o_{t+2} (Table 9). No imagined frame is involved, so the label cannot reflect world-model accuracy. T3 / T6 data source. T3 is generated "only from failed trajectories" (21,250 training samples), but LIBERO and Bridge v2 consist of demonstrations, which are predominantly successful. The paper should state where the failed trajectories come from. Similarly, T6 detects "unexpected perturbations" in demonstrations that contain no perturbations.

W3. Missing baselines; the state-of-the-art claim is not supported The paper cites π0 and π0.5 but does not compare against them, and it omits other strong recent LIBERO baselines such as OpenVLA-OFT. These methods report average LIBERO success above 94%, and above 85% on LIBERO-Long, which is far higher than the 79.4% reported here. The "new state-of-the-art" claim should be removed or properly qualified.

Questions for authors
  1. Where do the failed trajectories used for T3 come from, and how many are there?
  2. How does VLA-CoRe compare with π0 / π0.5 / OpenVLA-OFT and with simple gating baselines at equal compute?
Reviewer_zR8aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VLA-CoRe proposes a cognitive vision-language-action framework that utilizes an uncertainty monitor and three differentiable reasoning modules to gate deliberation and action decoding. To train this framework, the authors introduce CIDAS, a rule-based synthetic dataset of 171K reasoning conversations. While the method achieves strong quantitative results. As noted below, critical weaknesses exist for this work. Specifically, the core claim of "cognitive reasoning" is undermined by reliance on hand-crafted heuristics and rule-based labeling rather than genuine learned introspection, and the evaluation lacks necessary control experiments against simpler replanning baselines as well as comparisons with contemporary state-of-the-art models.

Critical strengths
  1. Uncertainty Awareness and Failure Recovery: this paper successfully addresses a genuine and important gap in current VLA models (i.e., their lack of uncertainty awareness, which is critical for handling compounding errors in long-horizon tasks).
  2. Automated Dataset Generation (CIDAS): this paper introduces CIDAS, an automated pipeline that generates approximately 171,000 rule-based synthetic reasoning conversations without requiring expensive manual annotation.
  3. Strong Quantitative Performance Improvements: Compared to baseline methods like ThinkAct, VLA-CoRe demonstrates significant quantitative gains on benchmarks. This includes an 8.5 percentage point increase in Task Success Rate (TSR) on LIBERO-Long (reaching 79.4%), a 38% reduction in Expected Calibration Error (ECE) from 0.203 to 0.126, and a substantial improvement in Failure Recovery Rate (FRR) from 35.8% to 57.2%.
Critical weaknesses
  1. Unvalidated Cognitive Reasoning (The Core Claim): this paper's primary claim of achieving "cognitive reasoning" lacks genuine validation because all labels in the CIDAS dataset are derived from hand-crafted, rule-based heuristics rather than genuine learned introspection. The confidence scores and routing decisions rely entirely on a hand-designed uncertainty score utu_t. The failure detection flag ftf_t is described as "learned," but Eq. 15 actually functions as a z-score rule with grid-searched thresholds. Finally, the causal diagnosis via attention attribution (Appendix B.4) fails to establish true root causes, meaning performance gains might simply stem from generic replanning rather than the proposed causal tracing.
  2. Limited Methodological Novelty and Missing Controls: The framework combines existing techniques (Perception Query, World Imagination, and Causal Tracing) without demonstrating sufficient methodological novelty or providing controlled comparisons against simpler alternatives (such as a single replanning step or random routing at equivalent invocation rates).
  3. Weak Baselines and Unsupported SOTA Claims: The evaluation lacks comparisons against contemporary, strong state-of-the-art models like π0\pi_0, π0.5\pi_{0.5}, and DreamVLA under matched evaluation protocols on LIBERO-Long and CALVIN, leaving the superiority of the method under question.
Questions for authors

Q1 (Causal Diagnosis): Clarify how failure flag ftf_t is trained and its thresholds chosen; provide control comparisons of causal tracing against simple replanning alone and random step-blaming.
Q2 (Module-wise Training Details): Specify training losses and supervisory signals for each module (such as the World Imagination output MLP, Causal Tracer cross-attention, State Integrator, and DiT action decoder), and clarify which components are updated during GRPO or use cached latents. Q3 (Multi-module Design): Justify the unique benefits over simpler alternatives by comparing performance against a single replanning step or random routing at the same invocation rate (26.9%).
Q4 (Contemporary SOTA): Provide head-to-head comparisons with contemporary models like π0\pi_0, π0.5\pi_{0.5}, and DreamVLA under identical evaluation protocols on LIBERO-Long and CALVIN. Q5 (Figure 5 Naming): Clarify why Figure 5 labels the method as "MCR-VLA (Ours)" (a name used nowhere else) and confirm if it was produced using the exact same model and protocol as Table 2.

1 comment

I appreciate the authors' efforts during the rebuttal to clarify the raised issues through additional experiments. While I found the explanations and experiments for Q2 and W3/Q4 satisfactory, the clarifications for Q1 and W2/Q3 only partially address my concerns. I stand with the decisions of the other two reveivewers and decide to maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.