- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VLA-CoRe, which uses a confidence monitor to selectively invoke local perception queries, world-model predictions, and history tracing before generating robot actions through a diffusion decoder. The authors construct CIDAS, a dataset of 171K cognitive dialogue samples, and combine supervised fine-tuning with GRPO, evaluating task success, calibration, and failure recovery in simulation and on a physical robot.
Critical strengths
- Using confidence to decide whether to act directly, re-examine observations, or predict outcomes is a reasonable approach to manipulation under uncertainty. In Table 4, the full method outperforms both always-on and disabled reasoning variants, providing preliminary support for dynamic routing.
2.The paper evaluates long-horizon manipulation, recovery from external perturbations, and physical robot execution, and reports a substantial recovery-rate decrease when history tracing is removed. If the experimental data and protocols are clarified, these results could provide meaningful support for practical utility.
Critical weaknesses
-
The CALVIN average task count mathematically contradicts the sequential success rates. Under Table 20’s definition, the average number of completed tasks equals ++, with . The reported first four rates imply an upper bound of 3.31, incompatible with the reported 4.14, directly undermining confidence in a main performance result.
-
The proposed “calibration reward” does not incentivize reporting the true success probability. Equation (13) rewards high confidence after success and low confidence after failure, making its expectation linear in predicted confidence and generally favoring extreme values rather than the true probability. The additional overconfidence penalty does not guarantee calibration either, requiring justification and comparisons with Brier or logarithmic scoring.
-
Algorithm 2 selects the “best counterfactual branch” by proximity to the next expert action, while Algorithm 3 identifies failure onset through motion or pixel changes and assigns causes heuristically. These labels may support auxiliary training, but they do not directly substantiate the claimed counterfactual reasoning and causal attribution capabilities.
-
The main text moves the target at a randomly sampled timestep, whereas Appendix E.4 applies perturbations at the model’s highest-confidence timestep, without clearly identifying these as separate experiments. These protocols may produce different recovery difficulty, leaving the relationship between the main and supplementary results—and their comparison fairness—unclear.
Questions for authors
-
Please explain the CALVIN numerical inconsistency. Can you provide –, the average completed-task count, and the aggregation code for the same evaluation episodes?
-
Why is Equation (13) appropriate for calibration? How do task performance and ECE compare when using Brier or logarithmic scoring instead?
-
Have the counterfactual and attribution labels been independently validated? Can actual branch execution or interventions with known causes establish label accuracy, rather than merely correlation with failure outcomes?
-
Which perturbation-timing rule is used for failure recovery? Please specify the protocols underlying the main and supplementary tables and whether different models receive perturbations in comparable states.
I appreciate the authors' efforts during the rebuttal to clarify the raised issues through additional experiments. While I found the explanations and experiments for Q2 and W3/Q4 satisfactory, the clarifications for Q1 and W2/Q3 only partially address my concerns. I stand with the decisions of the other two reveivewers and decide to maintain my score.