- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the need for VLA systems to detect failures promptly, recover execution, and maintain task context during long-horizon tasks. However, the current manuscript does not clearly describe the key recovery mechanisms, reports internally inconsistent primary CALVIN metrics, and does not adequately specify the recovery budget, training configurations, or the scope of skill-library updates during evaluation. Consequently, the current version does not sufficiently support its full claims regarding a new paradigm, generalizable recovery, and skill expansion.
Critical strengths
-
The paper addresses an important practical problem: VLA systems need to promptly detect intermediate failures, recover execution, and maintain task context during long-horizon manipulation.
-
The performance improvements across multiple VLA backbones suggest that the framework may be broadly applicable. The larger gains on long-horizon tasks and weaker backbones are particularly encouraging.
-
The component ablations, retry-only baseline comparison, and verification-frequency analysis provide useful evidence for understanding the framework’s effectiveness and computational trade-offs.
Critical weaknesses
-
The principal CALVIN metrics are internally inconsistent. According to the definition in Appendix B.1, average chain length should equal the sum of the success probabilities for completing at least one through five consecutive instructions. However, the TVA success rates in Table 2 imply average chain lengths of 4.307 and 4.479, while the reported values are 4.41 and 4.59. Table 4 also reports 4.59 for ABC→D, inconsistent with Table 2. These discrepancies exceed what rounding can explain and weaken confidence in the quantitative conclusions.
-
The key recovery mechanisms are insufficiently described. The manuscript does not specify the implementation, training data, or training objective of the future recovery-trajectory predictor. Tracking historical motion alone does not explain how future trajectories conditioned on a recovery goal are generated. Furthermore, the skill-novelty function, threshold, retrieval procedure, and reuse mechanism lack concrete definitions.
-
The evaluation protocol does not sufficiently distinguish the sources of the performance gains. Recovery-attempt limits, action budgets, model checkpoint configurations, and skill-library update boundaries are unclear. Consequently, it is difficult to determine whether the improvements arise from better recovery decisions, additional execution opportunities, or experience accumulated during testing. The absence of uncertainty estimates also limits the interpretation of smaller performance differences.
-
The contributions of explicit CoT and role specialization have not been clearly established. The configuration of “w/o CoT Planner” is insufficiently specified and may conflate removing explicit reasoning with removing task decomposition. The existing evidence does not clearly establish what distinctive contributions the framework offers beyond a conventional planning–verification–replanning pipeline with shared memory.
Questions for authors
-
What model or algorithm generates the future recovery trajectories? What training data and objectives are used? How are the visual points selected?
-
Do Full TVA and the retry-only baseline use the same model checkpoints, action budgets, timeouts, and maximum number of recovery attempts? Is the skill library reset between evaluation chains, or continuously updated throughout testing?
-
What functionality is retained in the “w/o CoT Planner” ablation? Does it preserve the same model, structured task decomposition, observation inputs, and output interface?