- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents SmartVLA, a framework designed to improve the fine-grained instruction-following and steerability of vision-language-action (VLA) models. The authors identify three important limitations of existing VLA models: fine-grained semantic understanding, multimodal instruction following, and mid-execution corrective control. To address these issues, the paper proposes an action-aligned data construction pipeline and a three-stage training strategy. Additionally, the authors introduce LIBERO-SMART, a benchmark consisting of 120 tasks and 6,000 evaluation instances to systematically evaluate these capabilities. Experimental results demonstrate improvements over existing VLA models across several simulation benchmarks and real-world robotic tasks.
Overall, I find this paper interesting and well-motivated. It addresses a clearly defined and practically important problem, and the proposed benchmark is a valuable contribution to evaluating the instruction-following capabilities of VLA models. The experimental results are generally encouraging. However, I have several concerns regarding the novelty of the benchmark, the comprehensiveness of the model evaluation, and the representativeness of the simulation environment.
Critical strengths
- Clear motivation and problem formulation. The paper identifies an important gap between conventional task completion and fine-grained instruction following in existing VLA models. The three proposed evaluation dimensions are well-motivated and relevant to practical robotic applications.
- Valuable benchmark contribution. LIBERO-SMART provides a systematic evaluation of semantic following, visual prompting, and corrective control. These capabilities are not sufficiently captured by conventional manipulation benchmarks, making this evaluation setting potentially useful to the community.
- Encouraging experimental results. The proposed method demonstrates consistent improvements across multiple benchmarks and two VLA architectures. The inclusion of real-world robotic experiments further strengthens the empirical evaluation.
Critical weaknesses
- Limited novelty compared with existing benchmarks. A substantial portion of the proposed LIBERO-SMART benchmark appears to overlap with RoboSemanticBench in terms of its evaluation objectives and task design. In particular, the emphasis on fine-grained semantic understanding and instruction-conditioned robotic behavior seems closely related to existing efforts. Although LIBERO-SMART introduces additional evaluation dimensions, the paper does not sufficiently clarify its unique contributions relative to RoboSemanticBench. A more detailed comparison of their task coverage, evaluation protocols, and capabilities would help establish the novelty and necessity of the proposed benchmark.
- Limited coverage of VLA architectures. The evaluated baselines primarily focus on conventional VLA architectures, leaving the performance of other important modeling paradigms unclear. For example, it would be interesting to understand how purely discrete action models, such as PD-VLA, and world-action models (WAMs), such as 4D-WAM, perform on LIBERO-SMART. These models may exhibit different characteristics in semantic understanding, action planning, and corrective control. Without evaluating such approaches, it is difficult to determine whether the identified limitations are general across different VLA paradigms or primarily associated with the architectures considered in this paper.
- Limited complexity and representativeness of the simulation environment. LIBERO-SMART is built upon the relatively simple RoboSuite simulation environment, which primarily involves constrained tabletop manipulation scenarios. While this setting enables controlled and reproducible evaluation, its limited scene complexity, interaction diversity, and environmental dynamics may not adequately reflect the challenges encountered in more realistic robotic applications.
Questions for authors
See Weaknesses.
I consider the baseline adaptation experiment added in the rebuttal to be a particularly important comparison, as it directly addresses one of my main concerns. I appreciate the authors' effort in conducting this key experiment, which makes the paper more complete and strengthens its empirical evidence. Based on these additional results, I have decided to raise my rating.