acceptodds
Desk rejected at ICLR 2027

SmartVLA: A Steerable Action Interface for Fine-Grained, Multimodal, and Corrective Instructions

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing vision-language-action (VLA) models can perform diverse manipulation tasks from task-level instructions, but strong task performance does not ensure fine-grained semantic following, multimodal instruction understanding, or mid- execution corrective control. To this end, we propose SmartVLA, which con- structs semantic and spatial supervision through action-aligned segmentation and grade-conditioned instruction augmentation, and combines warmup, pretraining, and adaptation to explicitly learn the mapping from current instructions to local actions. The resulting unified instruction interface supports natural human steering and enables SmartVLA to serve as a low-level control interface for higher-level agentic systems. To systematically assess these three capabilities, we introduce LIBERO-SMART, a benchmark comprising Semantic, Visual Prompt, and Con- trol suites with 120 tasks and 6,000 evaluation instances. We also evaluate basic task performance, generalization across distributions, and real-world deployment on LIBERO, LIBERO-Plus, LIBERO-Pro, RoboTwin, and the Piper real-robot platform. Experiments show that SmartVLA preserves strong task performance while more reliably following diverse instructions, visually specified commands, and instructions updated during execution.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Embodied Intelligence, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NMmXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents SmartVLA, a framework designed to improve the fine-grained instruction-following and steerability of vision-language-action (VLA) models. The authors identify three important limitations of existing VLA models: fine-grained semantic understanding, multimodal instruction following, and mid-execution corrective control. To address these issues, the paper proposes an action-aligned data construction pipeline and a three-stage training strategy. Additionally, the authors introduce LIBERO-SMART, a benchmark consisting of 120 tasks and 6,000 evaluation instances to systematically evaluate these capabilities. Experimental results demonstrate improvements over existing VLA models across several simulation benchmarks and real-world robotic tasks.

Overall, I find this paper interesting and well-motivated. It addresses a clearly defined and practically important problem, and the proposed benchmark is a valuable contribution to evaluating the instruction-following capabilities of VLA models. The experimental results are generally encouraging. However, I have several concerns regarding the novelty of the benchmark, the comprehensiveness of the model evaluation, and the representativeness of the simulation environment.

Critical strengths
  1. Clear motivation and problem formulation. The paper identifies an important gap between conventional task completion and fine-grained instruction following in existing VLA models. The three proposed evaluation dimensions are well-motivated and relevant to practical robotic applications.
  2. Valuable benchmark contribution. LIBERO-SMART provides a systematic evaluation of semantic following, visual prompting, and corrective control. These capabilities are not sufficiently captured by conventional manipulation benchmarks, making this evaluation setting potentially useful to the community.
  3. Encouraging experimental results. The proposed method demonstrates consistent improvements across multiple benchmarks and two VLA architectures. The inclusion of real-world robotic experiments further strengthens the empirical evaluation.
Critical weaknesses
  1. Limited novelty compared with existing benchmarks. A substantial portion of the proposed LIBERO-SMART benchmark appears to overlap with RoboSemanticBench in terms of its evaluation objectives and task design. In particular, the emphasis on fine-grained semantic understanding and instruction-conditioned robotic behavior seems closely related to existing efforts. Although LIBERO-SMART introduces additional evaluation dimensions, the paper does not sufficiently clarify its unique contributions relative to RoboSemanticBench. A more detailed comparison of their task coverage, evaluation protocols, and capabilities would help establish the novelty and necessity of the proposed benchmark.
  2. Limited coverage of VLA architectures. The evaluated baselines primarily focus on conventional VLA architectures, leaving the performance of other important modeling paradigms unclear. For example, it would be interesting to understand how purely discrete action models, such as PD-VLA, and world-action models (WAMs), such as 4D-WAM, perform on LIBERO-SMART. These models may exhibit different characteristics in semantic understanding, action planning, and corrective control. Without evaluating such approaches, it is difficult to determine whether the identified limitations are general across different VLA paradigms or primarily associated with the architectures considered in this paper.
  3. Limited complexity and representativeness of the simulation environment. LIBERO-SMART is built upon the relatively simple RoboSuite simulation environment, which primarily involves constrained tabletop manipulation scenarios. While this setting enables controlled and reproducible evaluation, its limited scene complexity, interaction diversity, and environmental dynamics may not adequately reflect the challenges encountered in more realistic robotic applications.
Questions for authors

See Weaknesses.

Reviewer_4LExOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that VLAs fine-tuned on task-level instructions often struggle to follow fine-grained commands, visual prompts, or mid-execution corrections, and introduces SmartVLA to overcome this limitation primarily through a data-driven approach.

Existing demonstrations, comprising 145K trajectories from DROID, AgiBot-World, RH20T, and MolmoAct, are segmented into roughly 859K control-phase segments using action, gripper, and visual cues. Each segment is categorized by interaction type, and a suite of grade-dependent rewrites (object-agnostic atomic motions, task rewrites, and in-image visual prompts) is generated using Qwen3.8-27B and verified with GPT-5.6-luna.

The training framework consists of three stages:

  1. Image-free warmup of the action head on atomic commands;
  2. Mixed pretraining on both original and local supervision;
  3. Domain-specific adaptation, where target-domain demonstrations are reconstructed using the same pipeline.

The authors also introduce LIBERO-SMART, an evaluation benchmark spanning Semantic, Visual Prompt, and Control suites (6,000 instances, evaluated via trajectory-level contracts).

Instantiated on a π0.5\pi_{0.5}-style architecture and Qwen3-VL-4B equipped with an OFT head, SmartVLA demonstrates substantial performance gains across LIBERO-SMART (scoring 14.7/14.0 vs. 7.2 for baseline π0.5\pi_{0.5}), LIBERO-Plus, LIBERO-Pro, RoboTwin 2.0, and a real-world bimanual Piper robot setup.

Critical strengths
  1. The data construction aligns each instruction with the specific control phase it governs, and the ablations point to this grounding, together with pretraining, as the main source of the improvement. Substituting phase-aligned segmentation with fixed-interval cuts of roughly the same number and length drops the LIBERO-SMART score from 14.67 to 7.45. Enriching whole-trajectory descriptions without local segmentation (a FineVLA-style baseline) yields just 7.00, while omitting the image-free warmup stage costs roughly two points (Table 5). I weigh this heavily because it points to a practical, reusable mechanism, how demonstrations are partitioned and relabeled, rather than an architectural redesign, and the main results hold for both action heads.

  2. The performance gains extend well beyond the authors' own benchmark. Under LIBERO-Pro's task perturbations, π0.5\pi_{0.5} achieves only 0.75, whereas SmartVLA reaches 32.35 and 36.10 (Table 3). On LIBERO-Plus, accuracy increases from 80.3 to 84.0; on RoboTwin Clean2Random, from 58.4 to 67.1 (Table 4); and across 600 trials per model on the bimanual Piper robot, the average success rate improves from 45.0 to 58.5, all while maintaining a 98.3 score on standard LIBERO. The LIBERO-Pro task-perturbation result in particular supports the instruction-following claim, and the other benchmarks show that it does not come at the cost of general performance.

Critical weaknesses
  1. The headline LIBERO-SMART comparison is not a like-for-like evaluation. The benchmark's atomic-motion and visual-prompt instructions follow the rewrite families generated for SmartVLA's training, including during the LIBERO adaptation stage (Appendix F.4). In contrast, the baselines were fine-tuned solely on the original LIBERO instructions. So part of the gap between 7.2 and 14.7 reflects whether a model saw these instruction formats during training. Table 5 points the same way: SmartVLA-Base adapted on the original LIBERO data reaches only 8.73. A necessary control is missing: the strongest baseline (π0.5\pi_{0.5} loaded from released weights) adapted on the same rewritten LIBERO dataset. While the "From raw VLM" ablation (scoring 7.05) indicates that in-domain rewritten data alone provides little benefit without SmartVLA pretraining, it initializes from a vanilla VLM rather than a robot-pretrained model, leaving the question unresolved. With absolute strict success rates around 15%, characterizing the resulting policy as a truly steerable interface also overstates what the evidence shows.

  2. Critical technical details required to interpret the pretraining results are absent. The paper does not clarify whether SmartVLA-Pi was initialized from pre-trained π0.5\pi_{0.5} weights or directly from PaliGemma (Appendix E.3 notes only that it adopts the π0.5\pi_{0.5} architecture). The effective global batch sizes in Tables 12 and 13 are also left blank (except for Piper). Without these parameters, it is hard to tell how much of the gain over the baselines comes from the 145K-trajectory corpus and data recipe versus the initial checkpoint and compute scale—a distinction that is central to the claim that data construction drives the improvements.

Questions for authors

Clarifications Requested:

  1. What are the precise initializations for SmartVLA-Pi and SmartVLA-OFT at the start of the warmup phase? What are the effective global batch sizes and total GPU-hours consumed across the warmup, pretraining, and individual adaptation runs?
  2. For the real-world Piper robot experiments, how many demonstration trajectories were collected, and were the baseline models adapted on the same rewritten dataset or restricted to the original task instructions?
  3. What is the extent of overlap between the LIBERO-SMART evaluation instructions and the rewrites used during LIBERO adaptation (specifically regarding atomic-motion vocabulary, templates, and phrasing patterns)?

Key Experiment Needed:

  1. Adapt π0.5\pi_{0.5} (initialized from its officially released weights) on the identical Dtask+DlocalD_{\text{task}} + D_{\text{local}} LIBERO dataset, then evaluate it on both LIBERO-SMART and LIBERO-Pro. If SmartVLA retains a clear performance advantage under this strictly controlled baseline, the contribution of its pretraining corpus and data recipe would be much better established.
1 comment

I consider the baseline adaptation experiment added in the rebuttal to be a particularly important comparison, as it directly addresses one of my main concerns. I appreciate the authors' effort in conducting this key experiment, which makes the paper more complete and strengthens its empirical evidence. Based on these additional results, I have decided to raise my rating.

Reviewer_2sCmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SmartVLA, a framework for improving fine-grained instruction following, visual grounding, and mid-execution correction in vision-language-action (VLA) models. It constructs action-aligned supervision from robot demonstrations through behavioral segmentation and grade-conditioned semantic and spatial annotation, followed by a three-stage training procedure. The authors also introduce LIBERO-SMART, a benchmark comprising 120 tasks and 6000 evaluation instances that assess semantic constraints, visual prompts, and corrective control. Experiments across two VLA architectures, several simulation benchmarks, and a real-robot platform show improvements over existing baselines in task completion and instruction following.

Critical strengths
  1. The paper makes a useful distinction between task completion and faithful instruction following. LIBERO-SMART evaluates not only goal attainment but also action scope, object restrictions, execution order, and termination conditions through trajectory-level success criteria. Its separation of semantic following, visual prompting, and corrective control provides a more detailed assessment of VLA behavior than conventional goal-based benchmarks. The standardized intermediate states used for corrective-control evaluation also help isolate the effects of instruction changes from preceding execution errors.

  2. A systematic supervision pipeline supported by meaningful ablations and extensive experiments. The proposed pipeline combines behavioral segmentation, grade-conditioned instruction generation, and annotation filtering to construct fine-grained supervision from heterogeneous robot demonstrations. The comparison with length-matched uniform segmentation is particularly convincing, reducing average LIBERO-SMART success from 14.67% to 7.45% and supporting the importance of action-aligned boundaries. Experimental results across two VLA architectures, multiple simulation benchmarks, and real-robot experiments further demonstrate the applicability of the overall framework.

Critical weaknesses
  1. The methodological novelty is insufficient. SmartVLA shares substantial similarities with Steerable Policies, which explores multi-level instructions, trajectory segmentation, and hierarchical control. FineVLA investigates action-aligned fine-grained supervision combined with task-level instructions. The paper does not clearly establish its methodological advances beyond these existing approaches. Although Table 5 provides useful ablations, the "Fine-grained sem. only" setting removes segmentation, warmup, and visual-prompt supervision simultaneously, making it neither a faithful FineVLA reproduction nor a controlled comparison of individual mechanisms. Moreover, external baselines differ in pretraining data and training budgets. Consequently, the reported performance gains do not establish the advantages of SmartVLA's proposed mechanisms over prior approaches. More targeted comparisons are needed to substantiate its novelty and methodological contributions.

  2. Limited evidence for robust instruction following and corrective control. Despite improvements over the baselines, SmartVLA-Pi achieves only 14.67% strict success on LIBERO-SMART, including 8.40% on Visual Prompt and 12.55% on Control. The corrective-control evaluation primarily uses standardized intermediate states, leaving performance under unpredictable interventions during the policy's own execution less clear. The limited analysis of correction latency, failure modes, and statistical variability further restricts the conclusions on robustness. The results demonstrate relative improvements under the evaluated conditions, but provide weaker support for reliable online steerability.

Questions for authors
  1. Novelty and comparison with prior work. What are the substantive methodological differences between SmartVLA, Steerable Policies, and FineVLA, particularly in action-aligned segmentation, instruction generation, and mixed-granularity supervision? Which of SmartVLA's new components account for improvements beyond these existing approaches, and what experimental evidence supports this distinction?

  2. Ablation of the training strategy. Removing a training stage also reduces the number of optimization updates and training examples. How do the authors distinguish the effects of the proposed training sequence from those of additional training? Is there controlled evidence that image-free warm-up improves language-action correspondence over a compute-matched alternative? The independent contribution of spatial visual supervision also remains unclear.

  3. Statistical robustness and evaluation protocol. Can the authors report per-task results and confidence intervals that account for the hierarchical structure of tasks and initializations? This would clarify whether the reported gains are consistent across tasks and layouts. Additionally, Appendix F.3 applies different success aggregation rules to direct policies and agentic systems. What motivates this difference, and how should the corresponding results be compared?

  4. Online corrective control. The Control suite primarily uses standardized intermediate states. How does SmartVLA perform when corrections are issued at unpredictable points during its own execution? Can the authors clarify whether the real-robot experiments cover such interventions and provide corresponding success rates, response latency, or representative failure cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.