- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents SMART, a framework for generating large-scale synthetic demonstrations for articulated-object manipulation. It combines articulation-aware asset annotation, constraint-aware motion generation, LLM-based task generation, and distributed data synthesis to construct over one million demonstrations. Experiments on simulation benchmarks and three real-world robot platforms show improved downstream performance, data efficiency, and promising zero-shot sim-to-real transfer without real-world demonstrations.
Critical strengths
-
Large-scale synthetic data generation for articulated-object manipulation. SMART integrates articulation-aware motion generation, task composition, and distributed data synthesis to produce over one million demonstrations. This provides a potentially valuable data source for VLA pretraining.
-
Promising real-world sim-to-real results. Experiments across three robot platforms and 12 tasks demonstrate that policies trained using synthetic demonstrations can achieve high real-world performance without real-world demonstrations.
Critical weaknesses
- Insufficiently established novelty over existing approaches. The main contributions appear to lie in scaling articulated-object assets and integrating articulation-aware motion generation into an existing synthetic data pipeline. Given the overlap with prior work in agentic task generation and synthetic manipulation data, the technical advance over existing approaches remains unclear.
- Insufficient experimental evidence for the claimed advantages of SMART-Data. The paper lacks controlled comparisons with representative synthetic pretraining approaches, making it difficult to determine whether SMART-Data provides unique benefits beyond those of large-scale synthetic pretraining in general. The modest improvement on LIBERO and the absence of a real-data-pretrained baseline on RoboCasa365 further limit the evidence supporting the dataset's effectiveness and generalizability.
Questions for authors
- Limited novelty over existing synthetic data generation pipelines. SMART combines articulation-aware asset annotation, constraint-aware motion generation, and agentic task composition. However, asset preparation, annotation, and the 23 manipulation skills still require substantial prior engineering, while agentic task generation and skill composition have been explored in prior work, such as GenManip and HumanoidGen. The paper would benefit from a clearer characterization of its technical advances over existing simulation pipelines and a quantitative analysis of the human intervention required in practice.
- Insufficient evidence of unique task coverage and dataset advantages. Many tasks shown in the demonstration video, such as opening cabinets and manipulating microwave doors and knobs, appear to overlap with InternData-A1. The paper does not clearly establish which task types or manipulation capabilities are uniquely enabled by SMART.
- Limited evidence for the effectiveness of SMART-Data pretraining. On LIBERO, SMART-VLA improves over training from scratch by only 2.1 percentage points and remains 1.0 point below the real-data-pretrained baseline. Such gains may reflect the general benefits of pretraining rather than the unique value of SMART-Data. On RoboCasa365, the improvement is larger, but the corresponding real-data-pretrained baseline is missing. Also, the paper lacks a direct comparison with existing synthetic pretraining datasets.
- Limited insight into real-world sim-to-real transfer. Although SMART-VLA achieves promising real-world results, the paper provides limited analysis of why it outperforms the baselines. The contributions of articulation-aware motion generation, contact quality, and domain randomization remain unclear, making it difficult to understand the underlying mechanisms and the conditions under which the proposed approach is most effective. A systematic failure analysis would provide stronger evidence for the proposed method's benefits.
5 comments
3. Failure case analysis
We refer the reviewer to our main response for experiment details.
We report the results in the table below.
| Data Source | Miss the target part (%) | Lose contact when manipulating (%) | Constraint Violation (%) | Others (%) |
|---|---|---|---|---|
| SMART-Data-1000h | 11.7 | 6.7 | 6.7 | 5.0 |
| InternData-A1-Art-1000h | 13.3 | 11.7 | 13.3 | 3.3 |
| SMART-Data-1000h (No-Part-MTL-Rand) | 18.3 | 10.0 | 10.0 | 5.0 |
| SMART-Data-1000h (No-ART-Design) | 11.7 | 15.0 | 11.7 | 1.7 |
We summarize our observations and analysis as follows:
- The advantage of SMART-Data over InternData-A1-Art-1000h lies in constraint adherence and contact maintenance.
- It halves the constraint-violation rate (6.7% vs 13.3%) and reduces the loss-of-contact rate by approximately 40% (6.7% vs 11.7%), while the miss-the-target-part rate remains nearly comparable (11.7% vs 13.3%).
- The performance gain therefore originates from the constraint- and contact-aware designs. Since the per-task performance gap follows the same trend as that of the ablated variants, we defer the detailed analysis to the ablation and failure case analysis below.
- Part-level material randomization mainly benefits perception, namely the visual grounding of the target manipulable part: disabling this component increases the miss-the-target-part rate from 11.7% to 18.3%.
- In the drawer-pulling task, the handle and drawer body share identical material and color, yielding weak visual cues for identifying the manipulable part.
- The policy trained on data augmented with part-level material randomization exhibits more stable performance under such visual ambiguity, demonstrating the effectiveness of this design for visual grounding.
- Articulation-aware designs mainly facilitate action inference: removing them doubles the loss-of-contact rate (6.7% to 15.0%) and elevates the constraint-violation rate to 11.7%.
- For example, in the cabinet door sliding task, policies trained without these designs often lose grip on the handle and only partially open the door, or violate the rail constraint by pulling the door outward and damaging it.
- When opening a toolbox lid, the policy pushes the entire box away rather than rotating the lid along the hinge constraint, so it fails to open the box.
- These observations demonstrate the contribution of articulation-aware designs to inferring actions that respect joint kinematic constraints.
We hope the experiments above can address your concerns in this part.
Answering the questions
Answering the questions
Q1: Regarding your concerns about the novelty of our work, we have elaborated on our designs and provided additional comparative experiments and ablation studies above. We further discuss this point below:
- The mentioned works, namely GenManip and HumanoidGen, are relevant to agentic / LLM-driven task generation yet have different focuses compared to ours.
- GenManip is a simulation platform for evaluating policy generalization. It proposes task-oriented scene graphs to represent task semantic layouts and enable effective LLM-driven task generation.
- Its task generation pipeline centers on task schema generation and does not produce the action sequences required for articulated object manipulation.
- While it annotates feasible states of articulated objects, it lacks the part-level visual and manipulation annotations that constitute a core contribution of our work.
- It collects trajectories via teleoperation and leverages MimicGen for articulated object manipulation. By contrast, SMART investigates how to synthesize trajectories for articulated objects, extend the pipeline to more object categories, and scale up data collection efficiently.
- HumanoidGen targets the adaptation of LLM-driven task generation for dexterous dual-arm manipulation.
- It does not perform part-level annotation for articulated objects and adopts only a generic manipulation formulation.
- It validates its method on a restricted set of articulated object categories and does not pursue data scaling.
In summary, for articulated object manipulation tasks, none of these prior works incorporate designs for agent-driven task generation or articulation-aware motion generation.
Prior datasets such as InternData-A1 serve as suitable baselines for comparison, yet they do not investigate how to effectively extend demonstration synthesis to a broader range of articulated objects or how to scale up data collection efficiently.
We highlight this research gap in our introduction and related work sections, which motivates the contributions of SMART.
Q2:
Q2.1 The comparison target mentioned above, InternData-A1, states that it uses 14 categories of articulated objects in its dataset. SMART uses 23 categories of articulated objects for demonstration synthesis, as shown in Fig. 4(c).
InternData-A1 provides 15 articulated object manipulation tasks, whereas SMART provides 44 types of atomic articulated object manipulation tasks.
This provides direct evidence of SMART’s advantage over InternData-A1 in articulated object category coverage and task coverage.
Q2.2 We show the advantage of SMART-Data in our response to the weakness section above.
Q3: We understand your concern about the pretraining effect, and clarify the role of the two simulation benchmarks as follows.
- LIBERO and RoboCasa365 are designed to evaluate general manipulation capability. They cover a broad mix of tasks, and articulated-object manipulation is only one part of them.
- SMART focuses on scaling data synthesis and supporting sim-to-real transfer for articulated-object manipulation, and SMART-Data consists mainly of such tasks.
- We therefore use these two benchmarks to evaluate how SMART-Data pretraining affects the general manipulation capability of the policy. We do not claim, based on the results on these benchmarks, that SMART-Data has unique value for pretraining.
- The concrete value of SMART-Data is demonstrated by our real-world evaluations and by the additional experiments reported in our response to the weakness section.
- These tasks are contact-rich articulated-object manipulation tasks, which align with the capabilities that SMART-Data targets. Training on SMART-Data yields better model performance on these tasks, which directly demonstrates the unique value of SMART-Data.
- We further explore the source of this value in our additional experiments and ablation studies. We refer the reviewer to the main response and to our response to the weakness section for details.
Q4: We have elaborated on our designs and provided additional experiments and analysis in our response to the weaknesses section.
We hope the above responses address the concerns you raised regarding our work.
We remain open to further discussion, and we would be grateful if you would consider raising your score should these concerns be resolved.
Rebuttal and Response to Reviewer 87rW
We appreciate your detailed and constructive feedback. We understand your concerns regarding the novelty and contributions of our work, and we address them below.
Addressing the concerns in the weakness section
We demonstrate the novelty and effectiveness of SMART by elaborating on its design and presenting comparative experimental analyses as follows:
1. The novelty and effectiveness of SMART’s design.
SMART’s articulation-aware designs are twofold:
(1) Part-level annotations and an articulation-aware skill library that let an agent generate executable articulated-object manipulation tasks. The designs in this part include:
(2) A demonstration generation step designed for articulated-object manipulation trajectories. The designs in this part include:
The effectiveness of these designs is supported by our results on synthesis efficiency (task-generation automation and trajectory success rate) and on downstream performance (comparison with InternData-A1 and ablations, which is shown in the section below). Please see the main response for the detailed design description, results, and conclusions.
2. The comparison experiments and ablation studies.
Since multiple reviewers raised concerns regarding this section, we address it in our main response at the top of this page. (Please search for “additional experiments” on this page). We refer the reviewer to that section for details on the experimental designs.
The results are shown in the table below:
We summarize our observations and analysis as follows: